Here is a nice presentation about the use of the Scala language at Twitter:
Besides the counless features of Scala (type interfence, Traits, Pattern Matching, ...), I found the reasons interessing the speaker gave against Ruby (Twitter was once a major landmark project for Ruby and Rails) : "Ruby's poor VM performance, monkeypatching and cultural issues, questionable for large systems". I found Monkeypatching terrible. Is there an equivalent to Liskovs principle for monkeypatching?
A well known feature of Scala is it support for message passing concurrency via Actors similar to Erlang. In Chapter 30 of "Programming in Scala" Scala's Actors is described in detail.
However, it is not so well-known that Scala also supports actors distributed across different nodes (Remote Actors). Here is the Ping-Pong example with the Ping actor and the Pong actor running in different processes on (possibly) different (cluster) nodes. The example also shows the actor-linking support of Scala. Linking an actor a to an actor b, means that a is notified if b terminated.
Messages:
package de.dirkmeister.pingpong
case object Ping
case object Pong
case object Quit
Ping:
package de.dirkmeister.pingpong
import scala.actors.Actor
import scala.actors.Actor._
import scala.actors.Exit
import scala.actors.remote.RemoteActor._
import scala.actors.remote.Node
object RemotePingApp {
def main(args: Array[String]) : Unit = {
val port = args(0).toInt
val peer = Node(args(1), args(2).toInt)
val ping = new RemotePing(port, peer, 16)
ping.start()
}
}
class RemotePing(port: Int, peer: Node, count: Int) extends Actor {
trapExit = true // (1)
def act() {
alive(port) // (2)
register('Ping, self) // (3)
val pong = select(peer, 'Pong) // (4)
link(pong) // (5)
var pingsLeft = count - 1
pong ! Ping // (6)
while (true) {
receive { // (7)
case Pong =>
Console.println("Ping: pong")
if (pingsLeft > 0) {
pong ! Ping
pingsLeft -= 1
} else {
Console.println("Ping: start termination")
pong ! Quit // (8)
// Terminate ping after Pong exited (by linking)
}
case Exit(pong, 'normal) => // (9)
Console.println("Ping: stop")
exit()
}
}
}
}
Pong:
package de.dirkmeister.pingpong
import scala.actors.Actor
import scala.actors.Actor._
import scala.actors.remote.RemoteActor._
object RemotePongApp {
def main(args: Array[String]) : Unit = {
val port = args(0).toInt
val pong = new RemotePong(port)
pong.start()
}
}
class RemotePong(port: Int) extends Actor {
def act() {
alive(port)
register('Pong, self)
while (true) {
receive {
case Ping =>
Console.println("Pong: ping")
sender ! Pong
case Quit =>
Console.println("Pong: stop")
exit() // (10)
}
}
}
}
Notes:
(1) By setting trapExit, the linked actor is notified by sending an Exit(sender, reason) message. Otherwise the termination is either ignored (if reason is 'normal) or the linked actor is terminated, too (reason != 'normal).
(2) alive(port) (member of the RemoteActor object) starts the remote service listening on the given port
(3) register(symbol, actor) (member of the RemoteActor object) registers the given actor using the symbol. The other actors can then lookup the actor by the hostname and port and this symbol.
(4) This lookup is done by the select(node, symbol) method that returns an proxy actor, which managed the complete transmission.
(5) The link method links the Ping actor with the Pong actor, so that the current actor is notified if the Pong actor is terminated. This example shows that this also works remotely.
(6) and (7) That proxy actor is used to send messages to the remote node. Sending and receiving remote messages is similar to local messages. Well, everything must be serializable, but the use of case classes is recommended anyway.
(8) The Quit message stops they Pong actor. See (10)
(9) When the Pong actor terminates, an Exit(sender, reason) message is sent to the Ping actor. This is linking system is used for error handling, here it is used to terminate the Ping actor, too.
(10) The Pong actor calls the exit() method, which terminates the actor with the reason 'normal.
I think Scala's Remote Actors are really nice. Well, it lacks Erlang's capability to spawn actors on a different node and maybe other things, but it has the property that it isn't written in a 20 years old language: It is written in a modern, OO/functional-hybrid language.
According to a comment on the Lambda blog, there is (or was?) and effort to use the Java P2P protocol JXTA for remote actors. That would be cool, but I found nothing newer about that effort. Here is a description about clustering Scala actors via Terracotta. Here one using Oracle Coherence for that.
So richtig offiziell habe ich es hier im Blog nach gar nicht gesagt, aber seit einiger Zeit habe (und benutze mit unregelmäßigen Abständen) einen Account bei Twitter. Ab heute sogar mit Profil-Foto.
Laut heise.de will sich das Land Sachsen mit 150 Millionen Euro für den Speicherchiphersteller Qimonda beteiligen. Durch die Pleite wären die Arbeitsplätze von 3200 Mitarbeiten bedroht.
Das sind mal eben 46.000 Euro pro Arbeitsplatz, die vom Steuerzahler aufgebracht werden müssen. Das sind Größenordnungen, die fast an die Kohlesubventionen im Ruhrgebiet herankommen.
Der Haushalt des Landes beträgt 3,4 Milliarden (Quelle). Das bedeutet, dass 4% des gesamten Haushalts in ein Unternehmen gepumpt wird, dass "seit Geschäftsbeginn im Jahr 2005 [...] Milliardenverluste einfährt" und deren "strukturellen Probleme seit Langem bekannt sind" (Quelle).
Mir ist nicht klar, was außer reinem Populismus Politiker dazu bringt, so das Geld aus dem Fenster zu werfen. Sie können doch nicht ernsthaft glauben, dass die Strukurprobleme der Firma irgendwie gelöst werden, wenn man mal eben etwas Bargeld hineinpumpt. Wenn die Firma dann 2010 anstatt 2009 insolvent geht, dann ist das Geschrei wegen der "Heuschrecken" und den bösen Unternehmen wieder riesig. Das ist doch abzusehen. Ich kann, wenn ich die Nachrichten im Moment verfolge, nur den Kopf schütteln. Mit 150 Millionen lässt sich so viel sinnvolles anstellen, aber der Firma ein oder zwei Jahre zu schenken, gehört ganz bestimmt nicht dazu. Aus Holzmann und Co. wurde offensichtlich nicht gelernt.
In der letzten Minute der Masterarbeit hab ich noch Latex-Probleme bekommen: Warum?
Bei normalem Buchdruck die erste Seite links ist, da dies die Rückseite des Buchrückens ist. Damit sind ungerade Seite links und gerade rechts. In Diplomarbeiten (oder Bachelorarbeiten, Masterarbeiten, what ever) ist die erste Seite (die Titelseite) rechts, weil es das erste gebundene Blatt ist. So sind gerade Seiten links und ungerade Seiten rechts.
Wenn man dort nicht aufpasst und nur "twopage" in "documentclass" verwendet, hat man schnell ein Dokument gesetzt, was jede Seite einzeln betrachtet richtig aussieht, aber global falsch gesetzt ist. Die größeren Rändern sind außen anstatt innen. Das Ergebnis ist hier zu sehen:
Daher müssen die bei Rändern so umgestellt werden, dass sie bei den geraden Seiten links größer sind und auf den ungeraden Seiten rechts größer. Ich habe dafür diese Befehle von Robort Close übernommen:
setlength{voffset}{-10pt}
setlength{topmargin}{10pt}
setlength{headheight}{14pt}
setlength{headsep}{25pt}
setlength{textheight}{660pt}
setlength{footskip}{30pt}
setlength{hoffset}{0pt}
setlength{oddsidemargin}{1.4cm}
setlength{textwidth}{426pt}
setlength{marginparsep}{0pt}
setlength{marginparwidth}{0pt}
setlength{marginparpush}{0pt}
setlength{headwidth}{textwidth}
setlength{evensidemargin}{paperwidth}
addtolength{evensidemargin}{-textwidth}
addtolength{evensidemargin}{-2.0in}
addtolength{evensidemargin}{-oddsidemargin}
Danach sieht es so aus:
Der zentrale Tipp ist es, sich die Datei (wie in den Screenshots) mit "Vorschau" anzusehen. Dort wird das Titelblatt richtig (rechts) angezeigt und alle nachfolgende Seiten (gerade links und ungerade rechts). Damit lässt sich erkennen, ob das Dokument richtig gesetzt ist.
Interessanterweise zeigt bei der Doppelseitigen Ansicht Adobe Reader die Titelseite links zusammen mit der 1. geraden Seite rechts und danach jede ungerade Seite links und jede gerade Seite rechts. Also genau wie es i.d.R. für Diplomarbeiten falsch ist. Daher Tipp nur 2): Wenn es in Adobe Reader richtig aussieht, ist es für eine Diplomarbeit in der Regel falsch.
With high probability, I will start as research assistent (aka PhD student) at the Paderborn Center for Parallel Computing.
There I will research in the area of storage systems, especially on the topic "data deduplication". It is really nice to be able to continue and complete the work of my master thesis.
Spiegel Video: http://www.spiegel.de/video/video-40242.html
Der Veranstalter sagt, es wurde ein Zeichen gesetzt.
Da hab ich Angst um dieses Land. Warum wird Linke-Gewalt in der Öffentlichkeit so geduldet? Ich verstehe es nicht. Die Spiegel-Reporterin meint lapidar: "Die Schüler spielen etwas 68er". Was machen diese Schüler erst wenn sie groß sind? Andersdenkende an die Wand stellen?
I knew that I had read it some where, but I forgot. I have found it again: The white paper "The New Metrics of Disk-based Data Protection" by the Strategic Research Cooperation claims that Diligent's Hyperfactor approach works in three steps - compress, encrypt, and then eliminate redundancy".
I really, really want to know how you can de-duplicate that have been compressed and encrypted before. Data that was similar to already known data elements has to be complettly different after compression and encryption. Then there - as I understand it - simply can't be any similarity left between the data elements.
How is data compression fundamentally different from data de-duplication? I really don't see it. But I'm not convinced that there is no need for a new word either.
Jon Bentley presented a compression algorithm for finding und eliminating common long strings in 1999. The approach works by dividing a stream of bytes into chunks of static size (he uses 1K chunks for the evaluation) and then hash them using Rabin's fingerprinting method. He maintains a table of all fingerprints seen up to date, and there he lookup up the hash value of a new chunk to check if a chunk with the same value was seen before. If there is such a collision, he checks these two chunks byte for byte. If the chunk was not seen before, the hash value is added to the table. If the chunk was there already, a link to an existing occurrences is stored.
The hash-based data de-duplication approach works as follows: The divide a stream of bytes into chunks (often using a more elaborate chunking method that is often falsely called "Rabin Fingerprints" because it used that fingerprinting technique), and then hash them using a cryptographic fingerprinting method (like SHA-1). Dedup systems maintain an index that contains all fingerprints seen up to data, and there they lookup up the fingerprint of the new chunk to check if the chunk was seen before. If the chunk was not seen before, the chunk is stored. If a chunk contains the same data (whp.) the data is not stored.
Looking similar?
In contrast to Jon Bentleys approach, it is common to skip the byte-by-byte comparison for performance reasons claiming that the collision probability using a cryptographic hash value is much lower than e.g. a random byte flip in memory. If there is a data loss, it is whp. not caused by a fingerprint collision. However, even that is not inherent to the de-duplication approach. You could easily perform a byte-by-byte-comparision in dedup systems, too.
One problem — from a students perspective writing a master thesis about that topic — is that before 2008 I found no evidence of the term "data de-duplication" (in a storage context, not in a data mining context) in research literature. The first usage I found was Zhu's paper about Data Domain's ways to avoid the disk bottleneck of hash-based data de-duplication.
There was an interesting discussion about the difference between the storage blogs "Backup Central" and "StorageMojo".
A StorageMojo author says:
I still don’t get why the industry refers to “de-duplication” rather than compression - why use a well-understood term when you can invent a new one? - but they did make the point that compression rates depend on your data types, backup policies and retention policies. Basically the more stuff stays the same the higher your back up compression rate.
W. Curtis Preston of "Backup Central" takes on that.
There are different definitions to distinguish compression from data de-duplication. Here are few tries:
A commentator at "Backup Central" pointed out that data de-duplication "has more in common with image compression than text compression" and both dedup classes "look like out-of-order MPEG-4 compression". I honestly have no clue about image compression and MPEG-4 (I should!), but maybe he is right. It seems like the text compression approaches based on LW and the implementations like ZIP are simply stated as "compression". The whole concept of compression is reduced to that.
For me the goals are the same: To save storage capacity or bandwidth usage by finding and eliminating redundancies. The approaches are sometimes remarkable similar, while the implementations are not. May be de-duplication is just a subfield of compression