On: Google Dumps MapReduce in Favor of New Hyper-Scale Analytics System

[espeed]

Yes, it's like Spark (http://spark.apache.org/) and SparkStreaming (http://spark.apache.org/streaming/) combined.

Here are the relevant papers...

* FlumeJava (iterative, data-parallel pipelines like Spark): http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F...

* MillWheel (fault-tolerant stream processing like SparkStreaming): http://research.google.com/pubs/pub41378.html

Pointers to the IO blog posts...

* "Reimagining developer productivity and data analytics in the cloud" http://googlecloudplatform.blogspot.com/2014/06/reimagining-...

* "Sneak peek: Google Cloud Dataflow, a Cloud-native data processing service" http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-g...

The Dataflow-specific talks at Google IO 2014...

* Big data, the Cloud way: Accelerated and simplified https://www.youtube.com/watch?v=Y0Z58YQSXv0

* The dawn of "Fast Data" https://www.youtube.com/watch?v=TnLiEWglqHk

* Predicting the future with the Google Cloud Platform https://www.youtube.com/watch?v=YyvvxFeADh8

* Keynote (starts at Urs Hölzle's segment on Google Cloud) https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932

[jey → espeed]

Cool. Does this mean Google is moving away toward languages that allow for easier use and serialization of closures than in C++ and Java? (For example, Spark uses Scala natively.)

[espeed → jey]

Dataflow is language agnostic. The Java API is being released first, and more languages will follow.