
On the Death of Map-Reduce at Google - aburan28
http://the-paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-reduce-at-google/
======
ot
Previous discussion:
[https://news.ycombinator.com/item?id=7947782](https://news.ycombinator.com/item?id=7947782)

------
enneff
> _So it’s no surprise to hear that Google have retired Map-Reduce._

As a Googler my reaction to this statement is "LOL". We still do a ton of map
reduces.

~~~
_ak
I'm not sure if you're allowed to talk about it, but...

I remember a talk by Rob Pike where he mentioned that Sawzall was one of the
first major MR users within Google. But IIRC, Sawzall has been retired at
Google some time ago, and the successor to it was developed in Go. Does the
successor follow the same or similar principles in terms of parallelization,
or did you follow another approach?

~~~
enneff
The successor to Sawzall is basically a framework for writing mapreduces in
Go.

------
dundun
MR is actually growing at Google still and heavily used by legacy systems and
experienced engineers.

Compared to newer frameworks like those described in the FlumeJava and
MillWheel papers, MR's growth is flat.

This is also happening in the Hadoop ecosystem too: if you're writing JavaMR
by hand, you're probably spending more time and writing less efficient jobs
than what you might get by an optimized pig/hive job with tez under the hood.
Or through something in the Cascading or Crunch family, which provides useful
abstractions on top of MR or other execution engines.

Then there are also a lot of tools popping up that take some if the use cases
that were shoehorned into MR which are more natural outside, like ML/iterative
computation through Spark.

MR isn't dying inside or outside Google, it's just being abstracted away.

------
rwmj
I love how the old research into this is just ignored. There is a ton of work
which has showed for 20 years or more that map-reduce is just a subset of
interesting parallel computation patterns. eg:
[http://pubs.doc.ic.ac.uk/parallel-
skeleton/](http://pubs.doc.ic.ac.uk/parallel-skeleton/)

------
MrTortoise
I'm kind of late to this map-reduce party in terms of large scale data
processing. What is the paradigm shifting towards. The point about MR being a
trivial subset seems rather obvious to me. Hadoop for instance is still in a
phase of getting a lot of adoption yet is being implied as being a dead man
walking. Raises a lot of very interesting postivie questions for me

------
geofft
The link makes more sense without the anchor:

[http://the-paper-trail.org/blog/the-elephant-was-a-trojan-ho...](http://the-
paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-
reduce-at-google/)

------
threeseed
Businesses are already mixing map reduce with other approaches. On the Hadoop
platform we have Spark, YARN, Storm etc which all work on the existing
infrastructure.

Map reduce however is still perfectly suited for many batch processes.

