
The Elephant was a Trojan Horse: On the Death of Map-Reduce at Google - danso
http://the-paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-reduce-at-google/
======
timr
_" and the real contribution from Google in this area was arguably GFS, not
Map-Reduce"_

This, a million times over. Map-reduce is not difficult to implement.
Implementing a _distributed, petabyte-scale filesystem_ to hold the data being
accessed by thousands of workers is what's difficult.

It's just a shame that Hadoop (and HDFS) are what we're saddled with in the
outside world. They're a total disaster area, from configuration difficulty to
memory usage to speed to monitoring. But since HDFS is the only commonly used
distributed FS, you're pretty much bound to Hadoop (and the rest of the
horrible Apache ecosystem).

The world needs a good, stable, well-written distributed filesystem (ideally,
one not written in a bloated language designed for remote controls and set-top
boxes).

~~~
random3
First, I don't believe that Map-Reduce is not used inside Google anymore.
Second, the map-reduce pattern is actually very useful basic, common
computation. It's equivalent to the SQL JOIN and it's not something that you
can really do without. Perhaps the large chunk (batch) approach is not ideal
for many of the use-cases Map-Reduce is being tried with (e.g. like
"interactive" querying with Pig or Hive). But that doesn't mean it's not
useful. If you're optimizing for throughput you'll generally want to
read/process in the batches optimized for some underlying sizes (it could be
page size, it could be blocks, etc.).

Also a system is a lot more than just the compute framework. It needs to deal
with various inputs/outputs, do scheduling, etc.

Between the distributed storage and the distributed processing, I'm not sure
it's easy to decide which one could be more difficult, either.

Saying that Hadoop is a disaster, is not far from saying we live in an awful
world. Working with them for years doesn't make me the most objective person,
however, given the huge adoption, I'd say they may not be that bad. More,
using something like Cloudera Manager makes it trivial (which sometimes makes
me wonder why the vanilla version hasn't been improved...) (BTW there's QFS
and other distributed file systems).

I wonder why is the Apache ecosystem horrible?

I get it that you don't like Java. Fair enough. What would be your language of
choice for the next gen, stable distributed file system? Go, Rust, JavaScript?

~~~
wyager
>What would be your language of choice for the next gen, stable distributed
file system?

Here's my heavily biased subjective opinion on this entirely hypothetical
software:

I think we should do one or both of two things:

A) Do it in very clean, fast, simple C. Put an emphasis on speed and
simplicity.

B) Do it in very reliable, secure, simple Haskell. Put an emphasis on
correctness and simplicity.

With some effort, the C one could be correct and the Haskell one could be
fast.

I mention these two languages because they compile to native code and have
very good cross-platform support. You won't have any trouble running either of
these on embedded devices (which I can't say for Java or Go. Go has some weird
compiler bugs on ARM platforms, and the JVM is frequently too memory intensive
for embedded). C has an advantage of allowing the absolute minimal
implementation, and Haskell has an advantage of allowing a massively
concurrent implementation. Yada yada yada

Of course, it could be that the question is completely irrelevant. Just define
a spec for a DFS, and then let different implementations pop up in whatever
language is best suited to that implementation's specific details.

~~~
nl
_You won 't have any trouble running either of these on embedded devices
(which I can't say for Java or Go. Go has some weird compiler bugs on ARM
platforms, and the JVM is frequently too memory intensive for embedded)._

Why is this important in _this_ use-case? If the DFS is being used for data
processing then presumably the nodes are reasonably capable machines.

There may well be a difference use-case for a DFS for embedded and resource-
constrained devices. That's not what Google or Hadoop is doing though.

~~~
vidarh
The biggest limiting factor even in our relatively low-density populated rack
is heat and power. With off the shelf servers and relatively low density, I
can trivially exceed the highest power allocations our colo provider will
normally allow per rack. The more power you waste on inefficient CPU usage,
the less you can devote to putting more drives in.

~~~
nl
The OP's claim is that _memory_ is the limiting factor in the case of Java. I
don't entirely agree, but even if I did it would almost certainly be a fixed
overhead per machine, and unlikely to be a problem on server class machines.

Also, the read/processing characteristics of compute nodes often means the CPU
is underutilized while filesystem operations are ongoing.

~~~
srean
I will leave with an elliptical meta-comment, for those whose competitive
advantage lies in others not getting it right, have little interest in
correcting misconceptions. You might have interest in this anecdote
[https://news.ycombinator.com/item?id=7948170](https://news.ycombinator.com/item?id=7948170)

~~~
nl
But how much of that is Java, and how much is Hadoop?

Spark runs on the JVM, and much, much faster than Hadoop on similar workloads
(Yes, I understand it isn't just doing Map/Reduce, but the point is that Java
doesn't seem to be a performance limitation in itself).

~~~
srean
Indeed and as I said it did surprise me that Hadoop was so much slower. But
the buck really stops at resources consumed per dollar of usable results
produced, and in that Java is going to consume a whole lot more. At large
scales, running costs far exceeds development costs. BTW my point was not only
about Java but also about your assessment of the hardware.

------
jmillikin

      > This morning, at their I/O Conference, Google revealed
      > that they’re not using Map-Reduce to process data
      > internally at all any more.
    

This is incorrect, so monumentally so that I couldn't continue reading. It's
as if the author had opened an article about climate change with "Now that
advancing glaciers have rendered Algeria uninhabitable..."

MapReduce doesn't work well for low-latency pipelines because it's got a high
fixed overhead, but it's still the undisputed king of medium-latency and
latency-insensitive workloads.

~~~
jbigelow76
You may want to correct Urs Hölzle, Senior VP of Technical Infrastructure at
Google, then or at least tell him to choose his words better.

From today's I/O keynote video
[https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454](https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454)

This is the exact quote:

    
    
        "... and today even when you use map-reduce, which we invented over a decade ago, it's still cumbersome to write and maintain analytics pipelines, and if you want streaming analytics you are out of luck. And in most systems once you have more than a few petabytes they kind of break down. So we've done analytics at scale for awhile and we've learned a few things. FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE. It's great for simple jobs but it gets too cumbersome as you build pipelines, and everything is an analytics pipeline."
    

_emphasis mine_

Of course the word "really" in the middle of the sentence gives semantic
wiggle room, but it's still a pretty big statement.

~~~
gdy
>"FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE" And this is said in the
context of talking about streaming analytics.

~~~
jbigelow76
But Urs also said, paraphrasing this time, that once you get into petabytes of
information everything pretty much becomes streaming analytics.

Since I would assume that any non-trivial service that Google provides is in
that petabyte neighborhood it explains why he would say that Google isn't
using MR anymore.

------
tupshin
Projects like Apache Spark have demonstrated the power of a more complex DAG
(Directed Acyclic Graph) approach that allows for more precise control over
the data-processing flow, compared with the simpler execution model of M/R.
All of the major Hadoop vendors are pivoting, and simultaneously adding
support for Spark (which can work with the Hadoop stack, but is not part of
it), while also supporting the development of one or more technologies that
are trying to retrofit Hadoop into a more powerful model, such as Tez, from
Hortonworks.

------
chris_va
'The company stopped using the system “years ago.”'

Hm, as a former Google engineer, that statement (from the journalist) is not
accurate. Though the definition of map reduce is malleable, so it's hard to
say what was meant in the first place.

------
nevi-me
Didn't Urs mean that it's not used for analytics anymore?

------
dpritchett
As a non-PhD who doesn't work at Google I'm having trouble reading between the
lines. What exactly is the offered improvement here?

Decoupled (compared to Hadoop) systems with distributed data and JIT
processing?

~~~
EdwardDiego
The offered improvement is, I guess, process Google-sized 'Big Data' without
the significant issues of Hadoop style clusters. Beyond that, there's not
enough details.

To give you an example, data import speed has always been an issue with Hadoop
- Facebook are quite proud that they can import 320TB of data into their
Hadoop clusters in a day.

~~~
tsotha
Yes, it all seems very hand-wavey to me. "Map reduce is finished, and it's
been replaced by... some other thing."

------
riffraff
> It will also be no surprise to me when, eventually, Hadoop does the same,
> and the elephant is finally given its dotage.

AFAIU, hadoop has already moved to support different workloads than MR when it
introduced YARN[0]

[0] [http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-
yar...](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-
site/YARN.html)

~~~
ambrood
That isn't "hadoop supporting different workloads". That is a Hadoop cluster
supporting different frameworks. They are two different things.

------
ohashi
So what's replacing it in the open source world? What are some real world
problems Map Reduce is being used for now that some other solution is better
suited for?

~~~
toast0
A lot of problems people are solving with Hadoop and friends are best solved
by getting a big box with lots of ram and lots of SSDs. Current generation
dual processor socket server boards go up to 1 TB of ram, a few years ago that
kind of ram would require several servers; and the Haswell EP Xeons, expected
towards the end of the year will support even more ram.

~~~
ohashi
So you're essentially saying it was a stop-gap solution until hardware caught
up for many use cases.

------
erikano
>[P]aradoxically, it’s easier to build robust, fault-tolerant systems from
unreliable components[.]

Giving this some thought, I think I see a parallell here to how most the rest
of the nature works; the "components" that make up a creature appears to often
be as simple as it can afford to be.

------
rlpb
So they're not using MapReduce, a particular implementation of the map/reduce
concept. But are they still using the concept having just changed their
implementation? I've read the article and this is still unclear to me; I think
commentators are conflating the two.

------
_pmf_
Map-reduce is a brute force approach. It's obvious that intelligent, specific
data handing strategies are orders of magnitude better for production systems
if there's the engineering power to pull it off.

~~~
jeffdavis
The author discussed the data flow model -- would you consider that to be a
"specific data handling strategy"?

Also, why do you think developing for a map-reduce framework requires fewer
engineering resources than developing for, e.g. a dataflow framework?

------
coldcode
Computing darwinism. Many things are tried and most of them fail, but each try
teaches something useful for the next generation.

~~~
kitd
I'd say it teaches the current generation. IME the next generation tend to
repeat the mistakes of their predecessors much more easily than our industry
should be happy with.

