Can anyone explain how MapReduce is built on top of Borg, when Borg doesn't support scheduling with data locality? I'm having a hard time reconciling that. One of the principal use cases of the MapReduce framework is reducing data movement...
From the MapReduce paper: "The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data."
One of two possible designs immediately come to mind: (1) The MapReduce master interacts with Borg directly on behalf of the programmer and places a requirement on the task to run on or near any node that has a replica of the data required or (2) for each Borglet, there is an accompanying MapReduce agent running to accept RPCs from the MapReduce master.
You seem to be suggesting that using local i/o to read inputs would be preferable to using remote i/o, but I don't think that's a supportable statement in a datacenter with a high-performance network.
Even with high-performance networks in place, you absolutely must budget your bandwidth between management communications, low-latency service communications, and batch job communications, amongst possibly even others. Having data locality is still a really, really big priority.
(Furthermore, building out separate networks is extremely cost-prohibitive once you've expanded out to a total footprint in the low-1e6 ballpark.)
Think about it this way: In recent history, we've added L3 cache to commodity processors because keeping data on-chip is still faster, in 2015, than going off-chip to RAM. The same analog holds when trying to fetch from local disk versus remote disk.
Brendan Gregg actually provides a really good time-stretched view of various operations in his book on performance tuning. Go give that book a read; there's some _really_ good stuff there.
It just doesn't seem very likely to me. It seems a lot more likely that the data would be spread around on thousands of individual hosts, one megabyte at a time. The only processes by which I imagine the entire contents of one shard of a mapreduce ending up in memory of a single host are 1) a miracle, 2) the previous stage of the mapreduce wrote it there, in which case it is effectively remote access before the fact, and 3) the data lives there all the time for some other reason, in which case you don't have a mapreduce, you have a coprocessor.
I agree about the one megabyte at a time. I disagree that that implies it's likely evicted from RAM.
For one, if I'm running a MapReduce, it's either likely on the same data someone else is investigating... Or I'm likely to have to re-run my MapReduce a few times, calculating exactly what I'm looking for.
So, I may be cheating by saying "Only the first time is the data cold. Every other time, the data is likely to be hot in RAM."
Because you'd be right to say, "Well, duh. I'm talking about the FIRST time. And since I'm talking about the first time, moving the data across the network wouldn't be that bad. After that, yes, of course it will still likely be hot, just as you describe."
My only real response is, I think it depends on how much data, versus how much code you're talking about. And can you maybe even broadcast the code to the right nodes, to even more effectively use the network.
"in which case you don't have a mapreduce, you have a coprocessor."
...a coprocessor which uses the MapReduce API. Yes, I think that is exactly what you have. And it works whether the data is local or remote, cold or hot. One unified API for all of those things, and it's probably optimized to work local and hot, because I'm betting more than half of all MapReduces actually work out to be local and hot.
Mapreduce workers can request shards (work) that have data local to the host it is running on. For large mapreduces this basically works because both the data and the tasks are in most of the cluster, and for small mapreduces it doesn't matter.
