The graph on main page of MapRejuice shows ~300 jobs per minute. What is "a job"? Is it one iteration of "map" or "reduce"? So, if we are to assume you're doing word count - can you count up to 300 words per minute? On 6.5M visitors site?
In the case of word count, during the map phase, that means that 300 input files are analyzed per minute. During the reduce phase that indeed means that only 300 unique word frequencies per minute are computed.
This is a purely academic consideration, though. Currently we don't have any running jobs; the clients are idling. When idling, the clients request a new job every 10 seconds, which is longer than a typical reduce run would take. We're also throttling the load on the server from the workers (workers only spawn 50% of the time on the include script) because we don't want to ruin our hackety-hack server. Latency between the workers and servers and the server and our free couchone instance make up large portions of the running time of smaller jobs.
This was just an experiment. Once (and if) we figure out the problems I mentioned earlier we can start building to scale :)
Did you ever do or would it be possible for you to do a benchmark on relatively large corpus of how fast can you count, say 10 million words? Or 100 million?
I'm really curious about how well does this method work relatively to usual map-reduce on clusters. For comparison: single home PC does about 2.5 million word counts per minute using local files and mergesort instead of network.
This is why I asked - 300 words per minute using using 6.5 million randomly available nodes vs 2.5 million words per minute using one home node - that looks like a huge waste. Would it be possible to do more fair comparison?
Yes, we could benchmark that. Most likely our framework would be orders of magnitude slower. There's a couple of reasons for this:
- Data transfer is slow. Streaming the data in is not possible on all browsers.
- There is a very high sunk cost for each single computation, that is multiple times bigger than the time it takes for the average map/reduce function to run. This constant overhead per step is hard to minimize.
- A single map or reduce computation is further chunked down into multiple iterations of the same function. This is to avoid the "slow script" warning that browser show for long-running tasks. This technique is necessary and further adds overhead to the computation.
As a result our framework might not be well suited for many traditional MapReduce scenarios like word count. More suited tasks are ones that have a lot of overhead even if I ran them on my own cluster. This could be any tasks that involves pulling in data from a third party that is not local to the computation. For example counting word frequencies in all wikipedia pages will be vastly faster on our cluster than it could ever be on a single machine, even if we have only few workers.
We still need to figure many things out. The opportunities here are the availability of idle processing power and the possibility to sandbox computations in browsers. We need to find out exactly what the best way is to exploit those opportunities.
