
Bigger data; same laptop - mrry
http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html
======
bane
I think we're starting to see the pendulum swing back towards caring about
performance. For years we could just throw more hardware at a problem, and now
performance curves are starting to flatten out. The bandaid has been to use
more systems, but old-timers are starting to remember back to their youth,
when systems were slow, that there used to be a phase in software development
called "performance optimization" that's almost entirely disappeared in the
modern discipline.

The amount of personal computing power every person with a modern computer has
at their fingertips is _staggering_ and almost nobody has any idea how to use
it.

I'd like to think that's starting to change.

~~~
gambiting
As someone who has recently obtained a Masters degree in Computing science and
who has few friends doing PhDs in big data analysis - I don't see that trend.
All the latest research techniques that I see being used currently concentrate
at throwing more hardware at the problem if it allows the researches to
express their ideas easier - I have a friend who writes algorithms for
processing 300GB graphs in Java. And basically the idea is that it really
doesn't matter if the algorithm takes 40 hours because 90% of the time is
spent garbage collecting. This is still preferable to writing much faster
program in C, and spending weeks trying to do memory optimizations and playing
around with pointer logic. If there is a need, then Uni will gladly pay for
any requested capacity on an Amazon cluster, it's part of the Russel group so
they have nearly unlimited budget for research. I am sure that's not the case
everywhere, but in my experience researchers gladly trade raw performance for
ease of use.

~~~
srean
To see that performance matters, talk to the guys who are closer to the
electricity bills that the big honking clusters demand. There is quite a bit
of money to be saved by using hardware effectively.

BTW if 90% time is spent in garbage collection, something is definitely wrong
in the choice of tools or methods.

~~~
gambiting
That was just an example, I shouldn't have said 90%, sorry.

------
eklavya
All that investment and involvement, those big systems have got to be good for
somethings! Please can anybody knowledgable share some insights? mrry?

~~~
jkot
It is one thing to run short benchmark on laptop. Other thing to sustain load
24/7 with demand spikes and hw failures.

~~~
mrry
This is absolutely true! However, I'm not aware of anybody doing this for the
graph processing systems being discussed in the post. Nor am I sure what that
would mean. Admittedly if something takes 100 times longer to run on multiple
machines than it does on a single laptop (as in the previous COST post [1])
then you might care about fault tolerance, but this is a contrived problem if
you could achieve the same results faster on yor laptop.

[1]
[http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...](http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html)

~~~
jkot
I was answering parent questions. There are usually good reasons for
clustering and big iron.

------
vardump
First you should optimize performance locally. By choosing an algorithm and
implementation that gives you the best results at data size _n_ you need to
deal with most. Lowest O complexity algorithm is not necessarily the best.
Worst cases and especially pathological cases also matter.

Then you need to deal with distribution between a large number of systems over
network... This will likely force you to choose a different algorithm, but at
least you'll have something the benchmark it against from local best case
scenario.

When it comes to graphs, I think good partitioning is very important in order
to minimize average case latency. So it'd greatly help if you can somehow
partition the data to minimize the number of nodes referring to a graph node
on a _remote system_. When walking the graph, instead of immediately sending a
query to the target system, it's probably better to batch the queries. Or not
if latency counts more. Either way, latency will hurt badly. On a local system
it's 100 ns. When the data is on a remote system, you're probably talking
about 1000000 ns.

To combat latency, maybe the best way is not to send the request in the first
place? Maybe you can ensure nodes referring to a remote system are duplicated
to some extent on multiple remote systems. It could help performance -- or
hurt it more, like when you need to modify the nodes.

Maybe you should have counters on nodes with connections to nodes on remote
systems. If some remote system is referring to it more than local system,
migrate the node there instead. It's of course not hard to see how this
strategy could blow up. :-)

Please don't take above ideas seriously if you're developing a graph
processing system. I wrote those to illustrate the biggest basic problem of
distributed systems, latency. Distributed systems are _very_ hard.

------
TheLoneWolfling
> it wears out my SSD

No it doesn't.

Reading doesn't cause (much) wear on an SSD - _writing_ does.

(To be technical - it does, but it only requires 1 write / 100,000 reads or
so. -
[https://en.wikipedia.org/wiki/Flash_memory#Read_disturb](https://en.wikipedia.org/wiki/Flash_memory#Read_disturb)
)

~~~
frankmcsherry
This probably isn't crystal clear from this post alone, but for PageRank the
42GB working set has a good mix of read and write. Specifically, about 14GB of
target rank vector gets +='d all over the place. If they got hammered
randomly, then lots of paging and writing would happen.

~~~
TheLoneWolfling
Oh, ok. Nevermind then.

------
andor
If you're wondering where the actual comparison is, open the link to "the
previous post" in the first paragraph.

~~~
frankmcsherry
That's a good point, thanks.

I've added the timings from the original post to the tables in this post,
which also makes clearer the intended point: the existing systems don't yet
have measurements for data at this scale (except Facebook's modified Giraph,
which I want to hear more about :))

