
Building a supercomputer from 64 Raspberry Pis and Lego - Sukotto
http://www.southampton.ac.uk/mediacentre/features/raspberry_pi_supercomputer.shtml
======
fpp
A technical description and how they got it running together with more
pictures:

[http://www.southampton.ac.uk/~sjc/raspberrypi/pi_supercomput...](http://www.southampton.ac.uk/~sjc/raspberrypi/pi_supercomputer_southampton.htm)

As you can see from the pictures each of the devices uses its own power
supply, so there should be quite some possibility to improve overall power
consumption.

Anybody with more experience of using MPI ?

~~~
apawloski
It is almost always the case that programs written for clusters -- including
those which use MPI -- are communication bound. Right now the limiting factor
is not these little processors, but rather the interconnect speed. If I
remember correctly, Raspberry Pi has 10/100 megabit ethernet. [Edit: just
checked, this is the case] So while this looks like a lot of fun, it's not
very useful for anything meaningful yet.

Of course it's not fair to compare this to an infiniband cluster (that's not
the point of this exercise), but I'd really be interested to see a cluster
built on $0.50 ARM chips with at least a gigabit ethernet interconnect. A
couple of years from now -- given the low entry cost and lower infrastructure
costs (cooling/power consumption/etc) -- that could be a game changer.

~~~
jacques_chester
Before the GFC killed them, SiCortex had a design where MIPS cores were
grouped together on a single die with MPI-specific fabric logic.

~~~
adestefan
This is the reason why Cray is still relevant. Their proprietary interconnects
are what you're paying for and not the CPUs.

~~~
wickberg
Oddly enough, Cray recently sold their interconnect tech to Intel [0]. Intel
seems to be planning to integrate it on-chip down the road [1], which seems to
leave Cray serving as a somewhat quirky system integrator longer-term.

[0]
[http://newsroom.intel.com/community/intel_newsroom/blog/2012...](http://newsroom.intel.com/community/intel_newsroom/blog/2012/04/24/intel-
acquires-industry-leading-high-performance-computing-interconnect-technology-
and-expertise)

[1]
[http://www.hpcwire.com/hpcwire/2012-09-10/intel_weaves_strat...](http://www.hpcwire.com/hpcwire/2012-09-10/intel_weaves_strategy_to_put_interconnect_fabrics_on_chip.html)

------
montecarl
I ran some numerical simulations on my Raspberry Pi and on my laptop. With the
LCD on, my laptop used 20.4 watt seconds to do the calculation and the RPi
used 17.1 watt seconds. The RPi drew 3 watts and my laptop drew about 60
watts. I think, even with my screen on, that my laptop would be more efficient
if I had used 2 cores instead of 1 in the calculation (or if I had just turned
off the LCD).

My conclusion, the RPi doesn't even win in FLOPS/Watt let alone $/FLOPS.

~~~
anthonyb
Does it have to? It sounds like they're using this as a learning platform
rather than a serious $/FLOP/Watt cluster.

In that case you'd want to make it easy to get funding, and be optimising for
_initial_ cost. What would be the cost of 64 of your laptops?

~~~
montecarl
It certainly doesn't have to! I just wanted to share the results of an
experiment I did with numerical computing on the RPi.

Certainly laptops aren't the idea platform to compare against anyways. I'd
compare to a 1u node that you would buy for compute cluster. One can by a 64
core 1u node with 64 GB of ram for about $7k. If one wants to play around with
cluster computing you can emulate an entire cluster using one of those!

------
smortaz
what a pleasant surprise to see our little OSS project mentioned/used...
surprised as this is a Windows/VS Python IDE & they're running Linux on the
nodes - <http://pytools.codeplex.com>

------
jevinskie
I wonder how this compares to a Microwulf cluster [0] in terms of cost /
gigaflop. I'm guessing it isn't very efficient.

[0]: <http://www.calvin.edu/~adams/research/microwulf/>

~~~
wickberg
RasPi is ~ 175 MFLOPS per unit (CPU only, discounting the GPU). So, this
cluster works out to 11 GFLOPS, with 16GB of RAM for > $2500 USD.

For comparison, you could buy a motherboard, 32 GB of RAM, and an Intel i5
processor for $500 that will do over 20 GFLOPS.

So, it doesn't really stack up well from a price/performance standpoint. The
value of these systems is more teaching students how to work with parallel
code.

~~~
aortega
> The value of these systems is more teaching students how to work with
> parallel code.

There are no practical applications to this, it's just to show that it can be
done.

You can spawn 64 processes in a Core I7 and it would be about the same, just
faster. Or 64 VMs.

Maybe the real value of these systems is to teach students to design and
calculate before building it.

~~~
wickberg
Yes, from a performance standpoint there's no reason to go after this type of
system. No one is arguing this is a good system for production work.

The application of this is a teaching model. It's a lot easier to demonstrate
parallelism gains on this type of platform. Scaling beyond a single ARM core
is going to give you immediate performance benefits. Scaling further out to
the entire cluster will continue to show returns.

With a single desktop, once you go beyond ~4 cores the gains will drop off too
quickly. You just won't be able to see gains out to 64 threads on a single
CPU, where on this you should.

It also doesn't hurt to have a quirky architecture to get students excited by.
And yes, you could also spend some time discussing the architectural trade-
offs and why this is not a cost-effective system for production use.

~~~
aortega
You are right. I'm still new in this site and don't know how to upvote
comments so I wrote this.

~~~
alter8
I couldn't get this joke. What is this supposed to mean?

~~~
aortega
It was no joke. How I upvote something here?

~~~
alter8
Click the arrow that is to the left of the commenter's name. It seems you are
one of today's lucky 10,000: <http://xkcd.com/1053/>

I made a bad assumption because your account is almost one year old. I'm
sorry. Now I see that information is not given anywhere in this site. I think
this vote method was taken from Reddit.

------
K2h
I was trying to find a real comparison of MIPS of this thing compared to an I7
and landed on this page for performance data for the RPi. The most interesting
thing to me is power consumption data at the bottom - that shows idle with
network is around 370mA, which should mean (ignore power supply efficiencies)
that 64 of these things should use about 120W at idle.

<http://www.elinux.org/RPi_Performance>

~~~
regularfry
How do you get 120W? I get 24W.

~~~
synx508
0.37amps X 5volts X 64

~~~
regularfry
Oh, I'm an idiot. Misread it as 370mW.

------
Tichy
Would 64 Raspberry Pis actually be faster than a single current Intel CPU?
Just asking... it probably was a fun project anyway...

~~~
lallysingh
No, not even close. This is more of a fun decoration you have in your home for
visitors, as a conversation piece.

------
krohrbaugh
This is pretty cool.

On the other hand, stop sucking up all the supply for Raspberry Pis! My order
keeps getting delayed and it's making me sad.

------
msprague
Pictures
[http://www.southampton.ac.uk/~sjc/raspberrypi/pi_pictures.ht...](http://www.southampton.ac.uk/~sjc/raspberrypi/pi_pictures.htm)

------
jff
Nifty... but did you really write 64 SD cards, one after another? This is
1995-level clustering, you gotta PXE boot.

~~~
noonespecial
_Can_ RasPi's netboot? I honestly don't know. I still haven't gotten one yet.
Still waiting since placing the order.

The amazing thing about this project? Someone managed to get 64 RasPi's all in
the same place!

~~~
sitkack
They cannot netboot. The bootloader is crap.

~~~
ChuckMcM
Well to be explicit, the cannot netboot directly. I've built a u-boot that
boots from the network via NFS on other arm based systems. You still need an
MMC card to start it all off but once you get the network boot loader loaded
and running you can do what ever you want. I started with some code for the
Pandaboard that used that trick to boot from a USB attached drive.

~~~
sitkack
I will have to look into this, <http://kernelnomicon.org/?p=133> u-boot on the
raspberry pi.

------
Sukotto
more info (and build plans) here:
<http://www.southampton.ac.uk/~sjc/raspberrypi/>

------
philip1209
I submitted the same story from a different source four hours before . . .

<http://news.ycombinator.com/item?id=4510931>

~~~
sciurus
But you should have submitted this source instead.

From <http://ycombinator.com/newsguidelines.html>

"Please submit the original source. If a blog post reports on something they
found on another site, submit the latter. "

