
Let's Build a 28-Core Raspberry Pi Cluster - ColinWright
https://ikarus.sg/how-i-built-kraken/
======
ikaruswill
Hello, original author here. I saw a bunch of traffic from Hacker News and
decided to drop by.

Thanks for sharing it here! I'd be happy to answer any questions.

~~~
gchamonlive
Hey there. Thanks for the article!

I wanted to ask you if it is worth it implementing something similar in the
cloud to learn kubernetes. The hardware is prohibitively expensive on Brazil,
so maybe a cluster of t2.micro on AWS running for the total amount of 750
hours could serve the learning purpose for free

~~~
tinco
I don't know how it is in Brazil but in Europe and the US offices frequently
go out of business that have dozens if not hundreds tiny desktop computers in
their bankruptcy auctions. They've got i5 cpu's 4-8gb of ram and 128gb SSD's
and go for (much) less than a hundred bucks. Stack 3 of them and you have a
kubernetes cluster that you could carry in one hand and I think outperforms
the 7 raspberry Pi 3's.

Servethehome has a series of articles about the machines, though they're
focusing on the slightly higher end ones.

~~~
numpad0
I think it’s lease companies replacing paid off business PCs. They rent boxes
to offices on multi year contracts, collect and recycle them when the term is
up.

------
abraae
I have to admire someone who spends their first paycheck on a pi cluster:

> The total damage my wallet sustained from Kraken was S$626.63 or in $449.73
> USD, which was virtually all I had left for the month after subtracting
> expenses and education loan repayments.

~~~
panpanna
All power to him.

Nobody got successful by playing it safe.

~~~
quickthrower2
This is cash for joy!

------
ArchD
What's the actual use case for rpi clusters like this? If the goal is to learn
docker or k8s using a realistic setup involving multiple machines, can't it be
achieved by using multiple VMs on one physical desktop instead of using the
cluster? The 7-node cluster has only 7GB RAM in total, which is not too much
to ask on a desktop.

I don't know what people do with rpi clusters and I'm honestly puzzled why
people bother building them. If you get a beefy expensive server, I can
understand -- you care about performance. If you get a single rpi to serve
files & media in a home LAN it makes sense too -- low footprint, low power and
low cost. But I don't know what a rpi cluster excels at. Is it just a toy? Or
a learning tool to make computer science & networking concepts concrete and
tangible? Or just to get a computer system on a shoestring budget, much
cheaper than even a single desktop or laptop?

BTW, computer hardware in Singapore is overpriced. $140 for a switch in
Singapore vs $73 for a similarly-capable switch on Amazon.
[https://www.amazon.com/TP-Link-Ethernet-Unmanaged-
Rackmount-...](https://www.amazon.com/TP-Link-Ethernet-Unmanaged-Rackmount-TL-
SG1016D/dp/B004ELA0SS)

~~~
ikaruswill
The initial point was to build it just because, there was no real reason
behind building it. But at this point, I've found quite a lot of use cases for
it, specifically hosting web services for family and friends. In the process
though, I've learnt to work under extreme resource/budget constraints and that
alone has been invaluable imo.

~~~
ArchD
What storage do you use for the web services, given that SD cards are
unreliable? Do you have one disk per node or let them share one disk over the
network or something like that?

------
ulucs
Reminds me of moot "revealing" the "datacenter" that hosted 4chan:
[https://imgur.com/sBLRuZY](https://imgur.com/sBLRuZY)

I'm putting the air-quotes because I still sincerely hope that it was a joke

~~~
giancarlostoro
I'm more impressed he got trips, that's pretty amusing about the USB being as
fast as an SSD. Technically SSD's are supposed to be based off SD card tech if
I'm not mistaken, issue with a USB being based off the same tech is the USB
speeds themselves vs a SSD can go significantly faster due to its connector
(SATA or otherwise). Although I can see USB C being 'fast enough' instead.

~~~
Hamuko
> _I 'm more impressed he got trips_

There's probably admin tools for that.

~~~
giancarlostoro
I wondered about this, but sometimes people say it cause the previous post
they see is 999 so they post it, sometimes you get lucky as heck haha. I doubt
they'd screw with the admin tools just to do it.

~~~
nforest
On big boards there's so many posts per hour that it's impossible to really
predict this way, it's very much luck.

I remember people used to run scripts to be sure to get the big gets (6+ last
digits identical) but I suppose such methods don't work anymore with captcha
and other anti spam measures.

~~~
Hamuko
Pass users don't have CAPTCHA though.

------
magicalhippo
So I was toying with the idea of something similar. I'm a programmer and not a
sysadmin, but I've dabbled with Docker and had a specific use-case.

But so far I've not come further than this: how to deal with the host OS?

Like, I don't want to log into N computers and manually run "sudo apt update"
etc. I want them to run the same OS but with slightly different
configurations.

Network booting via PXE seems to be the best alternative, but not sure how to
prepare the images for distribution.

Anyone got any pointers on how this is best done?

edit: I should mention I just found
[http://cobbler.github.io/](http://cobbler.github.io/) but haven't had time to
investigate it. A viable choice?

~~~
Legogris
For the common base image, I just build a normal Debian image, customize it
(chroot) and flash it directly. For customization, maintenance and operations
you want some sort of CM (Ansible, Salt, Chef or Puppet). There is also
Packer, but it's not that easy to use for bare-metal images for some reason.

You could also do something like userdata where you have a script or playbook
that runs on first boot and performs additional setup.

I've been meaning to try out PXE booting but am yet to.

~~~
chabad360
If you can, I'd suggest using unified kernel images, they get rid of the
"whole PXE setup", you just send one file and you've booted.

------
Merocle
Hello! I did it like this:
[https://www.instagram.com/p/CC76bkADL-c/](https://www.instagram.com/p/CC76bkADL-c/)
It costed about 3k euro for me.

And this one special for k8s:
[https://www.instagram.com/p/CAqtrRBpNRH/](https://www.instagram.com/p/CAqtrRBpNRH/)

~~~
rdc12
Is the design files available for the rack mounted version (in particular the
clip/holder parts).

Been quite keen to build something similar (albeit less nodes) for a little
while. I find the homebrew blade style cluster amusing for some reason.

------
kristianp
"I started noticing the performance bottlenecks started when I tried to run
Wordpress on it, where a single page load of a newly installed Wordpress blog
takes about 10 seconds!"

10 seconds is probably not bad for an uncached wordpress page.

~~~
chillfox
10 seconds is terrible, a WordPress page should render in under 2 seconds.

------
sxp
A slightly cheaper option would be to get a bunch of Pi Zero W devices.
They're $10 + SDcards. They would rely on WiFi which might be acceptable if
the use case allows having a few dozen radios chattering next to each other.
If not, then a $5 Pi Zero + $5-10 ethernet adapter might work.

Since the goal isn't to achieve maximum performance, the slow CPU & RAM
(comparable to a 20th century desktop) might not be an issue. And the lower
price would allow more devices so you could test more complex configurations
and server management options.

~~~
rcarmo
Zeros are borderline unusable for actual compute due to having a single core
and only half a gig or RAM. They make for great beefy microcontrollers and
single-purpose boards, but are not the kind of thing where you want to run
various network services.

(I have a cluster running Pi 2s, which have 4 cores apiece, but are still too
short on RAM)

------
qubex
How timely... just today I received my _CloverPi_ which I backed on
Kickstarter (no other affiliation), a “back panel to enable 4 RaspberryPis to
be clustered”.

Finally I can earn my “ _My other computer is a Beowulf Cluster_ ” bumper
sticker from what feels like aeons ago (courtesy of the now-defunct
_ThinkGeek.com_ ).

~~~
frandroid
> now-defunct ThinkGeek.com

Noooo!! They had the best toys.

Unfortunately their shipping to Canada made everything double the price for
me...

------
mdoms
Am I understanding that he is storing web data on the SD cards of the Pis? Is
that normal? SD storage is notoriously unreliable, especially in write-heavy
applications.

~~~
megous
Brand SD cards are quite reliable. If you configure your SW to avoid sources
of needless constant write churn (eg. like postgresql stats collector), and
use good power supply, they will last years.

~~~
ikaruswill
You're absolutely right. I've running these SD cards for about 2 years now. I
use log2ram to stream logs to a ramdisk and then sync that at regular
intervals to disk, to extend the life of the SD cards.

------
fock
"This build is for you if you:"

ahem, for 600 dollars I get a fine computer which can host all the kubernets
and docker I ever want in a nice formfactor...

I mean, it's cool and all, but effectively utterly useless except for the
blinking lights.

~~~
dumb1224
Very cool work! I wonder the same, for a 'cluster' I'm expecting performance
and large memory for example for scientific computing. Other than that I
couldn't think of any other use cases. If anyone could enlighten me? Cheers

~~~
noir_lord
They built a multi-thousand pi cluster to test the SMP and configuration stuff
before running it on a super computer.

Virtualising thousands of CPUs on a single machine is still not trivial.

Never mind simulating real world network issues.

[https://www.bitscope.com/blog/FM/?p=GF19A](https://www.bitscope.com/blog/FM/?p=GF19A)

~~~
fock
yes, if you want to optimize a worst-case MPI-cluster, then a Pi (4) might be
optimal for you (because sadly, 4 measly ARM cores with 100MBit/s is a some
magnitudes removed from 100 cores and 100GBit/s Infiniband). But then you can
also use a stack of old desktops, which is cheaper and you can just throw in a
standard image and everything (including CUDA and MKL) can work.

~~~
noir_lord
> But then you can also use a stack of old desktops, which is cheaper

If electricity and cooling is free.

------
moonbug
There's a new carrier board with integrated switching for Pi clustering:
[https://turingpi.com/](https://turingpi.com/)

------
alibarber
I wonder if there is something to be said for not using Ethernet in something
like this? Perhaps it is possible to make a kind of IP over GPIO interface, or
attach them all directly to a USB bus? It just seems to me like RasPi -> USB
-> Ethernet (for less than 1m) -> USB -> RasPi has a layer or two more than
necessary (and I believe the on-board ethernets on RasPi are USB interfaces to
the chip too). Could be an interesting project...

~~~
Accujack
It would be educational.

I'd say the most interesting way to approach this at the moment would be to
use RasPi 4s and link them via the PCI express bus.

None of this stuff is more than educational, though. I admire this build
because it's almost a rite of passage for nerds to build a cluster, but when
you're all done and the cool wears off you've basically got a pile of obsolete
hardware for which you need to find a home.

------
mvanbaak
People wanting to torture themselves with k8s at home ... I still dont
understand why.

~~~
aaronax
Imagine that the jobs you are interested in want to see some k8s experience.
Either you teach it to yourself at home or you justify implementing it at your
current job (resume driven development basically).

~~~
mvanbaak
I have played around and learned the basics of docker etc without never ever
wanting to run my home stuff on it. Learning and experimenting is not the same
as using it as the primary tool to run your 'production' services at home.

------
bullen
If you want to get more performance out of a cluster like this you can look at
my distributed HTTP app. server + JSON database solution:
[https://github.com/tinspin/rupy](https://github.com/tinspin/rupy)

It's actually usable for live services on RPi 2 and on RPi 4 it beats Intel on
float ops./joules at 100% utilization.

And if you use multicast UDP you can make these behave cooperatively to scale
horizontally beyond what's possible with a vertical solution at a fraction of
the purchase/maintenance cost.

I think we are peaking at 2 Gflops/watt for CPU, so if you need silence
(passive cooling) and small size this is as good as it will get, probably
ever:
[http://talk.binarytask.com/task?id=8015986770003767235](http://talk.binarytask.com/task?id=8015986770003767235)

I'm building a RPi 4 cluster with these heatsinks.

------
daneel_w
Neat setup. An alternative would perhaps be a Pine64 Clusterboard with 7
SOPINE modules: [https://store.pine64.org/product/clusterboard-
with-7-sopine-...](https://store.pine64.org/product/clusterboard-
with-7-sopine-compute-module-slots/)

~~~
pengaru
I cannot recommend doing business with pine64, their customer service is
nonexistent. If you cancel your order, or need to get ahold of someone @
sales@pine64.org for any exception, in my experience you may as well shout at
the nearest wall.

\- Someone out $165 spent on a cancelled pinephone order, no phone and no
refund, complete radio silence.

~~~
johntash
> \- Someone out $165 spent on a cancelled pinephone order, no phone and no
> refund, complete radio silence.

Hopefully you've asked your bank/creditcard to reverse the charge?

~~~
pengaru
Up until hours ago my pending PayPal claim seemed lost in a different form of
silent purgatory, but the refund appears to have finally been authorized as of
tonight.

I ordered the phone on 5/12, cancelled the order and filed a PayPal "item not
received" case on 6/26, escalated the case to a formal PayPal claim on 7/16
(under pressure from PayPal's "close or escalate" automated ultimatum), and as
of now it appears the claim finally got refunded on 8/12.

Not a _single_ email response has ever been received from numerous attempts to
contact sales@pine64.org throughout this total waste of time and frustration.
They ignored requests for updates before I cancelled, seemingly ignored the
cancellation, filed an "item shipped" update in the PayPal case on 7/8 despite
my cancelling the order on 6/26.

YMMV

------
cptskippy
> The maximum number of ports on a consumer-grade network switch was 8, enough
> for 7 nodes and a cable to the router.

Are there laws in Singapore that delineate between consumer and business grade
networking equipment based on port numbers or something?

Here in Yankey Land you can pickup a consumer grade 16-port unmanaged gigabit
switch for $50 at the local Wally World. The going rate on switch ports is
$3-4 per port.

~~~
rtkwe
It looks like it was more a question of price in their market. In the post he
mentions a 16 port switch but it's 10x the price of an 8 port. I bet it's a
question of there being fewer people in the Singapore market looking for an >8
port switch on the low end of the market.

~~~
cptskippy
The switches he references in his post are from two entirely different product
verticals. One is from the unmanaged vertical, the other is from the managed
PoE vertical. Not only is he ignoring that a 16 port switch exists in the
unmanaged vertical, he's ignored an entire vertical of managed switches
without PoE.

TP-Link makes a $40 unmanaged 16-port gigabit switch and a $80 managed 16-port
gigabit switch.

He's created a False dilemma/dichotomy by pretending that other options don't
exist. This is like saying "I'd like a Toyota automobile larger than the Yaris
but I can't afford the Lexus LS".

~~~
rtkwe
It's entirely possible that those options weren't readily available in the
local store he's using. I'm not familiar with Singapore in particular but a
lot of Asian countries don't have the huge electronics box stores that are
common in the US and Europe. I've seen quite a few videos where the best
electronic source the person had was what would be a small sized store in a US
mall (20 feet wide, one isle with a counter running down one side for
example). Online shopping does exist but shipping times and cost can also be
pretty large.

~~~
cptskippy
> I'm not familiar with Singapore

Singapore is a very small (~250 sq mile) country, so it would take less than
30 minutes to drive across.

It's very wealthy.

It has, almost dead center in the middle, an electronics mall called Sim Lim
Tower that's like 10-20 stories tall.

They most certainly have options.

~~~
paines
And you forgot the superb food and it's variety.

~~~
cptskippy
I hope you're not referring to durian. :)

I was just pointing out characteristics that would lend themselves toward
sourcing a network switch (e.g. accessible, wealthy, giant freaking
electronics malls).

------
WaxProlix
Seems down, here's a google cache of it
[https://webcache.googleusercontent.com/search?q=cache:lIePhG...](https://webcache.googleusercontent.com/search?q=cache:lIePhGOXF9AJ:https://ikarus.sg/how-
i-built-kraken/+&cd=1&hl=en&ct=clnk&gl=us)

~~~
kanobo
Thanks, I wonder if the site was hosted on the Pi Cluster?

~~~
ikaruswill
Yes the site is indeed hosted on the cluster.

~~~
justinclift
Seems to be down again now. Guess it'll be spotty until traffic settles down.

Maybe the traffic will provide an interesting exercise in optimisation?

eg move from wordpress to static content or something, maybe add some CDN
pieces, then whatever else turns out to be needed?

------
tobmlt
Nice! I tried this out a while back as well, though I did nothing nearly as
snazzy.

Southampton had instructions here, I believe it was:
[http://www.southampton.ac.uk/~sjc/raspberrypi/](http://www.southampton.ac.uk/~sjc/raspberrypi/)

(Maybe still useful if you want to do without docker?)

------
throwawaynothx
Personally I would not use pi's for this. Massive waste of money, the
experience is worth it but other than that..

~~~
ikaruswill
What would you use then?

~~~
rumanator
600$ buys you a considerable amount of time in a cloud vendor. For example you
can have a 1vCPU instance in Hetzner for 3€/month, which means 600$ can buy
you about a year's worth of 20 instances running 24/7.

If you don't need 99.9999 availability of your cluster and are only interested
in firing up the cluster from time to time then those same 600$ can buy you a
few years of cluster time

~~~
wahern
The 3€/month is a single thread (not even a core), and it's not even
dedicated. The lowest dedicated instance is 2 vCPU (1 core) for 23€/month with
8GB or RAM. That's 276€/year.

8 of those instances is equivalent to an 8 core (16 thread) chip with 64GB of
memory. Using, e.g., Supermicro AMD EPYC 3251 + parts, you could build an
entire server for less than 1/2 that price. You could have _twice_ the compute
for the same price the first year. Servers easily last 3 years minimum, so
Hetzner has a 6x markup at the low end.

Admittedly, building your own server doesn't include transit, but transit is
_cheap_. Assuming you have a fast home connection (a big if for many people),
the hard part is access to static IP addresses. There are dedicated services
that lease IPs over a VPN tunnel, or you can setup your own VPN into, e.g.,
Hetzner or Vultr using a handful of the cheapest instances. (Not sure about
Hetzner, but Vultr let's you lease and attach 2 additional IPs to the same
instance for $2/month/IPv4 address.)

Now, the above setup is probably far too much trouble for most people, and it
has many downsides[1]. But the point is that if we're _just_ talking about CPU
time, not even Hetzner or some other cheap VPS provider can come close to the
cost of running your own equipment.

[1] Power and cooling are two downsides, but the 3251 has a TDP of 50W.
Granted, it doesn't have the highest clock speeds, but there's a reason cloud
providers don't normally advertise Ghz. Anyhow, rather than build 2x 3251's
you could build a single 16 core rig using a higher-clocked and newer EPYC or
Xeon for about the same money (definitely at or under 2200€ all-in).

~~~
rumanator
> 8 of those instances is equivalent to an 8 core (16 thread) chip with 64GB
> of memory. Using, e.g., Supermicro AMD EPYC 3251 + parts, you could build an
> entire server for less than 1/2 that price.

Your choice of comparisons suggest you completely missed the whole point. If
the author wanted a desktop I'm sure he would buy one for 600$ instead of
spending that money to put together an underperforming cluster that's entirely
unusable as a desktop.

If instead you are interested in gaining experience managing a cluster that
runs distributed systems then yes you would do something like putting together
a raspberry pi cluster.

Yet, if you seek to have the same experience (or arguably far better) without
spendig a hefty sum as capex and having access to more computational resources
for a negligible opex then cloud providers such as Hetzner are clearly a far
better option.

------
klickitat
Wow, its wonderful.

------
john_alan
Until Raspberry Pi has native hardware crypto instructions (AES) it’s useless
for real work.

------
daniel-s
I think that there are a lot of use cases of Pis for which online cloud
providers might be a better option.

Where does it become a better idea to just rent a few instances in
Azure/EC2/Google Cloud?

