
Build your own bare-metal ARM cluster - alexellisuk
https://blog.alexellis.io/build-your-own-bare-metal-arm-cluster/
======
heavyset_go
One caveat when deciding on an ARM platform to adopt is the fact that someone
needs to maintain a custom Linux branch for each individual SoC configuration.

That means you'll want to choose a platform that's very well supported by its
parent company if you don't want to get stuck on an obsolete kernel in a year.

Hardkernel, who design and produce popular ARM SoCs, chose to produce x86
based platforms in the future for this reason.

~~~
wtracy
While the article talks about not being able to run x86 binaries on Arm, this
is the real drawback to the platform.

I really wish we would see a standard Arm kernel instruction set come along
and get adopted by every manufacturer who isn't doing something unusual enough
to justify a custom architecture.

~~~
jabl
The ARMv8 server platforms follow a common standard, SBSA. Which is
essentially taking the most-hated parts of the x86 platform, ACPI and UEFI,
but once you suppress the gag reflex it does get the job done, and you have a
mostly x86-like experience that you just boot a generic armv8 distro and it
all autodetects stuff on the fly and it all just works.

Haven't heard of any of the single-board platforms adopting it, though.

------
syntaxing
This is going to be a stupid question...but what does clustering do? Does it
increase the amount of "cores" in a system? What about the RAM? Or is the
clustering software a task manager that allocates different boards as needed
depending on the the task load?

~~~
jacquesm
There are no stupid questions. Clustering allows you to do more jobs at once
than you could do with a single machine of the same power. It allows for
redundancy if you set things up in such a way that if one hardware components
fails another takes over. It can allow for better isolation between processes
for security reasons than what a process isolated from another process on the
same machine can give you. And many other reasons besides, so clustering is a
technique that can give you many different benefits.

But...

There is a price to pay. Clustering isn't easy, convenient or without
overhead. As soon as you cross the boundaries of what a single machine can
give everything suddenly has a price, either in terms of latency, throughput,
complexity or loss of control. Clustering software tries to hide some of this
but under the hood these will always be there.

Here are some good hints about what limitations you are operating under no
matter how good your distributed environment:

[https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...](https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing)

[https://en.wikipedia.org/wiki/Amdahl%27s_law](https://en.wikipedia.org/wiki/Amdahl%27s_law)

~~~
syntaxing
Thank you for the explanation!

------
dang
We took "Show HN" out of the title above. Please read the rules:
[https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html).

------
genpfault
> You can't run binaries designed for a normal PC on an ARM board

...doesn't Debian ship some multiarch/QEMU/binfmt plumbing to make this work?
Binfmt to taste the binary, see that it's i386/amd64, and then launch QEMU
(and associated ARM multiarch libs) with user-mode emulation?

Slow as dirt I'm sure but it'll work.

~~~
RealityVoid
Surely that does not qualify as "running the binaries" if you emulate another
arch! I mean if you want to be nitpicky, yes, you're running them, but that is
not what most people think when saying that.

~~~
qubex
...but don’t people/users speak of “running Java apps” all the time?

------
leowoo91
Only practical reason I can recommend this would be either you'd like to
experiment with cluster tools or connection outages (e.g. seeing how system
tolerates when you unplug an ethernet cable).

Also, while OpenStack/K8 could seem to be a reason having something like this,
I discovered it is a bit overkill for home needs. For that part, I'd go for a
small server instead and manage it manually (no shame).

------
noir-york
I was looking for something similar earlier this year and ended up with
SOPINE's cluster board. It only supports 7 ARM modules at a time but you can
always stack multiple boards. Its also inexpensive. The SOPINE board I have
has worked very well and recommend it.

[https://www.pine64.org/?product_cat=clusterboard](https://www.pine64.org/?product_cat=clusterboard)

------
ArtWomb
Also check out Rancher. Lite weight K8 for distributed IoT. Scaling to 1000s
of ARM nodes, clusters morph into incredibly smart supercomputing grids ;)

[https://thenewstack.io/how-low-can-kubernetes-go-rancher-
arm...](https://thenewstack.io/how-low-can-kubernetes-go-rancher-arm-team-to-
find-out/)

~~~
darren0
(Disclaimer: I'm from Rancher Labs)

Unfortunately, none of the tech referenced in this articled is available yet
and should be released end of February. There is a more interesting part that
isn't referenced in this article and the is that we'll have two variants of
k8s for ARM. One is RKE which is our CNCF certified k8s distribution. The
other will be based off k3s (github.com/ibuildthecloud/k3s) which is an
optimized k8s distribution for small clusters (smaller memory, easier setup,
etc). I expected the second will be far more suited for something like this
project.

------
flyinghamster
It's too bad that OpenMosix withered on the vine. Back when it was a thing, I
found it really handy to fire up clusterKNOPPIX on a bunch of idle PCs for
large compile jobs.

I guess with today's multi-core CPUs, it wasn't thought to be worthwhile, but
it could still come in handy for resource-constrained systems like the Pi.

~~~
guenthert
I share your sentiment regarding Mosix. I don't think however, that clustering
SBCs is of any value beyond the learning experience. Even a moderately
powerful laptop is much faster than a *Pi, e.g. a 2013 Chromebook Pixel is
about 16 times faster than a Banana Pi, when measured using `openssl speed`.

~~~
wybiral
It depends when you're trying to do with them, but the network speed of the Pi
is definitely a bottleneck for me. If it weren't for that, the throughput
could definitely be used for some cool stuff even with modest CPU/RAM.

My cluster: [https://www.youtube.com/watch?v=lZDUe-
_mzHc](https://www.youtube.com/watch?v=lZDUe-_mzHc)

~~~
nickpsecurity
Neat video. What cool stuff would you use it for?

------
sannee
Note that the Raspberry Pi can boot over PXE without an SD card these days,
which should make for a much more reliable setup.

~~~
floatboth
I found that to be extremely unreliable, I had to reset the Pi a few times
before it actually made TFTP requests.

A tiny SDcard with just U-Boot on it, configured to netboot, has been actually
reliable.

~~~
corndoge
Unless you are running 3B+ pis, the firmware code for pxe booting has several
bugs. Also, Pis have issues with retransmission of DHCP requests on low end
network hardware that doesn't set link up fast enough to hear the Pis intial
dhcp request. Almost all of these bugs are fixed one way or another at this
point though, the github issue tracker for raspberrypi/piserver and friends
has lots of info.

~~~
floatboth
Yeah, (a while ago) I also tried just having the firmware update on the SD
card, it was a bit more reliable but still far from 100%…

------
jacquesm
Neat! I built something like this long ago
([https://clustercompute.com/](https://clustercompute.com/)) to digitize a
very large number of CDs for a Dutch broadcasting corporation. If I were to
tackle that project today this is probably how I would do it.

------
gnulinux
Wouldn't doing something like this with FPGAs be ultimately cheaper and more
efficient. Sure, it will have a big overhead to learn, build and set up your
FPGAs specifically for your problem, but that seems like a constant time
overhead?

~~~
wtracy
That assumes that the nature of the work load doesn't change over time.

------
st3fan
"In my experience the 64-bit OSes all run much slower on the RPi3/\+ than
Raspbian."

This should be backed up by some numbers. I have seen plenty of benchmarks now
that pretty much show that many workloads on ARM64 are a lot faster.

------
dangoor
Given what the article says about k8s not working so well in the RPi's 1GB of
RAM, is there another cheap ARM SOC that has more RAM available that could
work similarly?

~~~
floatboth
RK3328: Pine64's ROCK64 comes in 2GB and 4GB of RAM

(also the RK3399 based ROCKPro64 is even more awesome, bigger form factor, but
two A72 cores for actually good performance! Oh and some other manufacturers
even made RK3399 based RPi-sized boards!)

~~~
hipboi
ROCK Pi 4, rk3399 based, comes in 1G/2G/4G ram, starting from 39usd.

[http://rockpi.org](http://rockpi.org)

------
eismcc
Definitely will look at K8s for my [http://parallac.org](http://parallac.org)
clusters.

------
achntrl
I'm really grateful to Alex Ellis for his gist[0] to install kubernetes on
RPis. I built my own 4 nodes cluster following this gist. It's been great to
get my hands dirty with kubernetes

That being said, I had to buy an Asus Tinkerboard with 2GB RAM to serve as the
cluster master because 1GB RAM was a bit short. My master node went below
100MB of free RAM and pod evictions (of kube-system pods) started to happen.
If I were to do it again, I think I would use only Tinkerboards just for the
convenience of having the same OS/hardware on each node.

[0]
[https://gist.github.com/alexellis/fdbc90de7691a1b9edb545c17d...](https://gist.github.com/alexellis/fdbc90de7691a1b9edb545c17da2d975)

