
BitScope: 3000-core Raspberry Pi cluster computer - NicoJuicy
https://www.raspberrypi.org/magpi/bitscope-3000-core-raspberry-pi-cluster-computer/
======
ballen
Another neat use-case for this hardware is scale testing of systems management
tooling, eg. provisioning, configuration management. HPC centers are looking
at the possibility of having to manage 100k+ nodes in a single cluster.

Additional links:

[http://cluster.bitscope.com](http://cluster.bitscope.com)

[http://www.lanl.gov/discover/news-release-
archive/2017/Novem...](http://www.lanl.gov/discover/news-release-
archive/2017/November/1113-raspberry-pi.php)

My favorite form factor to go build a Pi-like Cluster is something like this:
[http://www.friendlyarm.com/index.php?route=product/product&p...](http://www.friendlyarm.com/index.php?route=product/product&path=69&product_id=211).

\- Gigabit Ethernet MAC on-board, Pine64's SOPINE and Raspberry Pi Compute
Module both require per node networking components to be on the carrier board.

\- Basic headers used for connecting to a carrier board

\- Carrier board "only" needs an embedded switch and 5v power. However, I've
not yet come across an embedded switch yet that has a non-blocking ratio of
1GbE ports to 10/25GbE uplinks.

\- Carrier board should probably be mini-ITX or other standard form factor to
fit in existing chassis. Form factor and embedded switch options are going to
limit the number of nodes per carrier board.

~~~
anfractuosity
I'm just looking at the friendly arm website at some of the other NanoPi's,
which I'd not heard of before, I'm quite tempted by the Gbps ethernet. How do
you find the software/OS support, are they pretty reliable, with longish
uptimes?

~~~
pero
i have some nanopi neo(1)s and they're rock solid - my rule of thumb with arm
boards is that support is good enough if it has an armbian _debian_ (vs just
armbian ubuntu) release

~~~
anfractuosity
Cheers I might look at getting one then, as I had problems with my Pi's
network connectivity being slow compared to a laptop's using the same ethernet
over power line adapter.

~~~
pero
no idea about PoE on either board but if i/o is a concern you might want to
take a look at the $25 rock64 with gbe and usb3 - i/o on my neo1 maxes out
somewhere just below 20mb/s but my rock64 has no problem doing 80mb/s with
some ancient spinny disks and some armbian devs have done ~200mb/s with ssds
iirc

os support on the rock64 is not quite there yet however but given the
features@pricepoint i expect it to catch up

------
bradfa
It's a neat way to teach people how to program distributed systems at this
scale but I don't understand why they wouldn't use the Raspberry Pi Compute
Moudle instead of normal Raspberry Pis. Using the Compute Modules would, I'd
think, make for a much easier to integrate system, even possibly allowing
higher density of processors per unit volume.

~~~
danielvf
Raspberry Pi Compute Modules don't have ethernet built in - that would require
quite a non-standard networking system, wouldn't it?

~~~
srcmap
Any comments on using Pi Zero as base line module?

    
    
      Supposedly only $5 per PCB from  
           https://www.adafruit.com/product/2885
    
    
      Power and network can go thru usb. (usbnet)

~~~
squeaky-clean
Like stan_rogers says, you're limited to buying them one at a time from pretty
much any retailer. And it's been this way for years, so it doesn't seem like
it will change anytime soon.

You can find more on eBay, but then they're usually priced $15-20 each and it
would still be difficult to buy them in the hundreds.

------
logronoide
35U? That’s a full rack. The cluster looks like a 8U server case.

~~~
mnw21cam
One of the comments mentions that it is a 6U server case. So presumably there
are five of them, with 150 rPi3 units in each case.

~~~
jacquesm
Correct. 144 nodes + 6 spares in one box.

------
xori
I bet the IO throughput sucks. Is the ethernet bus still shared with the
usb/sd card bus in the rpi3?

~~~
pero
yes

------
jacquesm
Now, to port _lisp and C_ to it.

The interconnect will be the most interesting part of this cluster.

~~~
pjmlp
You forgot the _star_ reference. :)

~~~
jacquesm
No I didn't, HN mangled the text to italics and I did not spot it. Too late to
edit. That's why lisp and C are in italics :) How do you make an asterisk in
HN anyway?

~~~
pjmlp
Like this __* (3 for 1)

~~~
jacquesm
Ah, that's the trick. I tried two and that didn't work. Thank you!

------
pero
waste of (taxpayer) money using the (overpriced) rpi's gimped (no aes, usb3,
gbe) soc

the rpi value prop is the community support but that would mean that this
cluster is running a 32bit os - so what really is the point of using these
instead of something smaller and cheaper or same size and more powerful for
the same money

~~~
technofiend
The Raspberry Pi 3 model B uses the Broadcom BCM2837 64-bit A53 ARM processor
and has been supported since Fedora 25.

Here's the link to the AArch64 server distribution:

[https://dl.fedoraproject.org/pub/fedora-
secondary/releases/2...](https://dl.fedoraproject.org/pub/fedora-
secondary/releases/27/Server/aarch64/images/Fedora-
Server-27-1.6.aarch64.raw.xz)

Having said that I agree the Pi is overdue for a refresh; it needs gigabit
ethernet and usb 3 at a minimum but faster interfaces would be great. I think
people execute these projects because the Pi is a great reference architecture
that can be bought at scale and that has been proven by the large user
community.

~~~
jandrese
I'd love it if the storage and Ethernet weren't hanging off of the same USB.
Or on USB at all. The USB is a perennial bottleneck on the RPi.

There are a lot of boards that implement it correctly in hardware, but then
make a hash out of the driver support with binary blob drivers that are fixed
to a particular kernel version and crash from time to time.

It's probably not technically feasible currently, but I'd love to see a board
where all of the hardware is open (even the 3D acceleration!) and already
mainlined into the kernel so you could just install whatever distro you want
on it and available at a price point under $50. I'm really tired of "you need
to dump this proprietary binary blob into the graphics chip before the rest of
the board can even start to boot." Why is it taking so long to come up with a
universal boot solution, something that could be integrated into GRUB so you
don't need to program magic offsets into the bootloader to make it work? PC
hardware manufacturers more or less solved this problem 30 years ago, and I'm
not taking "but the hardware is so specialized that you can't do it" as an
answer anymore. The SBC world is absolutely crammed with stovepipes for no
good reason.

