
The state of netbooting Raspberry Pis - alexellisuk
https://blog.alexellis.io/the-state-of-netbooting-raspberry-pi/
======
geerlingguy
Not to mention the onboard LAN is over the USB connection, but is topped off
at 100 Mbps, so no high-throughput uses will work over the network.

Alternative SBCs have started adding onboard eMMC, USB 3.0, and GigE; while
they’re still not desktop PCs, having faster IO makes tinkering so much less
annoying (for many applications). I hope if anything, the next Pi has Gigabit.

~~~
mschuster91
Given that many people use RPi as base for home entertainment systems... I'd
prefer GigE and one or two SATA ports.

~~~
api
There are boards with all that.

The fastest low-cost networking board right now (that I can find) is the
espressobin, built around the Marvell Armada chipset. It's got three full-
duplex GigE ports that can actually run as such as well as pci-E and SATA.
Unfortunately it's oriented toward routers and stuff so it lacks HDMI, so no
good for home entertainment.

~~~
mschuster91
> There are boards with all that.

Problem with these: I can never be sure what level of acceleration for video
playback is supported - and in what quality. The Pi is pretty rock solid in
that department. Also, the Pi has by far the biggest community and extensive
documentation, and I can rely on there still being Pis available in years,
compared to the Chinese outfits where no one knows how long they'll stay in
market.

~~~
heegemcgee
Concur. I've played with the ODroid C2, Beaglebone Black, and the three "full
flavor" Raspberry Pi models. The RasPis have, far and away, the best
supportability. The ODroids are great _when they work_ \- love that GigE, but
they aren't maintaining an OS distribution with the same care and man-power
that Raspbian is getting.

------
mirceal
this is a bit misleading in that it's not necessarily about netbooting (which
per linked article just works).

As someone who's played with raspberry pi's on "real" networks I can tell you
that netbooting one is not as reliable as you think. It may work on a small
home network, but once you have 50+ devices tftp-ing something will show it's
limits.

The firmware that netboots is also not as reliable as one would think. If it
fails it just hangs. A proper netboot/PXE client will retry forever.

The approach I followed was to build a minimum ramdisk image which I've placed
on the SD card. When that starts, it creates a in-memory file system and
downloads the actual image to this file system. The SD card is in read-only
mode after that.

~~~
ChuckMcM
I don't disagree, it reads a bit like submarine marketing for the clustering
hardware. But the difficulty in net booting is for the Pi1 or Pi2. On the Pi 3
it "just works."

That said, net booting used to be a thing. When I joined Sun Microsystems in
1986 it was all about the 'diskless workstation.' At the time was the brand
new Sun 3/50.

~~~
mirceal
point was that it does not just work on a pi 3 if you put it in a real-life
environment.

also, net booting is still a thing, but not for the diskless scenario :)

------
znpy
The real issue with raspberry pis is that ethernet is actually on the usb2
controller, so it's capped at 80mbps afaik, and it's also shared with other
usb storage.

I hope raspberry pi 4 has gigabit ethernet and netbooting becomes a first
class function.

~~~
blacksmith_tb
And/or a usb3 controller to tie networking to...

------
revelation
Netbooting is one of these problems that turns out to be much more difficult
than it's inherent complexity warrants. Setting up DHCP and TFTP doesn't cause
a full psychosis yet, but then we enter the "no win zone" of NFS and
filesystems. One of the persistent frustrations with any sort of Linux
filesystem is that their makers feel you would only ever want to use them with
a full-blown Linux kernel, too. No libext, the only userland tool you get is a
bunch of 10000 LoC C monsters that an autoconf behemoth ensures can only be
built on a modern Linux glibc system. When at the end of the day we are
talking about an abstraction that should work perfectly fine with just read(),
write() and seek().

My ideal "netbooting" setup is a faux SD card that translates block IO to UDP
packets on a GBit link and has a little software on my computer
reading/writing from a binary blob. I can still mount it if theres something
mountable in the blob. Being a SD card it solves the other big problem with
netbooting, which is that it's not transparent. The bootloader needs to know
it's netbooting. The kernel needs to have support for NFS. init needs to know
you are netbooting to mount the differently-named FS. And then when a service
decides on startup to reset the network connection (hello Android), you're
still fucked.

~~~
kardianos
If you haven't seen it, try out the all-in-one tool:
[https://github.com/google/netboot](https://github.com/google/netboot) go get
go.universe.tf/netboot/cmd/pixiecore

It is completly self contained. The user just provides it with a linux kernel
image and the initrd image.

~~~
revelation
That's a very good start, if you install a DHCP server on Linux they usually
assume it's because you trying to run a federated thousand machine network,
not just to stub some initial config values, and so the complexity matches
that expectation..

But the biggest pain point remains the whole NFS mess. I'm not sure you can
even do something like SELinux over NFS, and even if you could, you still need
to configure the right _host computer permissions and attributes_ only for
stuff to match on the _netbooting machine_.

~~~
kardianos
Netboot is most effective when the system is a minimal system and is
configuration driven, see tinycore linux or CoreOS container linux with
ignition.

NFS is a mess. So design your solutions to not need it. I see no reason why
the pixieboot solution isn't something capable of scaling to thousands of
machines on a network.

~~~
revelation
I mean, we are in agreement. There are just systems like mobile SoC running
platforms like Android where you desperately want some sort of network booting
scheme for development because you are constantly rebuilding parts of the
software, where the storage options for the part are highly constrained, slow
and unreliable (like the RPi) and the performance and architecture necessitate
cross-building.

------
Improvotter
I'm pretty curious about using RPis for a Kubernetes Cluster. Is there a
benefit to using a RPi cluster compared to a simple server or is this purely
for research purposes?

~~~
bendews
Depending on the amount of nodes you’re wanting, it usually is better to just
grab an Intel NUC and virtualise. Much smaller than a RPi cluster and much
more powerful.

