
Show HN: Coinboot – a framework for diskless computing - Wotm8
https://github.com/frzb/coinboot
======
frzb
Hello, I am the creator of Coinboot and I will try to answer some of your
questions.

Coinboot was initially created with only one use case in mind: To run the
nodes of GPU-crypto-mining farms diskless. Because you don't need disks for
mining. But these farms have only cheap commodity 1Gbit/s network hardware so
the filesizes to transfer need to be as small as possible.

Being stateless is also a benefit for operations - all software +
configurations are loaded in the RAM during boot, so you always have the most
recent version in place and don't have to use a configuration management.
There is also a plugin system which makes it easy to extend the functionality
with further software. Just put the updated plugin archives in the plugin
directory on the Coinboot server and reboot your nodes.

Coinboot is still in Beta phase but has been running for months on hundreds of
GPU nodes.

My plan is to make the first stable release by the end of the year.

Centralised logging is already on the roadmap, also load-balancing the file
download via local Bit-Torrent during the boot phase, and a Web-UI plus many
other things. Ubuntu was chosen as base because the proprietary GPU driver for
OpenCL has only support for Ubuntu and RHEL. RHEL was no option. Coinboot does
use TFTP only for the NBP, all other file transfers are happening over HTTPS.
If your hardware is using UEFI HTTP, then all file transfers are happening
over HTTP. Boot time in the production environment is less then 15s.

Running diskless is done since decades and seems to be quite common in HPC -
ages before everything got *-less. ;-) There was no implementation which
fitted the requirements of the use case or was not outdated. Coinboot is also
not using NFS - just because there is only cheap commodity 1Gbit/s network
hardware in place. The whole root file system is put onto a TMPFS. After boot
~500MB of the RAM are used for the root filesystem. CoreOS is very impressive
but there image size ~380MB was way to big. Coinboot is using currently ~130MB
for the kernel + initramfs archives and the GPU driver Coinboot plugin.

~~~
blauditore
Can you maybe in a few words summarize (for a non-expert on OS') what's the
"secret sauce" in this, i.e. how it differs from running a classic linux
distro without permanent storage media?

~~~
frzb
A standard Ubuntu installation plus the mentioned proprietary GPU-Driver
package are using ~4GB of the available file system capacity. Thats way to
much to fit into a ramdisk while having enough RAM left for OS. Most of the
nodes in the GPU mining farms I know have only 4GB of RAM at all. So there is
some optimisation needed to shrink the filesystem usage by getting rid of the
things that are not necessary on the filesystem to run the OS. Source files,
man pages, caches and so on. At the end of this optimisations you have a
system which is much more lightweight but does still have the same well known
usability. Having a more lightweight storage footprint than a "classic linux
distro" is the key to be usable for network booting and diskless operation at
scale on hundreds of nodes. Coinboot also has a highly customized Early
Userspace ([https://github.com/frzb/coinboot-
debirf/blob/427b5fb8f3b9313...](https://github.com/frzb/coinboot-
debirf/blob/427b5fb8f3b93139c301cc84332b11b365a76669/debirf#L234)) so after
extracting the root fs onto the ramdisk, additional file archives a.k.a.
Coinboot Plugins can be downloaded and extracted on the root file system. This
is how the plugin system of Coinboot works, which enables the extension of the
core system without repackaging the Initramfs archive.

------
andrewstuart
I didn't really understand exactly how it works from the linked page.

~~~
mikestaszel
Same. It's really confusing. How do I boot a machine exactly? What's Docker
used for? Is this like a locally hosted
[https://netboot.xyz/](https://netboot.xyz/) ?

------
CyberDildonics
When I use the internet am I doing computerless computing?

~~~
ahartmetz
Yeah, I was going to say... the "-less" terms are pretty stupid. "Serverless"
is interesting for the no implied context work packets approach that is also
interesting in e.g. highly threaded game engines, but is not about not using
servers to run the work. Who comes up with this shit?

~~~
pjmlp
Marketing departments.

------
frzb
Docker is used to provide a working setup (DHCP-, PXE-, Webserver, the
configuration and needed files) in the most easy way.

------
mariopt
What would be a good use case for diskless computing? RAM is very expensive
even when compared with top of line NVME SSD drives.

~~~
ExBritNStuff
Any kind of thin client/zero client usage, where local storage is not required
or allowed. Many years ago I used a similar solution for a finance company.
The users just needed to get access to a browser to connect to a specific web
service.

~~~
Immortalin
What was the implementation like? A unikernel spun up on demand?

~~~
juliangoldsmith
You could easily do that with Linux. You'd just need to boot via PXE, and use
an NFS root. You might even be able to just use an initramfs, and not even
have a mounted root.

------
haolez
I’ve used CoreOS tools in the past to achieve the same goals. It’s really
useful once it works! I hope CoreOS lives on after the Red Hat and IBM
acquisitions.

~~~
evol262
You can do this with pretty much any tool you want. I also did it about 10
years ago with livecd-iso-to-pxeboot (and running from the generated
initramfs)

Primary obstacles in 2018 are primarily the lackluster support for ro root
filesystems. CoreOS/Alpine are fine. Writable memory filesystems are fine.

CoreOS is not even a little bit dead inside Red Hat after the acquisition. If
you're not paying attention to the news from Fedora, CoreOS is alive and well,
which says a lot about how it will go downstream.

The IBM acquisition is not final yet.

~~~
haolez
Yep. I’ve used FAI in the past as well (I think that was the name).

I just really like the CoreOS ecosystem and I hope that it keeps following the
direction that they set in the beginning.

