Hacker News new | past | comments | ask | show | jobs | submit login
Gokrazy: A pure-Go userland for Raspberry Pi 3 appliances (gokrazy.org)
366 points by type0 on Mar 25, 2018 | hide | past | web | favorite | 87 comments

This is great! I think it could have uses far beyond the Raspberry Pi. As someone who's worked in the consumer electronics industry for years, I think we desperately need to rethink our approach to Linux-based electronics with respect to both the userland and the primary application. A legacy-free framework written in a safe language could go a long way to improving our currently lousy reputation with respect to security and reliability. (Yes, fresh code will have fresh bugs even in a safe language -- but it's a start!) I'm also keeping my eyes open for similar ideas using Rust.

Shameless self-promotion: also take a look at https://github.com/u-root/u-root :)

(The gokrazy and u-root people all work at Google, I just contacted them to see if we can help each other. Not a competing project.)

No, what sigjuice said is right. The name was derived off u-boot, though, and u-root can be used (and is in use) with LinuxBoot.

Doesn't look like it. U-boot is a boot loader for PowerPC/ARM/MIPS/<other CPUs> boards. A typical Linux-based embedded system would consist of U-Boot, the Linux Kernel and a filesystem. u-root would be one of the many ways to construct this filesystem.

Exactly, the security in legacy systems is non-existent. I've been pretty pleased moving systems to the nerves-project [1] (which like GoKrazy has an A/B root file-system) to replace deprecated and ancient firmware systems. One of those setups required a Windows XP machine to flash the particular MCU that's obsolete now, a huge pain. It was almost impossible to update / patch.

Nerves is based on a Buildroot core but most of the system is built with Elixir and BEAM/OTP. It's fantastic having a high level language built for robustness and reasonable security. Still, I have dreams of porting BEAM to Redox OS one day [2] and replacing the C userland-bits with Rust-bits. Especially if you could represent the micro-kernel services as actors. That would be amazing!

1: https://nerves-project.org/ 2: https://github.com/redox-os/redox

> As someone who's worked in the consumer electronics industry for years, I think we desperately need to rethink our approach to Linux-based electronics with respect to both the userland and the primary application.

Google has created a stack called NERF (Non-Extensible Reduced Firmware) for this:


There's a presentation about it on youtube:


A large portion of the recent widely exploited NAT box / IP camera vulnerabilities and malware are exploiting logic bugs, backdoors and default passwods - instead of memory safety bugs. I agree memory safe systems are needed, but nontechnical changes are also needed to fix the widespread IoT vendor negligence.


gokrazy generates strong passwords by default and listens only on private IP addresses by default to reduce the attack surface.

Yeah, the idea is great.

There are a few Basic, Pascal, Oberon and Java (not Android related) options available, but only a minority cares for them.

Still, big enough to keep those vendors doing business.

Having a minimal bare metal runtime for Go would also be nice, but it is a gigantic effort to make it portable across all types of hardware.

Maybe you should look at TockOS[1].

[1]: https://www.tockos.org/

Super nice. Glad to see A/B partiton updating scheme, that is the way to go on appliance / embedded device.

Want something like this for one of the languages that I know... Node.js, Python or Rust. I'd include C++11, though for this kind of deployment method I think one needs packages easily available in a way separate from Linux distro (since there is none).

The A/B updating scheme is a massive waste of space in most implementations... In many hardware devices, flash storage space for the OS is actually one of the most expensive components, and one of the main reasons that many devices run little hard-to-program embedded OS's rather than linux or something else high level.

Instead, a filesystem like btrfs should be used which can give the appearance of multiple versions, yet only use actual storage space where versions differ.

I'd prefer the root filesystem to be something using a FUSE-like layer which can store format specific binary diff/patches between versions, and then transparently apply (and cache) them on access.

Taking it a step further, I wish desktop linux distros had a package-manager-fs, which is a readonly virtual filesystem which displayed all the files installed by all packages, but the underlying real filesystem only contained all the original compressed packages and a cache of recently used files.

Sure it is quite inefficient. Both in storage and in transmission of updates.

But it is very simple and robust. What one tests pre rollout is identical to what will go on production devices, down to the byte. Each device with same version is identical. It can be cryptographically verified easily. Automatic rollback can be performed on update fail, including self-checks.

This approach of shipping a custom userland without any of the generic Linux userland should drastically bring down the size.

Transmission of updates can be quite small: For the info-beamer hosted (https://info-beamer.com/hosted) OS, we also do A/B booting. Updates are implemented using zsync.

The initial installation of the OS is simply done by unzipping a carefully crafted [1] install.zip file to the Raspberry Pi SD card. By exploiting zsync features, it's quite efficient to reconstruct the original install.zip from the unpacked files. Once that's done, all future updates also use zsync and only fetch changes to the locally stored 'install.zip' file. After updating this file we then unpacked everything again to the next A/B partition, so the exact state of the system is always known. Updates are quite small that way. It also helps that the complete OS is only ~37MB compressed at the moment.

[1] The root file system is squashfs and is stored uncompressed in the zip file to avoid double compression and allows zsync to use the squashfs blocksize to find a minimal set of changes. Similarly the initramfs.gz is compressed --rsyncable and also also stored uncompressed in the zip file.

I strongly agree regarding the simplicity!

A minimal gokrazy root file system is 16MB in size. On a 8GB SD card, I certainly am willing to sacrifice another 16MB to get A/B updates :).

> The motivation is that @stapelberg spends way more time on C software and their various issues than he would like. Hence, he is going Go-only where feasible.

This is so cool! I’m very happy to learn an effort like this exists, and looking forward to perusing the code for it... in Go (a language I really enjoy reading).

It is really interesting that Gokrazy uses an upstream kernel (with a couple of minor patches).


Why is that interesting? Just that they're not having to carry special patches for the RPi3 or what?

Yes. I have only tried Raspbian and I'm not used to getting RPi kernels from kernel.org

E.g. Fedora has for years been using for RPi the same close-to-mainline kernel source that is used for other ARM boards and x86-64 and any other supported hw platform.

Even the raspberry Pi Zero and Zero W are supported in the latest kernels, and those were the latest Pi boards until the B+ came out.

Thanks! That’s good to know. Perhaps getting gokrazy going on the Pi Zeroes will not be a huge task. 32-bit userland probably already works on the 64-bit Pi 3 kernel.

Does anyone know of a good tool for building custom PI distributions based on the Raspbian / Raspbian Lite or other distributions that does not involve forking the official image builder and hacking it up? My ideal tool would allow me to do this with a Dockerfile and just export the image after the configuration is complete.

http://resin.io is close to this, but also handles updates for you.

My answer is most likely out of context and off-topic but have you checked the Nerves Project[0]?

Apologies if this is not helpful.

[0] https://nerves-project.org

I'm having a really hard time figuring out what problem this project solves, can anyone explain?

They say it right in the header:

"For a long time, we were unhappy with having to care about security issues and Linux distribution maintenance on our various Raspberry Pis."

Yep - for home automation style stuff (which the author seems to have built it for - see the showcase section), having a full Linux distro is actually a real pain. I've taken to building my own stuff with the ESP series of microcontrollers, home made PCBs, etc - once it's built - it just keeps chugging along, unlike my original stuff I built atop RaspberryPis with Linux.

I've done a lot of this sort of this as well. The ESP8266 is so great for simple on/off switches and sensors.

But-- there are so many applications for having a full IoT Linux distribution that I can't believe this isn't already a solved problem. I'm imagining an ultra-stable distro that is easy to deploy to a headless machine, handles security updates, handles flaky wi-fi issues, tests the SD card (I've had more than a few burn out on me), and automatically logs to the cloud for easy debugging.

In other words: Why am I still maintaining my RasPis as if they are apache servers from the 1990s?

ResinOS does all that and more.

It's container-based. Works super-well on a Pi3.

Oh, this is awesome. Can't wait to try it. Thank you!

Linuxkit is another option, slightly different approach.

Thanks. Where would alpine linux fit into this all. It seems like a good candidate for embedded setups.

You could run whatever you like in an alpine container on ResinOS

Android-things solves this problem at the scale of embedded linux, and mbed solves this problem at the microcontroller level.

I'm a wary of Android only in that I'm so used to the Raspian ecosystem, but I will check it out. Thanks for the recommend!

People have been building stuff like this. Often for firewall appliances (eg SCC Sidewinder, Immunix Linux, genua w/ OpenBSD). They usually do one of these: (a) get big under new management that doesnt care about either security or the little people (enterprise focus); (b) sell out to company like that; (c) go bankrupt cuz market values assurance of security the least.

Sad but true. There's still a few companies selling microkernel-based solutions in defense and mobile with Linux user-lands. They'd OEM license their stacks for a new set of products aimed to do this. They cost a fortune. Whoever pays it wont get it back, though, cuz folks dont usually pay for security in that volume. So, nobody is doing it for small players for now.

What is it about Linux that doesn't keep your stuff chugging along?

You have to keep it up to date because of all the security issues in other apps that are installed.

If the system only contains the Linux kernel and your software you only have to worry about your security issues (unavoidable) and Linux security issues (also difficult to avoid - too much work to write a new kernel too, plus remote Linux vulnerabilities are quite rare).

So why not just create a stripped down Linux distribution instead of reinventing the wheel and creating a new userland from scratch (with probably a few brand new and undiscovered security issues already)?

I tried: I have used debootstrap to create minimal Debian images in the past.

A big part of the motivation for gokrazy is to use a safe, high-level programming language.

This allows us to re-implement the userland with small amounts of code, see e.g. the NTP client at https://github.com/gokrazy/gokrazy/blob/master/cmd/ntp/ntp.g...

Replacing the implementation language eliminates whole classes of attacks, while stripping down an existing Linux distribution can only go so far. Plus, it’s not clear to me how one would maintain such a stripped-down distribution in a way that isn’t a lot of effort.

I still find this effort odd. You already have a well tested and widely used userland, even if it is written in a non memory safe language.

The easiest solution to me seems to be to grab the tools you need from the existing userland, instead of reimplementing them in Go. Currently you have reimplemented some of the userland, and packaged it up into a distro. So you have to maintain a stripped down distribution and the tools that you reimplemented. If you just took existing tools you would only have to do the latter. Am I missing something here?

The userland in question is an NTP, a DHCP client and the init system / web interface. It really isn’t much code. Implementing these in Go reduces the number of moving parts, both in terms of libraries involved in the project and more importantly in terms of programming languages involved.

Have a look at the git history to see how little maintenance is required in practice. I have barely had to touch the code over the last year of the project. Just incorporating new upstream versions of a typical Linux userland would have been more work.

Also note that “distribution” is a rather big word for the few files we’re talking about:

/mnt/gokrazy-root % find -type f ./cacerts ./gokr-pw.txt ./gokrazy/dhcp ./gokrazy/ntp ./gokrazy/init ./user/hello ./localtim ./hostname

If my reasoning doesn’t convince you, so be it. Not every project has to make sense to everybody :). I wanted to do a pure-Go userland, and I’m happy with the amount of work this results in and the properties that come out of this decision (easily automated updates, massively reduced attack surface).

So it’s like it’s own OS almost? What’s a userland?

It's everything the kernel doesn't do. In general, that means the surface with which you interact as a user. So if you're used to using a terminal, your SSH connection, your Bash session, are all userland programs (that use kernel features to do things). You can think of it as kernel is a framework like Rails and userland stuff is developers using and expending the interfaces... sort of.

When your computer runs headless, the init/process tree system is userland. Background (cron) jobs are userland, daemons are userland. Etc.

So it's a distribution ?

... that will have security issues, require updates, distribution maintenance, and all that stuff they claim is unacceptable ?

It’s not a full-blown distribution; it doesn’t distribute third-party software. Instead, it allows application authors to build their own distribution (at which point “appliance” is a more fitting word than “distribution”).

Updates are easy to fully automate with gokrazy (and I’ve been using that mechanism for a year at this point) because assembling a new image and installing it over the network is just one simple command. New kernel and firmware versions are provided automatically once they are verified to boot on real hardware.

There is no maintenance aspect in the actual installations (i.e. you don’t need to SSH into them, ran commands to update, etc.) — the root file system is read-only, and an update overwrites it completely. Unless the applications you want to use with gokrazy introduce state, gokrazy defaults to being stateless.

Hope that clarifies things, let me know if you have further questions.

Forgive me if this sounds dumb, but if the image just contains the firmware + kernel + a light layer for managing processes, do this mean it doesn't include stuff like SSHD?

That’s correct: there is no SSH daemon by default. You can optionally include https://github.com/gokrazy/breakglass, though, which is an SSH daemon that can be started on demand to facilitate interactive debugging.

Userland is the part of the OS that is not the kernel. It is the part you actually interact with.


Yes, it's pretty much the OS except for the kernel.

Why is Go considered safe?

Go is memory-safe. See also https://en.wikipedia.org/wiki/Memory_safety

Go is not memory safe (though is certainly safer than C++). Various Go constructs like interfaces and slices are implemented via non-atomic multiword structs, and data races may result in invalid values leading to memory corruption, etc. See https://blog.stalkr.net/2015/04/golang-data-races-to-break-m...

Yours is a stricter and not-commonly-used definition of "memory safe", which typically means "in the C sense" i.e. pointer arithmetic and bounds checking. Go is certainly memory safe by the typical definition.

I don't see why it's a different definition, though. What's he's saying is that it's very easy to break Go's bounds checking with race conditions, and produce memory corruption. This isn't true of languages traditionally considered as memory safe - e.g. Java or Python.

A crash on out of bounds will always happen, unless unsafe package is being used to subvert memory accesses.

This is only true if you can prove that data races will never cause any bounds checks to be wrong. The blog post linked upthread suggests otherwise, and even appears to provide a counter-example to your claim.

As mentioned in that blog post

> You'll notice my address() trick to get the address of a function or variable that I used in my previous exploit. We're relying on package fmt and its %p format specifier which is implemented by using the unsafe package, so we don't need to use it ourselves.

So? The code they wrote doesn't use `unsafe`. So presumably, you need to modify your claim to this then I guess?

> A crash on out of bounds will always happen, unless unsafe package is being used to subvert memory accesses, or if you've used something that exposes details that only the unsafe package exposes, such as fmt's %p specifier.

I mean, you are like, the ultimate pedant, so I figured you'd uniquely appreciate this.

You can't just say, "use unsafe or anything that uses unsafe," because it's turtles all the way down. You've got to draw that line somewhere. If the %p specifier really is required for an exploit and there's nothing else that exposes that (sans unsafe), then that's great news! But is that the whole story?

unsafe code is not required or used. The exploit constructs a confused interface value, whose vtable refers to a different type than is what is stored. The result is that Go treats an integer as a function and jumps to it.

The unsafe address() is used to get the address of a function to store in the int, to demonstrate the potential for shellcode injection. But you could just as easily assign pp := 12345 and produce a SIGSEGV or SIGILL, which would not be possible if Go were memory safe.

On the contrary, if Go was unsafe that would just corrupt memory and the program would just carry on without anyone noticing the memory was corrupted until it was too late, in another completely different stack.

The exploit was only possible thanks to using the unsafe package, even if indirectly.

It is exactly because Go is memory safe that a SIGSEGV or SIGILL is produced the exact moment the memory gets corrupted and not 2 hours later in a complete different stack trace.

What's the typical definition? With data races, you can access array values out-of-bounds, which seems like the most fundamental thing you'd want protection from in a memory safe language.

You can't access data out of bounds. Go will panic with a runtime exceptions if you do. Memory safety doesn't mean "memory IO errors never happen" it means "if a memory IO error happens, you'll know"

If you have an out of bound in C, your program keeps running in 9/10 cases. In Go you'll cause a panic in 10/10 cases which you can either handle and propagate as error or let the program crash entirely, the memory will be in a defined state if you choose to recover (your array might be garbage but you won't have written data into random memory belonging to other threads)

But with Go you CAN access data out of bounds. See the "Exploiting the slice type" section where writing to a slice modifies a function pointer which happens to be stored after the slice. It's an OoB write which does not panic.

an OoB write under a data race in a program that won't occur like this in the wild. The conclusion section of the post you mean contains that statement too.

The %p formatting used to gain access to the pointer of a variable uses the unsafe package, the formatting itself is incredibly rarely use to begin with.

I don't see anybody doing any security-related exploits doing that unless you explicitly write for it.

Go won't do out of bounds, edges cases do happen. You can find the same stuff on Java and Rust (yes, I had OoB in Rust, though on a baremetal environment without OS, even Rust ain't safe.)

The typical definition is memory corruption caused by out of bounds writes, stack corruption, use after free.

On safe languages an out-of-bounds will cause a crash on the access point, while on unsafe languages it will silently corrupt memory.

Is there any language not safer than C++?

Many. I like bashing C++ as much as the next dev, but modern C++ has nice type checks, interesting types/wrappers which help with common patterns (shared, unique ptrs, etc), and the compilers learned some great tricks which allow you to run memory safety checks yourself. (Asan and others)

Also multithreading gets some improvements which will take more of the manual sync details away from the business logic. (http://www.modernescpp.com/index.php/multithreading-in-c-17-...)

It's still delivered attached to a foot-gun, but you could do worse.

Are there not many statically typed, compiled languages that ship a runtime with a garbage collector? Why is Go a good choice here?

Why is this choice better than a language that provides memory safety using linear types so you don't even need a runtime or garbage collector?

Are there not many statically typed, compiled languages that ship a runtime with a garbage collector?

Not really a huge number, no. Java is probably the most obvious, if you count that as compiled. Maybe Haskell or D? I suppose even Swift of Objective C if you include RC in the definition of GC.

I would say the focus shouldn't be entirely on GC, however. Safety as a whole is the most important factor, and it's possible to be say without GC, as you point out. I reckon Go is actually quite an attractive language for this sort of application regardless – quite safe, quite lightweight, good libraries.

You can read more about why I like Go at https://michael.stapelberg.de/posts/2017-08-19-golang_favori....

Of course I used my favorite programming language for building gokrazy :).

I've only dipped into Go recently (and not for any production code) but your article very nicely summarises my impressions with it so far. I've found a kind of fluid pleasure in writing Go that wouldn't be apparent from a bullet list of its (lack of) features. It really allows my attention to fall mainly on the problem domain in a way few other languages afford.

I can't resist while here thanking you for i3. I've been a mac user recently, but will likely be back to linux in the next year or so. Using i3 again will be probably be the greatest pleasure involved in that return ;)

Hey, the creator of i3!

I love your stuff! I enjoyed your episode on Go Time as well!


Glad you like my work; thanks for the kind words!

I've added gokrazy to my list of things to play with over the weekend (when I theoretically have time, hah). One question I'd ask in context of the other poster, can I easily run a Lua VM and tie it into the gocrazy env so that a Lua app would appear like a go app?

It might be possible to do this, but certainly not easily.

gokrazy’s unit of application is a Go package, so you’d need to write a Go program which then executes the Lua VM. I don’t know what dependencies Lua has — if it cannot be compiled statically, I’d recommend against using Lua in this context.

There are a number of languages that meet the memory safety criteria, and they each have tradeoffs that may make them a good choice for a certain range of use cases. This project happens to be based on Go, but similar projects in other languages would be interesting as well.

Could/should one use this to build a touch screen user interface for an RPi as part of a kiosk based application?

I really like Go, and suspect a Go app would beat the pants off anything powered by Electron (performance wise).

I’d love to see this, but note that you’ll need to use a pure-Go graphical toolkit (or implement all graphics yourself), and likely interface the touchscreen yourself, too. If you’re not comfortable with this, it might be better to use something else.

Maybe a Flutter port would make sense here?

This is a repost from last year:


"Reposts" are fine. Better to formulate it as previous discussion.

Looks great. I'd love to use it. How is the GPIO/PWM support?

Have a look at https://periph.io/.

In case any of periph.io’s features require changes to the kernel configuration, I’m happy to change it accordingly.

Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact