
Building Raspberry Pi Systems with Yocto - codetrotter
https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-Yocto.html
======
tkfu
Just a quick tip, if you are interested in Yocto, and especially Yocto on the
Raspberry Pi: there's a layer called meta-updater[1], originally developed for
Automotive Grade Linux, that lets you add over-the-air updates to your Yocto-
built systems, and it supports Raspberry Pi. It uses OSTree[2] to give you
atomic updates and nice, small update sizes. (The whole filesystem is a
content-addressed object store that's hard-linked into the actual directory
structure at boot time.)

There's also a guide[3] for doing this using HERE's servers for delivering
your updates. It's free to use, but if you prefer a free-as-in-speech
solution, you can either ignore the update client and run the updates yourself
from a bare OSTree repo, or run the open-source community edition[4] of HERE's
server software.

(Full disclosure: I work for HERE, and wrote the quickstart guide[3] I'm
linking.)

[1] [https://github.com/advancedtelematic/meta-
updater/](https://github.com/advancedtelematic/meta-updater/)

[2]
[https://ostree.readthedocs.io/en/latest/](https://ostree.readthedocs.io/en/latest/)

[3] [https://docs.atsgarage.com/quickstarts/raspberry-
pi.html](https://docs.atsgarage.com/quickstarts/raspberry-pi.html)

[4] [https://github.com/advancedtelematic/ota-community-
edition/](https://github.com/advancedtelematic/ota-community-edition/)

~~~
codetrotter
With regards to image update management, personally what I wish to do for my
Yocto image is to have it be:

\- Small enough that it fits into RAM.

\- Fetched over the network on boot, PXE-style.

\- Configured to not write any files to the greatest extent possible, and
where writing files can not be avoided one would use tmpfs.

With the above three criteria satisfied, the following is achieved:

a) We never write to an SD card so we avoid SD card corruption issues.

b) Since the whole system image is sitting in RAM, we _could_ do without NFS,
Samba or similar either.

We do need to persist _some_ data, but that data is coming from our
application and is going to be stored in a database on a server on the local
network.

I am currently working on developing the application that our image will run,
but I have dug up some preliminary information relating to what I said above:

\- Read-only Raspberry Pi [1]. Applies to Raspbian but certainly provides a
starting point for doing the same thing with Yocto.

\- Poky NFS Root [2]. Like I said, I don't need the NFS root part if the image
can fit in RAM, but the portions that deal with gPXE might certainly be
relevant.

\- The State of Netbooting Raspberry Pis (December 2017) [3]. "Net-booting
works well on the Raspberry Pi B Model 3 without an SD Card." So actually we
won't be needing gPXE.

\- Network Boot Your Raspberry Pi [4]. Referenced by [3] and provides the
steps needed to configure Raspberry Pi 3 for network booting without an SD-
card.

I am not completely decided upon whether or not to avoid NFS yet though.
Because we never persist any files on the root file system anyway, serving
said fs over NFS can be done in read-only mode, and we would put each revision
of the corresponding root file system in a uniquely named top-level directory
so that a client running the current version image has one place that it gets
the files it expects from and the next version of the image which might expect
other files or differing layout would get its files from another directory.
Old root file system directories which are no longer in use would be removed
so this scheme doesn't require much in terms of storage space.

It should be noted that NFS has some security issues [5], but our deployment
is going to be running on an isolated network exclusive to the hardware that
we are providing, so this concern is already resolved. Said isolated network
will connect our embedded Raspberry Pis, a server (TFTP, database and NFS),
and a tablet configured and provided by us. The tablet provides a graphical
user interface that our customers will use to interact with the system.

[1]: [https://learn.adafruit.com/read-only-raspberry-
pi/](https://learn.adafruit.com/read-only-raspberry-pi/)

[2]:
[https://wiki.yoctoproject.org/wiki/Poky_NFS_Root](https://wiki.yoctoproject.org/wiki/Poky_NFS_Root)

[3]: [https://blog.alexellis.io/the-state-of-netbooting-
raspberry-...](https://blog.alexellis.io/the-state-of-netbooting-raspberry-
pi/)

[4]:
[https://www.raspberrypi.org/documentation/hardware/raspberry...](https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md)

[5]: [https://www.tldp.org/HOWTO/NFS-
HOWTO/security.html](https://www.tldp.org/HOWTO/NFS-HOWTO/security.html)

~~~
kingosticks
Be wary of some packages that expect their log folder to exist at
/var/log/some_package else they won't start, I can't remember which of nginix
or apache has that issue. That adafruit script is very simple and there are
more robust setups out there, albeit not as well documented.

Netbooting isn't 100% reliable and last time I checked there were certain
network switches/hubs that caused issues. Maybe that's a non-issue if you are
also providing the network infrastructure.

Lastly, remember that the more RAM you consume for your filesystem/logs etc,
the less available for our actual application. That becomes more of a problem
when you start also assigning a large chunk of that RAM to the GPU.

~~~
codetrotter
> Netbooting isn't 100% reliable and last time I checked there were certain
> network switches/hubs that caused issues. Maybe that's a non-issue if you
> are also providing the network infrastructure.

We will be providing the network infrastructure indeed, but being aware of
this will certainly be valuable if we face those sorts of problems, so thanks
for that :)

Do you happen to know of any specific switches or hubs that would work
reliably in this context?

~~~
kingosticks
I'm sorry I don't remember what I have but [1] suggests a Netgear GS108. That
link also explains a couple of the issues and they may have even been fixed in
more recent bootcode versions. The 3B+ has a later version (and also has
netboot enabled by default so you don't need to mess with
program_usb_boot_mode).

[1] [https://www.raspberrypi.org/blog/pi-3-booting-part-ii-
ethern...](https://www.raspberrypi.org/blog/pi-3-booting-part-ii-ethernet-all-
the-awesome/)

------
xyproto
The Yocto package format supports a mix of inheriting BitBake package files,
Bash and Python. It's over-engineered and unpleasant. Values can easily be
defined 3 or 4 layers away from the file you are looking at, either in one of
the many configuration files (that can be overridden by other configuration
files), or in one of the files the package is inheriting from.

Add clunky and unintuitive tools to the mix, and Yocto can easily be an
unpleasant experience, at least for teams over a certain size.

It does make switching between architectures really easy and sweet, though,
but that's about it.

~~~
codetrotter
Having just found Yocto yesterday and built my first custom image that I made
by extending the layer in the OP, I am not yet "married to" Yocto.

Yocto initially appeared to be exactly what I was looking for. When someone
else raised concerns about Yocto ITT I somewhat brushed them off, mainly
because they were talking about managing the layers on a headless system. I
used Yocto from the command-line on my workstation, so a) I am not on a
headless system and b) even if I was doing this on a headless server,
everything I have done so far would be done exactly the same way.

But, since then, multiple other people have raised concerns about Yocto. I
also talked to one of the mods on /r/raspberry_pi and they said something
similar. I looked through some past discussions about Yocto on /r/raspberry_pi
using the search feature on Reddit, and found a couple of other similar
experiences.

After looking through what has been said, I find that these concerns resonate
with me rather deeply. They are all pointing to a single sentiment that seems
to be rooted in similar real-world experiences; Yocto makes things complicated
in a way that with time will make managing things needlessly complex and which
will result in a lot of wasted time.

I was still a bit inclined to think though that perhaps these experiences were
had long enough ago that Yocto has since improved, but after a little bit of
further thought I realized that fundamentally Yocto is engineered in the
fashion that you and others are pointing to as being a fundamental flaw of the
system.

I am going to try Buildroot instead. Same guy that wrote the post I shared in
the OP also made a post about using Buildroot.
[https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-
Buildro...](https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-
Buildroot.html)

------
TickleSteve
Although Yocto gets more publicity, I always prefer BuildRoot myself.

Yocto's 'layer' abstraction seems overly complicated and difficult to use
(especially in a headless situation where you can't use a nice GUI tool to
manage them).

BuildRoot is much more like the Kernel Configuration... Just choose your
options from a hierarchical menu.

Much easier IMO.

([https://buildroot.org/](https://buildroot.org/))

Also, building a minimal system is much easier in BuildRoot than in Yocto...
just deselect the options rather than trying to remove layers and the contents
of layers.

~~~
codetrotter
I found creating a layer of my own that extended the one in the OP with a
couple of things extra was “easy” (in terms of what I would expect from the HN
crowd. For a beginner it would not be easy), but I will agree that removing
things will probably be more complicated.

Here are a couple of photos of my Raspberry Pi 3 with said Yocto image running
some graphical applications without X11 (nor Wayland):

gstreamer videotestsrc ! glimagesink
[https://i.imgur.com/dPQqgRX.jpg](https://i.imgur.com/dPQqgRX.jpg)

tspress [https://i.imgur.com/CLx1RrD.jpg](https://i.imgur.com/CLx1RrD.jpg)

~~~
codetrotter
In the time since I wrote the above comment I've decided to give Buildroot a
shot. See
[https://news.ycombinator.com/item?id=18093599](https://news.ycombinator.com/item?id=18093599)

~~~
TickleSteve
Glad you tried it with an open mind....

For background, I'm a softie with over twenty years experience with embedded,
desktop and server systems from microcontrollers to multi-core DSPs so I've a
fair amount of experience in this area.

The last time I tried Yocto was while building a mesh router for an IoT
product. When building real products, removing all cruft and only having a
minimal system is very important from many viewpoints and this is crucially
very difficult in Yocto. I gave up and went the BuildRoot route.

I just find that Yocto has overcomplicated something that is essentially very
simple. BuildRoot on the other hand works exactly how I expect and therefore
I'm much more productive using it.

~~~
codetrotter
I just finished building my first BuildRoot image and here are my thoughts.

The Yocto build system is sneaky because when it is running it shows the total
number of tasks and the number of tasks that have executed, so it feels a bit
more comfortable in that way than BuildRoot.

In this respect, BuildRoot vs Yocto feels a bit like running a big compilation
of ports on FreeBSD using make directly vs running the compilation with
poudriere. The latter has a web user interface that lets you track the
progress a little bit. The web interface of poudriere looks like so:
[https://blog.shatow.net/static/poudriere-31.png](https://blog.shatow.net/static/poudriere-31.png)

However, like I said, Yocto is sneaky. Because while the above mentioned
aspect might seem like a big plus, the end result of using BuildRoot was very
impressive.

I based my BuildRoot config on [https://jumpnowtek.com/rpi/Raspberry-Pi-
Systems-with-Buildro...](https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-
Buildroot.html) with some additions and removing some things I didn't need.

I noticed also that there was an option "remount root filesystem read-write
during boot" that could be disabled. Having a read-only image is something I
was planning on doing already, so I gave it a shot. And it works!

Thanks to BuildRoot, I have a lean 251 MB (lean in the context of what I am
doing, I am aware that some Linux images for some routers etc are just a
handful of megabytes) that can run with a read-only rootfs and which does
everything that Yocto image I had did, and it took much less time to compile
because like you said, it was easy to customize it to only have what I needed
basically.

My workstation has 32 GB of RAM and an 8 core, 16 threads AMD Ryzen 7 1700
Prosessor. Building the Yocto image took in excess of 3 hours, close to 4
hours I think. Building the BuildRoot image to 80 minutes, because the
BuildRoot image, unlike the Yocto image did not have a lot of extra cruft,
exactly because like you said, with BuildRoot it is easy to remove what you
don't need.

Now one thing I did notice because I had overlooked some of the configuration
options for one of the packages was that changing the options of a package
does not cause BuildRoot to regard it as necessary to rebuild the package.
Apparently this is by design [1]. I think that is a tiny bit disappointing,
but whatever. Rebuilding individual packages is quite fast anyway. And
hopefully if I need to do full rebuilds of the whole thing then ccache, which
I made sure to enable before running the first build, will shave some time off
the build. But I don't even think full rebuilds will be necessary all that
often.

All in all I am glad that you suggested I try BuildRoot, and I am glad that I
listened. I will be sticking with BuildRoot. Thank you :)

[1]:
[https://github.com/buildroot/buildroot/blob/master/docs/manu...](https://github.com/buildroot/buildroot/blob/master/docs/manual/rebuilding-
packages.txt)

------
kingosticks
I looked into switching to yocto a while back for a rpi project I work on.
However, one thing that put me off is the end user's inability to install
regular debian (raspbian) packages as they will inevitably expect to. But I
think that people have used yocto as a kind of docker hypervisor, the openhab
project used to do something like that before moving to HassOS. Does anyone
know if you can include a minimal raspbian-based docker image in your yocto
system which users could then apt-get install whatever extra packages they
want? Does that make sense?

~~~
huling0
I recently ran into that issue - I have a nice fine-tuned yocto image and now
a customer wants to "log in via ssh and install a custom raspbian package". I
was thinking of the docker solution as well.

~~~
v_lisivka
"We have small and fine-tuned ghetto, but customers don't want that. What to
do?"

I have similar problem: we have consumer device on top of imx6. I used iMX6
SoloLite evaluation board kit (1GB of memory) with Fedora for arm as
development prototype. For me, installing a package is not a problem at all:
"dnf install package". I planed to use Raspbian or Fedberry on actual device,
but our management blindly switched to Yocto, without comparing of
alternatives. To install package on Yocto, we need to create our own recipe in
our own layer,then rebuild Yocto, fix bugs and repeat, and then deliver
update. We spent man-year to achieve same result as we already had with
Fedora. We seriously lag behind schedule because two developers (and two
consultants) are working on developing of our own custom OS instead of working
on our application. Moreover, instead of single system for development/testing
and production (Fedora or Debian on developer workstation, in Docker for CI,
on actual device) we now have mix of two systems. I created configuration
script with about fifty options to be able to compile for host and for Yocto
at same time (including workaround for number of bugs in Yocto). Enormous
development time was sink into ground because of this schizophrenia.

~~~
zubairlk
ouch. That sucks.

You could run Fedora in a container on your Yocto base using resinOS.

I wrote more about this in another comment.
[https://news.ycombinator.com/item?id=18093336](https://news.ycombinator.com/item?id=18093336)

~~~
v_lisivka
I already have OS. In my case, I just use chroot script to access full Fedora
distro with all development tools on Yocto. If I will be able to replace
Yocto, I will replace it with major distro, like Debian, Ubuntu, CentOS, or
Fedora, not by yet another toy distro with thousands of packages per
maintainer.

------
cm2187
Unrelated but when I saw the title I thought it would be about yoctopuce,
which are usb environment sensors that are easy to script for in python or
other languages. I used them to create a few weather station loggers using
raspberry pis.

[http://www.yoctopuce.com](http://www.yoctopuce.com)

~~~
stevekemp
Those seem like fun products; though the hardware-tinkerer in me is thinking
they could be home-made via ESP8266 hardware very very easily.

~~~
cm2187
Most likely, but they are a good solution for someone like me who is
comfortable with software, but doesn’t really want to invest time in the
electronic / hardware. They are a bit pricey though (but I guess it’s not a
big market).

