
Embedded developers need an open source OTA software updater without the lock-in - ralphmender
http://embedded-computing.com/guest-blogs/embedded-developers-need-an-open-source-over-the-air-software-updater-without-the-lock-in/
======
protomok
Actually the Jeep Cherokee was running QNX [1] which is not 'Linux-
based'...it's a totally different OS initially released many years before
Linux.

Frankly I'm not sure one can link the Cherokee hack to lack of a secure remote
update mechanism. Of course signed images and secure updates are key elements
of a secure embedded system. But the fatal flaw was IMHO a very unfortunate
decision to route their IPC (DBUS) over IP with no authentication. A system is
only as strong as the weakest link!

[1] Excellent white paper link from authors of the Cherokee Hack:
[http://illmatics.com/Remote%20Car%20Hacking.pdf#page=20](http://illmatics.com/Remote%20Car%20Hacking.pdf#page=20)

------
karmicthreat
I have been digging around in this space searching for a good solution where I
might have connectivity, but I also might not. So I need to be able to update
off a USB as well. Mender seems to have a story for this. Resin.io told me to
have people hook a hotspot up. Our units can be in the middle of factories in
the middle of nowhere. Its not always possible to be connected.

Tried resin.io and while it had good ideas, it was difficult to integrate. It
is also completely dependent on their hosted management system. Which can be
good, but is another hurdle to get past. Mostly because we would have started
at greater than 25 devices and would have rapidly hit 200. So its hard to get
our owner behind that up front.

Mender has been much easier to get going. Its dependencies are far less and
without the docker containers easier to get going initially. Docker I think
will have its day, but host<->container configuration for peripherals is kind
of clunky.

Also Mender can be self hosted so that makes it far less friction for me to
get going with.

~~~
alexandros
Hey there -- resin.io founder here. Resin is indeed starting with our hosted
service, but we have committed to, and are making decent progress towards,
building a completely open source platform.

If you need pure OTA, mender is a great solution. Resin is about containers
first and foremost, with full-blown host updates being a lot more rare, since
the host is a lot simpler.

Different things work for different projects, hope yours goes well!

~~~
ralphmender
Hi there, this is Ralph with Mender.io. Thanks for the kind words! We've met a
few people from Resin at conferences and they are very nice folks. We very
much appreciate your technology's approach and look forward to seeing your
open source platform.

------
evancordell
The issues described in this article sound like exactly what the Uptane[0]
project was designed to address, which is also open source.

[0] [https://uptane.github.io/](https://uptane.github.io/)

~~~
dane-pgp
Yup, and Uptane is supported by ATS, which is also open source and provides a
full management system:

[https://advancedtelematic.com/en/press-releases/ats-
presents...](https://advancedtelematic.com/en/press-releases/ats-presents-
open-saas-developer-platform-ats-garage-for-connected-cars.html)

------
thom_nic
In addition to the other great projects mentioned, I'll throw out fwup [1]
(A/B partition update, used by nerves.io) and snapcraft [2] (containerized
packages, not whole-OS.)

For myself, I'm comfortable with some form of A/B partition update (assuming
the problem of distribution and verification is solved elsewhere) however at
the very least you still have to patch or script u-boot (or your bootloader of
choice) to make it smart enough to try booting the primary partition and
_also_ have some sort of canary file it can check on the next boot to validate
that the partition booted successfully and e.g. made it all the way to the
target runlevel. And if that check fails fall back to the secondary partition
or some recovery image.

Embedded firmware updates - especially for M2M devices that are not
necessarily user-facing - are sort of like changing the engine of your car
while driving down the highway. It's a very tricky thing to get right. Plus if
something goes wrong, it's not like you can just SSH into your server or kill
that cloud instance and start a new one :)

[1] [https://github.com/fhunleth/fwup](https://github.com/fhunleth/fwup) [2]
[https://snapcraft.io/](https://snapcraft.io/)

~~~
abhirag
"Embedded firmware updates - especially for M2M devices that are not
necessarily user-facing - are sort of like changing the engine of your car
while driving down the highway. It's a very tricky thing to get right. Plus if
something goes wrong, it's not like you can just SSH into your server or kill
that cloud instance and start a new one :)" \-- I worked on marking 'embedded
firmware installation' process better in a previous job of mine and you
described the problems involved perfectly. There was a part of the process in
which communicating with the device and attaching debuggers was a major pain
and I had to rely on extensive tracing to keep an eye on the process. I
personally believe that the better way would be to emulate what NixOS
does([https://nixos.org/~eelco/pubs/nixos-jfp-
final.pdf](https://nixos.org/~eelco/pubs/nixos-jfp-final.pdf)). Painless
rollbacks and reliable updates would help embedded development a lot and
treating the state of the device as immutable is a great way of doing that.

~~~
thom_nic
Yup I did not mention nix but from what I understand it is very similar in
spirit to snapcraft.

Unfortunately creating an embedded linux distro leaves one with few package
manager/ updater options - Debian (apt/deb, although Sid has a package for
snap), debootstrap (none) or Yocto (ipk). Seems NixOS for embedded [1] has not
seen much love.

[1]
[https://nixos.org/wiki/Embedded_NixOS](https://nixos.org/wiki/Embedded_NixOS)

------
metaphor
>> without the lock-in

Anyone know more precisely what he's talking about? The article doesn't
specify.

>> The V850 controller software was built to only listen to the CAN bus and
not write commands to it.

If the hardware was intently designed for sniff only, you'd think a simple
uni-directional buffer between the V850 and CAN bus would have mitigated this
risk in one fell swoop. Maybe it's hinde sight bias, but directly interfacing
a complex, OTA-enabled embedded accessory to a bus used for safety-critical
control sounds about as sensible as strapping a Raspberry Pi to a flight
control computer's 1553 bus and thinking it's all gravy because the former's
software is designed to "only listen".

That being said, I understood from a former Delphi engineer who designed
engine control modules that there's a pretty strict and expensive approval
process for electronics in the automobile industry, and reliability
predictions are taken quite seriously. I suspect even if an engineer were to
design in a true read-only interface, it'd probably get nerfed during design
review as a consequence of lacking traceability to a hard requirement.

~~~
ralphmender
Thanks for the comment.

>>> without the lock-in

>> Anyone know more precisely what he's talking about? The article doesn't
specify.

Apologies for the confusion, yes I had meant vendor lock-in. That is usually a
blocker for many embedded projects starting out. This is why we've created a
open source project with both client and management server licensed under the
permissive Apache 2.0.

If you are interested, we are always looking at ways to improve our project
and if you'd like to participate in a one hour user test, we conduct those
over Google Hangouts:

[https://mender.io/become-a-tester](https://mender.io/become-a-tester)

------
ratliffchrisb
I spent a good amount of time working with Brillo when Google was first
pushing it. The thing I liked most about working with it was the system they
had for OTA updates. Even in beta they had support for staged rollouts with
some analytics on the rollout. There was lockin with the solution as updates
used Google's cloud and was signed by both your and their key, but it was a
really robust system considering how new it was.

I really hope the project gets more attention. Not only was the firmware
update system nice, but having the promis of having the core OS managed and
only having to worry about your application was nice as well.

------
green7ea
libOSTree is a great solution for this. The updates are based on crypto
(sha256) can be atomic and allow for rollbacks. Yocto's meta-updater provides
an easy to deploy integration of libOSTree. Even Qt has an OSTree layer.

~~~
karmicthreat
I think OSTree has a better update model. Especially with connected devices
that might have expensive cell connectivity. I don't think the rest of the
stack for update infrastructure is there though. Qt does seem to be trying to
build some of that out.

------
OrangeTux
A good thing of Mender.io is that is a complete solution: it has both a client
that runs on the embedded device and performs the update, but is also has a
server to manage the updates on a lot of devices.

------
dwalkes
I've started a project to do embedded software updates, leveraging existing
open source projects. It currently supports one platform (the RedBear Duo) and
I'm looking for feedback about additional platforms I should support. The
first platform uses Bluetooth BLE for transport but could work with other
transports as well. See [https://trellis-
logic.github.io/sdu/](https://trellis-logic.github.io/sdu/)

------
cmurf
Red Hat and Fedora use fwupdate which supports ESRT/UpdateCapsule. Pretty sure
this is UEFI specific.
[https://github.com/rhinstaller/fwupdate](https://github.com/rhinstaller/fwupdate)

------
1wd
OT: Who thought that pulse gif was a good ideas?

[http://cloud2.opensystemsmedia.com/share/images/pulse.gif](http://cloud2.opensystemsmedia.com/share/images/pulse.gif)

------
Pica_soO
Why have embedded logic at all- not just a centrall server, with a secure
connection to minimal hardware front ends (drivers and a webstack)?

------
tehlike
do any of the solutions allow you to a/b test, collect metrics, and compare?

that'd be pretty sweet to have, and it would potentially be a multi-hundred
million dollar company.

------
kruhft
Try Urbit. It does OTA software updates.

~~~
RealityVoid
They don't want to. They want you to use their product. The article rather
seems to be a commercial for this:
[https://www.mender.io/](https://www.mender.io/)

~~~
ausjke
thought mender.io is open source?

~~~
RealityVoid
It seems to be, but I think they're makins money off of it somehow. Not that
it's a bad thing, it just adds background to the article.

~~~
Sanddancer
They offer support contracts at $99 for 100 devices, and an unknown per device
discount after that.

