
System76 ME Firmware Updates Plan - jcastro
http://blog.system76.com/post/168050597573/system76-me-firmware-updates-plan
======
jackpot51
I am the engineer at System76 currently working on this. We are using ME
cleaner with -S on all systems where possible - HAP bit will be set AND code
removed. All systems will then be tested thoroughly in this configuration
before it is released to customers.

Relevant source code can be found in the following places, keep in mind that
it is still work in progress:

\- System76 Driver with Firmware Update support: [https://github.com/pop-
os/system76-driver/tree/firmware_artf...](https://github.com/pop-
os/system76-driver/tree/firmware_artful)

\- Firmware Update Frontend: [https://github.com/system76/firmware-
update](https://github.com/system76/firmware-update)

Please ask me anything

~~~
616c
Next time I consider a laptop, you made it tot the top of my list, even if it
costs more.

Hear that, Intel? I will put money where my mouth is. I am not sure I can
afford any of these RISC-V chips and they are still alpha quality beyond the
Talos, but those of my ilk will do their damndest to make you pay for tone
deaf reaction.

~~~
dsr_
I've been buying System76 laptops for about ten years now, for work and home.
Their customer service is first rate.

~~~
na85
How's the battery life?

~~~
jackpot51
Not as good as I want it to be. When we manufacture our laptop chassis here in
Denver, Colorado we will have more control of battery size and will always
prefer longer battery life

~~~
jackhack
Security focused and American manufactured (not just assembled)??

Good tools cost more, and are worth it.

Your product just went to the top of my shopping list.

------
jimmies
You don't need to buy branded laptops to be able to disable Intel ME. Thanks
to me_cleaner, I have been running my Chromebooks with Intel ME disabled for
years. You can do it too if you have 30 minutes and a raspberry pi.

Here is what the intelmetool [1] says on my laptop:
[https://i.imgur.com/yKTt5ga.png](https://i.imgur.com/yKTt5ga.png)

What's even more interesting is that there is a simple, automatic, no-frills
method to clamp a SOIC clip connected to a Raspberry Pi zero to any Chromebook
and it will clear out the Intel ME automatically with no user interaction.
It's not even hard, all you have to do is to configure the Pi to enable SPI,
and then make the pi automatically run flashrom to pull out the ROM from the
Chromebook's flash chip, run ./me_cleaner on the ROM image, and then flash it
back.

It can be done safely and automatically, and you wouldn't have to risk frying
your laptop, so everyone could do it provided they can open their laptop.
However, I'm too lazy to document it properly: either by providing the image
tool to create the said Raspberry Pi Zero image + h/w instruction or by
providing a premade hardware to do it.

I have reasons to believe that downloading a random binary image from a random
guy nicknamed jimmies and use it to flash the firmware to your laptop because
you don't trust Intel is probably not a great idea. The act of creating such a
script to customize Raspbian, and testing out to make sure that it works for
every Chromebook or laptop, and make a hardware compatibility list is quite a
daunting task. I was talking about that briefly to some of the security
people, but then as a grad student trying to graduate it got to the pile of
TODOs. So if anyone is interested in it, let me know and I can provide some
more details.

Currently, I'm running a Dell 13 inches Chromebook that can be had for $300
and does everything I need.

1:
[https://github.com/coreboot/coreboot/tree/master/util/intelm...](https://github.com/coreboot/coreboot/tree/master/util/intelmetool)
\- disclaimer: I contributed a patch to the intelmetool to make it work on the
Chromebook.

~~~
korethr
While this is great, and I'm glad it exists, not everyone is like you and I:
willing and able to open a computer and use a SOIC clip and a Raspberry Pi to
purge the ME from the computer's firmware. You and I are what the normals tend
to call geeks or nerds, and from my experience amongst geeks and nerds, one of
the less common subtypes. That's a depressing way to look at it, and one could
argue that it's a condescendingly elitist view, but that doesn't change the
truth of it. I'm confident most other posters here could immediately think of
a half-dozen friends and family members for whom even opening the computer was
deep magic, nevermind going after the ME with a SOIC clip.

For the rest of the population, there's value in there being a vendor who
sells magic black boxes, certified to be freshly purged of all known evil
demons.

~~~
jimmies
>For the rest of the population, there's value in there being a vendor who
sells magic black boxes, certified to be freshly purged of all known evil
demons.

Sure, but there are things that I like about my Chromebook that the System 76
laptops, despite being way more expensive, are unable to to provide. I like
how mine has a backlit keyboard, has a FullHD 1080p IPS screen, and an 10
hours battery life, all that for $300.

I just looked at the number of people who upvoted my post and I am really
think that this actually could work if there is a person who builds the
hardware, which I think is trivial. The problem is that not many are
comfortable with both the electronics (wiring the clip) and be able to compile
a big software package with dependencies on the Raspberry Pi. Make the
frustrating 2 hours job to a 5 minutes job of opening the laptop and identify
the chip, I think it might just be what many people want.

I talked to a very smart netsec person and he just said that because he
doesn't understand electronics, he couldn't have done it himself, but if
that's something prebuilt or foolproof, he would absolutely do it.

Perhaps I will actually take the plunge and publish the work over the next
couple of months as a Show HN post.

~~~
PuffinBlue
> I like how mine has a backlit keyboard, has a FullHD 1080p IPS screen, and
> an 10 hours battery life, all that for $300

That sounds very compelling. Which model are you using please? I'd like to
take a closer look but I don't seem to see one with these features/price - but
I am in the UK.

~~~
jimmies
The Dell Chromebook 7310. They don't sell it anymore now. They used to have
the option to choose Celeron/i3/5/7 and 4/8GB of RAM. I have the 8GB i3 one.

------
cjbprime
> System76 will automatically deliver updated firmware with a disabled ME

Having the ability to automatically push new firmware of your own creation to
customers' machines is more power than you ought to want. My security threat
model as a System76 customer shouldn't have to include you (perhaps with you
being hacked or coerced) pushing me malware that's undetectable to my OS after
it's been installed.

Please reconsider this feature (of automatic firmware updates). Firmware
updates are rare enough that it should be fine for them to be explicitly opt-
in. It's great to want to make Intel's firmware more secure. But replacing
Intel as a possible attack vector with _yourselves_ is strictly worse for the
machine's security.

~~~
cassidyjames
System76 employee here. I suppose the "automatic" could be re-worded; the data
is automatically pushed to the machine but then the user always opts into the
actual install.

But on the other hand, System76 customers are trusting that System76 hasn't
been hacked or coerced to ship malicious firmware from the factory _in the
first place_. These updates are signed and verified with industry best
practices.

jackpot51 (the System76 engineer currently working on this) could probably
detail it better than I can, though.

~~~
mulmen
Is the user intervention _required_ to install the firmware or do you still
have the capability to install it without asking? Just because you ask for
permission now does not mean that you _have_ to.

Why push the firmware before asking?

~~~
cassidyjames
I asked David Jordan, the engineer working on the firmware updater.

>Not only aren't updates initiated without permission, I wrote the code to
make that literally impossible without changes to the installed python code.

The code is available at [https://github.com/pop-
os/system76-driver](https://github.com/pop-os/system76-driver) and
[https://github.com/system76/firmware-
update](https://github.com/system76/firmware-update)

~~~
mulmen
Thanks for the reply! So you can push an update to the python code that allows
you to push an update to the firmware without prompting? Sounds like we still
rely on the security of your systems to prevent malicious firmware from being
pushed.

~~~
yjftsjthsd-h
Well... yeah? They're the OS vendor; there is literally no way for them to do
their job without having the ability to update the system.

~~~
mulmen
Yes they have to be able to update the system but in this case they are also
able to update the firmware without asking which means anyone who can
impersonate or coerce them can also update the firmware.

~~~
ibotty
If you control the OS, you also control the firmware (if you want a way to
install new firmware from the OS). No way around.

------
ef4
This is a strong contrast with my experience in trying to patch the
vulnerability on an Asus desktop motherboard.

The process was so byzantine that I very much doubt more than a small fraction
of home users would get through it, or even bother starting.

The correct steps were (1) flash a newer bios, (2) install the Intel ME driver
for windows, (3) run the actual vulnerability patching tool. Discovering those
steps required a bunch of trial and error and navigating Asus's really
terrible website full of badly named downloads.

~~~
flukus
> This is a strong contrast with my experience in trying to patch the
> vulnerability on an Asus desktop motherboard.

This is why I don't buy "enterprise users" as a reason for having IME. I've
never once worked in a company that patched firmware, even though they have
specialists capable of it. They want the option to perform enterprise wide
upgrades with ease but they aren't willing to pay the true cost of having this
ability.

> Discovering those steps required a bunch of trial and error and navigating
> Asus's really terrible website full of badly named downloads.

Gets even worse when your international. Half the links will be to a US
address that will then try to redirect you to a localized one which will then
not have the resource you were looking for. Then you've got some really
byzantine export restriction procedures and you have to create an account but
it still probably won't work. I've had these issues with ordinary drivers too,
it's the biggest reason I support the linux in kernel tree and no stable ABI
model, it's better for users.

------
freedomben
This is awesome. System76 does a lot for the Linux community. Everyone reaps
the benefits of their work, even people that aren't customers. One example
from just this blog post:

> System76 will investigate producing a distro-agnostic command line firmware
> install tool. Follow us on your preferred social network for updates.

~~~
phaus
They sell re-branded clevos with marginal build quality. Some models have been
unusably defecitve. I wish they were the Linux-laptop company that many
hackers dream about, but they just aren't.

~~~
djsumdog
There was another comment on another threat that said that, although they do
buy Clevos, the cheaper ones Clevos sells aren't up to the same specs. They
also order really specific hardware for their branded laptops that have known
working drivers. Either that or they work on getting the correct binary blobs
working and merged into the official linux-firmware package.

Things like getting newer drivers working (and contributing that openly back
to the community) plus getting ME disabled are some pretty big value added. If
they had a 4k model I might be more interested.

~~~
cassidyjames
System76 employee here! We do have a number of HiDPI models.

Galago Pro
([https://system76.com/laptops/galago](https://system76.com/laptops/galago))
is probably our most popular, and its resolution is 3200×1800—not technically
4K, but the right resolution for that small of display.

Oryx Pro usually has a 4K HiDPI option, but it looks like it's currently sold
out. Our WS models also have 4K options, but they're pretty specialized
machines.

------
LeoPanthera
> You must run Ubuntu 16.04 LTS, Ubuntu 17.04, Ubuntu 17.10, Pop!_OS 17.10, or
> an Ubuntu derivative and have the System76 driver installed to receive the
> latest firmware and disabled ME on laptops*

Ubuntu is the distribution that once sent everything you typed into the
desktop search box to Amazon so that it could deliver you ads. Current
versions may not do that but it's clear that Canonical prioritizes profit over
privacy.

It's disappointing that if you choose not to run Ubuntu you can't take
advantage of their firmware update tool.

~~~
ythn
So? Install Ubuntu, receive updated firmware/disabled ME, reinstall distro of
choice. Also:

> System76 will investigate producing a distro-agnostic command line firmware
> install tool. Follow us on your preferred social network for updates.

~~~
orblivion
> Install Ubuntu, receive updated firmware/disabled ME, reinstall distro of
> choice.

For a single software update? With whatever else a normal human has going on
in life? There's no way.

~~~
dopeboy
I agree - that's too much work. Can this task be performed in a VM?

~~~
barkingcat
Think about what you are asking first. A VM is a virtual machine that
theoretically is separated from the base hardware. I don't think a VM is the
solution for Bios Updates or changing bits on the physical hardware itself.

Of course, you can monkeypatch the VM's bios all you want! But that would be
erased the next time you start a new vm.

The only other way is to find a security hole in KVM, qEMU, or XEN, and then
exploit it to break out of the VM[1] and get access to base hardware. Hard to
do, but it does happen.

[1]
[https://en.wikipedia.org/wiki/Virtual_machine_escape](https://en.wikipedia.org/wiki/Virtual_machine_escape)

------
filereaper
If anyone's interested, here's the full article from Positive Technologies on
how they went about discovering reserve_hap and disabling it.

[http://blog.ptsecurity.com/2017/08/disabling-intel-
me.html](http://blog.ptsecurity.com/2017/08/disabling-intel-me.html)

------
tomkat0789
Hi I'm a mostly average Linux user just now learning about these hardware back
doors as I'm planning on building a new computer. Can I avoid the
disadvantages of ME et al by building my own computer from a separate
motherboard and CPU? I remember reading that somewhere but I haven't see it
explored again.

My first thought was "Ah ha! AMD CPU!", but they seem to be in on it too.
What's a "normal person" to do?

~~~
yjftsjthsd-h
Today: Buy really old systems and/or patch the firmware.

Soon™: RISC-V, we hope:) Or Talos, or probably a handful of others.

~~~
jackpot51
Yep - RISC-V is the best bet for full freedom

------
jlgaddis
I hope other vendors will follow their lead but I'm certainly not holding my
breath.

~~~
jstanley
Purism have already been doing this. I think this is system76 following
purism's lead.

[https://puri.sm/products/](https://puri.sm/products/)

Edit: not to detract from system76! It's excellent that they're doing this
too.

~~~
prophesi
Yeah, Purism is awesome with their privacy, security, and libre focus. I'm
really hoping their Librem phone is a success.

~~~
allemagne
I've heard a lot of great things about them and I love their mission
statement, but I've also heard a lot of bad things about availability,
shipping, and spotty customer service (but again, occasionally great customer
service). I'm not sure what to think and whether to get my next laptop from
them.

It also makes me nervous that they're putting resources into phones when they
obviously haven't solved all the issues with non-free components of laptops.
It gives me Firefox OS and mobile Ubuntu vibes.

~~~
cyphar
> It also makes me nervous that they're putting resources into phones when
> they obviously haven't solved all the issues with non-free components of
> laptops.

By not using x86 (the board is going to use i.MX8), they remove a whole series
of issues that plague modern laptops (no ME, no EMC, etc). They have a very
long run-down on why they chose i.MX8, and the freedom status of the
components[1].

[1]: [https://puri.sm/posts/librem-5-roadmap-to-
imx8/](https://puri.sm/posts/librem-5-roadmap-to-imx8/)

------
nkkollaw
system76 are cool, but their laptops IMO should be better for the money they
cost.

I'd still like to get one, but there's better stuff around at those prices.

~~~
mStreamTeam
You can get the same laptop from a different distributor for a lower price.
Try searching for 'Clevo' or 'Sager' laptops. Clevo is the company that makes
the base model, but they don't do any sales to consumers. Sager is another
name under which the same laptops are typically sold.

~~~
nkkollaw
Yes, that's exactly what I don't like.

They are little more than repackaged laptops, and not that good in terms of
reliability from what I've heard.

~~~
platinumrad
They announced earlier this year that they were going to start doing their
hardware design in-house[1]. I doubt this will make them any cheaper but might
help on other fronts at least.

[1] [http://blog.system76.com/post/159767214983/entering-phase-
th...](http://blog.system76.com/post/159767214983/entering-phase-three#Linux)

~~~
nkkollaw
I can't wait. I would love for them to do well.

------
lorenzhs
I pieced together how to update a ThinkPad's ME firmware without Windows last
week:
[https://news.ycombinator.com/item?id=15744152](https://news.ycombinator.com/item?id=15744152)
\- the process is a pain, as Lenovo only provides a Windows binary for
installing the update, so you have to piece together a WinPE image that
contains the ME driver and firmware update tool, boot it, load the driver, and
then execute the firmware update.

The process is probably the same for most other vendors, so if your vendor
provides a Windows ME firmware updater (no pun intended :P) you might be able
to make it work that way.

------
horsecaptin
Do AMD chips have a "management engine"?

~~~
TazeTSchnitzel
Yes, it's called the Platform Security Processor:
[https://libreboot.org/faq.html#amd-platform-security-
process...](https://libreboot.org/faq.html#amd-platform-security-processor-
psp)

The system won't boot without it.

------
mykull
I used me_cleaner on my old as heck gigabyte sandybridge desktop and was
surprised how straight-forward it was. I ran the python script against latest
rom, reflashed, everything works and I confirmed it's disabled.

------
silur
Great to hear that you kill ME but I'll still stick with puri.sm or
alpha.store because systen76 machines are just too freakin overpriced. They
talk about freedom and FOSS yet their machines can cost 1.5 times a macbook,
with free OS and rebranded chineese 200$ clevo cases. Available price is an
essential part of FOSS and OSHW phylosophy, greed never had any place in this
community

------
justinjlynn
I wonder if system76 might consider laptops with alternative architectures
like POWER9. A number of them are open all the way down to and including their
concept of microcode.

------
EdSharkey
Is there any appliance or software that can passively monitor a network to see
if any ME commands and related data flow back and forth from an attacker?

It seems to me that using ME as an _actual_ backdoor would be only an
occasional thing, (maybe once in a lifetime thing), but it would be so cool to
at least know when it is happening and maybe capture some packets.

------
shmerl
Is there some way to disable this for AMD CPUs?

~~~
Joakal
Intel Management Engine is already disabled for AMD CPUs.

~~~
shmerl
It's not, it's just called differently.

------
lottin
Shouldn't the news be that the ME can be disabled?

From what I heard the consensus was that it couldn't be turned off.

~~~
zamalek
It happened more than a year ago[1]. It doesn't remove ME entirely, but
drastically reduces the attack surface in the currently accessible ring - the
GitHub project has more details. Ultimately, the network-enabled bits that
keep getting exploited are removed.

[1]:
[https://github.com/corna/me_cleaner](https://github.com/corna/me_cleaner)

------
cement_head
Just to be clear, the firmware upgrade tool is the same as the System75 Driver
(pre-installed on System76 laptops)?

------
brightball
This will actually give people a way to vote with their wallet. Thank you!

~~~
aaronmdjones
How so? You're still buying a system with an Intel CPU. Intel still gets their
"dues". The number of people that care about this is a fraction of 1 percent
of their user base.

Just so you don't misunderstand me, I am one of those people, and what Purism
and System76 are doing is great.

I just don't think it's going to affect Intel in any way whatsoever.

~~~
brightball
It probably won't but if it shows an uptick for System76's business then you
might see the other companies who actively ship systems that devs pay
attention to doing the same. If the action ever gets Dell or Lenovo to follow,
it's still progress.

------
znpy
Unrelated: any plans about integrating a trackpoint/ultranav solution in
System76 laptops?

------
knodi
Just stop buying Intel.

