I am the engineer at System76 currently working on this. We are using ME cleaner with -S on all systems where possible - HAP bit will be set AND code removed. All systems will then be tested thoroughly in this configuration before it is released to customers.
Relevant source code can be found in the following places, keep in mind that it is still work in progress:
Next time I consider a laptop, you made it tot the top of my list, even if it costs more.
Hear that, Intel? I will put money where my mouth is. I am not sure I can afford any of these RISC-V chips and they are still alpha quality beyond the Talos, but those of my ilk will do their damndest to make you pay for tone deaf reaction.
So IF they had a Power0 chip it wouldn't be Intel. Though I have zero idea how System76 would put Power9 into a workstation laptop but they do put a desktop cpu into their Serval lineup. https://system76.com/laptops/serval
Counterpoint: I bought a System76 laptop roughly eight (?) years ago (Bonobo Pro 17 in.) and I have no plans to buy from them again.
My system was delivered with a BIOS issue that made it only boot perhaps 1 out of 30 cycles. Delivered on a Thursday night, after spending a lot of time troubleshooting, I had to wait until their support got in on Monday to get anything done. Then I had to mail it in and wait a long time to get it returned with a reflashed BIOS (I'd be lying if I said I remember exactly how long, just remember it feeling way too long).
I explained that I had bought the laptop specifically for on-site work with a client and that I needed a functioning fix or replacement ASAP. They told me they couldn't make anything happen any faster, and even told me that they had no way for me to pay out of my own pocket for faster return shipping, though IMO they should've offered to do this themselves.
I did eventually get back a mostly-working system which I still use around the house, though it's had a habit of hard-locking when the GPU is under stress the whole time I've had it. But my experience with Sys76 customer support is the biggest factor in deciding not to buy from them again.
Do bear in mind this was several years ago and that it is reported from memory. It is entirely possible they've gotten their act together and/or that I'm misremembering some details, but this was my experience as I recall it.
>>> Do bear in mind this was several years ago and that it is reported from memory. It is entirely possible they've gotten their act together and/or that I'm misremembering some details, but this was my experience as I recall it.
Not as good as I want it to be. When we manufacture our laptop chassis here in Denver, Colorado we will have more control of battery size and will always prefer longer battery life
> Next time I consider a laptop, you made it tot the top of my list, even if it costs more.
I won't, if it means having to run Ubuntu (and derivatives) or Pop. I'd like to run the distro of my choosing, thanks, but no thanks. It will be possible to disable ME on other laptops/desktops as well.
You can run whatever distro you want, and the hardware issues should minimal at most? The nice thing is that the hardware is supported by the kernel and you won't run into a hardware issue like we use to all the time 10 years ago.
You cannot run whatever distro you want on a System76 if you want ME disabled.
> You must run Ubuntu 16.04 LTS, Ubuntu 17.04, Ubuntu 17.10, Pop!_OS 17.10, or an Ubuntu derivative and have the System76 driver installed to receive the latest firmware and disabled ME on laptops
My guess is just a QA burden: they probably test the firmware cleaner utility on those operating systems, and they don't have the resources to test and certify the firmware tool on every conceivable distro.
Maybe it will run fine on Fedora, but if it bricks your laptop, they don't want to get blamed.
I'm not sure their motives for this choice, as their means of disabling as much of ME as possible is ME cleaner, which is not tied to distros.
I don't see why you can't attempt to do it manually yourself, without the aid of their driver and firmware update tool. Except that you might forfeit any warranties/support.
System76, can you clarify?
Sadly it's standard to only see official support for Ubuntu or RHEL in the Linux world. Not that I don't understand, it's all very fragmented, and that creates a lot of work as to support multiple distros. People are working on remedies, though.
? Unless you intend to fab your own RISC-V processor, once they become available, they will most certainly be priced competitively (or they won't sell). The first models will be available in 2018 and more will follow.
I have been skimming too light on the market details for that bad boy. Only proves the point that the high-cost might be the price of helping proprietary hardware feel the sting of changing cultural tides.
Talos II is POWER9. POWER also has an open design, so it's effectively the same from a freedom perspective as RISC-V (with the distinction that you can buy POWER chips today, but you can't buy RISC-V ones).
Just don't develop any hardware problems or you'll get stuck in a customer support loop around uninstalling and installing nvidia drivers. I haven't been able to use my oryx for about a year, and each time I ask support (the display doesn't turn on) they just advise removing and re-installing the nvidia drivers.
System76 employee here. Sorry to hear about the continued issue with your computer. I've started up the conversation again in the existing ticket. We will get to the bottom of this problem, and get your system working again.
Not currently. I have experimented with coreboot but did not get far yet. We are tied into AMI for most machines right now, and I would love to cut them out and go open source!
We may be experimenting with coreboot more in the near future
FYI, there is a good technical reason in additional to the philosophical one.
I used to work on embedded systems where I have full control of uboot. With that, I can easily reserve the section of memory in boot uboot and Linux kernel when the keys logging info (such as FTRACE of ISR, sched switches etc) from kernel are kept.
When a kernel panic happens, the system will go thru the warmboot sequence and detected there are logging info in that section of DDR and dump it out before continue to boot.
This "postmortem debugging features" make debugging certain category of very difficult bugs (kernel driver panic) much easier.
With typical close source x86 bios in Linux laptop, it is very hard to enable this kind of debugging.
Would you consider partnering with someone like Dell as a supplier?
I've written this elsewhere, but I'll repeat: it seems a shame that folks like System76, Purism, aspiring Kickstarter campaigns et al aren't leveraging the supply chain already in place for Chromebooks. In the case of Purism, for example, x86-64 Chromebooks comparable to Purism's low end models can be had from Dell for 3/4 the asking price of the former. Considering that the driver situation is necessarily a solved problem for Chromebooks, and they're already using coreboot, it seems silly to start a parallel effort to deal in hardware the way System76 and Purism are doing, rather than just approaching Dell about a white label deal to supply you with hardware components that are already known to work.
Alienware went this route and they turned then into mostly generic video card boosted mediocrity.
The business model of system76 was what I was personally looking for and I've been very happy with my latest laptop from them. For me personally, the luster would have been lost coming from Dell.
Dell makes lovely linux-laptops, but I don't think they run coreboot. Partnering with Google for the chromebook supplychain seems like the better theory/plan.
A white-label, more configurable, plain ubuntu version of the pixelbook sounds like a dream.
> Dell makes lovely linux-laptops, but I don't think they run coreboot.
Dell makes Chromebooks. These Chromebooks run coreboot. Ergo, Dell makes laptops that run coreboot.
There's nothing in my comment about building off the back of Dell's existing program to sell Ubuntu machines to customers—a program whose success story is reported to be bizarrely hit-or-miss with respect to driver issues, even.
Dell's Chromebooks are a success. They're shipped in large quantities. They have no driver issues. They run coreboot. So my advice is simple: if you're a company trying to sell to folks who want a "Linux laptop", then stop sourcing components/machines from otherwise clueless Wintel vendors and trying to shoehorn a traditional distro onto that. Start with the BOM used in Chromebooks. Approach Dell and say you'd like to place a large order—because that's the business they're in, by the way. Now build your company's products on that platform instead.
Substitute "Dell" there with any other competent vendor that has demonstrated their hardware plays well with the FOSS stack powering ChromeOS .
> Partnering with Google for the chromebook supplychain seems like the better theory/plan.
No, that's an ignorant theory. Google is not known for hardware, and certainly not as a supplier for commodity hardware. Even ignoring all that, the response you should expect from Google in regards to playing along should not be enthusiastic agreement.
Can you please consider offering ECC memory in your laptops?
ECC is a very important technology for businesses. Errors in non-ECC memory, while somewhat rare, are not once-in-lifetime-of-universe, they are more like 1 bitflip per year kind of thing, which for some businesses can be considered worth talking about.
There is a demand for this, and almost no one is doing this.
The only other company that has done it so far is Lenovo, and if you know anything about privacy and hidden adware and history of spyware - you will not buy any Lenovo product.
ECC memory would require a Xeon, but it would be possible to underclock it to bring it to a mobile heat emission profile.
I've got a T470P (i7-7700hq), that's a 45W part in a 14" laptop.
If I balls to the wall it I can kill the battery in 90 minutes (though they are swappable) but I average ~6 hours under actual work conditions (VMs and Intelligent).
It spends the vast majority of time sat at a few percent utilisation but the processing power is there for those few second operations you run infrequently (build from scratch, indexing a large project etc).
I love it frankly, not turned desktop on at home more than once or twice since July.
Do you expect any flak from Intel on this decision? I hope not, because I'm going to be in the market for a new laptop in the next year or so and this just put you guys at the top of the list.
To compile to the same bit pattern binary, you'd need the same optimization settings (easy) with the same compiler (harder).
Intel can just give the source, but you can't trust Intel to just give you the same compiler they used, because the compiler might insert a backdoor. You'd need the compiler source, audit it for backdoors, then compile that with a trustworthy compiler. Then use the resulting compiler to compile the ME source.
The comparison between a person administering an unsecured computer network and a drunk driver has just made my list of legal IT analogies, along side BitTorrent being a car that might be used as the getaway vehicle in a robbery.
> You'd need the compiler source, audit it for backdoors, then compile that with a trustworthy compiler. Then use the resulting compiler to compile the ME source.
It is possible (though somewhat time-intensive) to audit binaries, too. If there is real demand for this, it should be possible to crowdscale the auditing problem over a large group of OSS enthusiasts.
Auditing binaries wouldn't really do anything as it's their hardware that'd run the binary. So the hardware can be programmed to lie or to still have some backdoor.
As long as the build is reproducible then they don't have to open it up. It would mean however that you'd still need to audit the compiled program to find anything injected in by the compiler.
Reading this:
> System76 will investigate producing a distro-agnostic command line firmware install tool. Follow us on your preferred social network for updates.
I am wondering. Have you considered using the [Linux Vendor Firmware Service](https://fwupd.org/) ?
It doesn't look like any work at all is actually being done on fwupd from my point of view. I've basically given up hope at this point that System76 is going to do the right thing.
To answer more harshly than before, you did not really try to help us get into LVFS. Instead, you continued to question our method of updating firmware, even though we have open sourced a large amount of our infrastructure.
Still, I have been changing our firmware updater to better suit LVFS and fwupd for when you finally decide to reconnect in a more positive way.
Seems a little unfair to announce a general indictment of System76 like that, to this broad audience, based on conversations only you and they have seen. Why make the account and escalate the drama? I'm sure both sides are doing good work.
Not trying to tone police you, but you run a project that necessitates bringing vendors to the table. Odd to publicly bash one that's trying to work with you.
I am working on embedding all firmware flashing files into one EFI executable when it is built by our CI - this should make it much easier to use from fwupd.
Hopefully you and the LVFS,fwupd maintainer both cool down a bit and get back to the discussion table.
You will both make history here, with more laptop sales and more awareness of LVFS+fwupd. Win-win, as I see it.
All the best.
I have so much respect and a little envy for the work you're doing at the moment. You're working on one of the most far-reaching and undeniably positive software projects I've heard of in years. Best of luck!
It's possible that at some point (depending on if it gets open sourced) I'll be grabbing some of your work and trying to wrangle it to work on my desktop. Is this an area where I seriously risk bricking my CPU, or are there safeguards that cause these kinds of instructions to either work or safely fail?
Yes, you risk bricking your computer. You can create a Raspberry Pi with a SPI clip for fixing bricked computers.
If you have BootGuard, you will brick your machine, unless you have the manufacturer signing key. We have the signing key for our laptops, so we are able to sign updates for the four systems with BootGuard: galp2, galp3, lemu7, and lemu8.
> Is this an area where I seriously risk bricking my CPU
As far as I'm aware, there's no particular state involved for you to brick the CPU, but you can easily write a firmware which will prevent boot, or fail some time after boot (In the case of removing critical ME blobs). You could brick your board temporarily, but you're going to be using an external flasher (with a convenient little SOIC clip :- ) anyway, so fixing it is a matter of writing back the old/working image.
This is, of course, assuming you are able to write anything that works to the flash, which you might not be able to thanks to BootGuard.
Does Ubunutu need to remain on the laptop after the driver has been loaded?
I'm aware that you're working on a distro-agnostic tool. In the interim period, however, if someone with a System76 laptop wants to use Fedora while still disabling ME, can they simply load stock Ubuntu/pop_OS long enough to install the driver and disable ME and then wipe the drive and reload their OS of choice?
Firstly, the ME section is locked until a message is sent to the EC. Flashing firmware on a live system can be dangerous, more so on a laptop. Our flashing method has to take place in EFI.
Signing keys are fused into the CPU for boot guard when the CPU is attached to the motherboard during production, for soldered CPUs like the U and H class. Having a customer signing key would seriously complicate BIOS updates, as only one key can be utilized, meaning our firmware updates would not work.
It would also make returns much more difficult, as all CPUs with customer keys would have to be replaced.
Right now, boot guard is only used on the Galago and Lemur.
There's no way to use TPM as a store for these signing keys because they are needed too early in the process?
Yes, I expect as a general solution for your customers it might be overkill, but as this is basically the cutting edge of public domain research on these topics it would nice to know what's possible for customers who would take the risks.
Understood, I have more reading to do in to how the EFI shim/flasher works.
Give us a System76 laptop with a TrackPoint and you will need to start carrying a snorkel to be able to breathe underneath the piles of money I will throw at you.
(It's never going to happen, I know. But a fellow can dream!)
This - coming from thinkpads the trackpad / keyboard appears to be too much of a downgrade for me to spend money on a System76 but I have been following them for some time.
Was it damaged at all? I would suggest sending it back. In the future, don't accept packages from UPS that show obvious signs of mistreatment. Although I'm not really sure how to do this if they just leave it on your front porch.
On the side note - a new initial setup was released today for Pop that may have fixed your account creation issue. I am sorry if you ran into problems with Pop.
Good to know. I haven't decided if I'd prefer going vanilla Ubuntu. I'm going to give Pop some more time yet. I can see benefits; I'm just not sure they're for me.
I'm not too familiar with the inner workings of Intel ME. Have you verified that setting the HAP bit is effective with the ME code removed? If so, then that is obviously the way to go. However, if the code that interprets the HAP bit and shuts down the ME is removed (as well as everything else not necessary for the computer to run), which option is actually safer, HAP or code removal?
The safest option is to do both, and that's what's happening here (though the post is not clear on it, they're running me_cleaner with options that both erase all but the initial loader module of ME, and set the bit so it disables itself after startup instead of loading other modules)
I think gnu8 was asking: Does removing the code interfere with HAP mode (e.g., by removing code necessary for HAP to be activated, maintained, operated, etc.)? Is both really safer than one or the other?
For example, there may be two drugs that can treat my illness; taking both might not be better than taking one or the other.
Would be a good place to start, as far as not supporting dell goes. Linux is also a first class citizen w/ system76 - Dell has the factory linux xps13 e.g. but I've heard from a handful of friends who use one at work that its obvious Linux was just kinda slapped onto the laptop at the end of manufacturing or whatever. Support is also probably much better with System76.
My home laptop is from a Kudu Pro from System76, while my work laptop is a Dell Precision 5520, with Ubuntu installed. My current Kudu is a replacement for another Kudu that died after a number of years.
For my particular case, the Dell has superior hardware (also, it's about a year newer), but its Ubuntu install has strange problems, and I get crash notification popups somewhat often (including immediately upon logging in for the very first time). The Kudu's install has been rock solid.
I work on Redox at night and on weekends. I try to write software that can be used in both places.
For example, the graphics library orbclient was used in the firmware updater, and the UEFI library from the firmware updater may be used for a Redox UEFI bootloader.
The article and repo seem to be targeting Laptops and there seems to be some specific desktop-related lines in there. In the laptop side of things, it clearly states the Intel ME does not have any function, but it does not say the same for desktops. I imagine the ME is also useless for desktops, but maybe I'm wrong here and there is a reason this distinction is made?
Why why why won't you just use LVFS? A number of OEMs have absolutely no problem delivering updates through all sorts of tools and channels.
It feels NIH and is exactly why I'll be buying Dell and never System76. I can one-click flash my Dell forward without configuring or installing anything.
For that matter why is there a "HiDPI Service" running? Why is the driver a set of udev rules that could've been upstreamed?
Relevant source code can be found in the following places, keep in mind that it is still work in progress:
- System76 Driver with Firmware Update support: https://github.com/pop-os/system76-driver/tree/firmware_artf...
- Firmware Update Frontend: https://github.com/system76/firmware-update
Please ask me anything