
Meltdown and Spectre Linux Kernel Status - dankohn1
http://kroah.com/log/blog/2018/01/19/meltdown-status-2/
======
lima
RHEL/CentOS state:

Red Hat built their own mitigation for Meltdown and Spectre and were the first
distribution to have an updated kernel right when the embargo was lifted.

As far as I can tell, their meltdown mitigations are similar to what's in the
upstream kernel now, and the Spectre mitigations relies on Intel's new MSRs
(the microcode update that had since been retracted - new microcode_ctl
packages remove it).

Interestingly, you can enable/disable the mitigations at runtime with Red Hat,
which is useful for performance testing. I found no way to do this with the
upstream kernel.

I'm not happy that Intel, Google, Red Hat and a few others have known about
the bug for months and the upstream kernel devs are scrambling to build
mitigations and still don't support the microcode mitigations. Google
apparently patched their stuff in September, while everyone else is left out
in the cold. This is a real mess. Surely there must be a better way than
telling a few select companies?

The Red Hat guys in particular surely would've loved to collaborate with the
upstream kernel. Must be frustrating not to be able to tell anyone.

~~~
masklinn
> I'm not happy that Intel, Google, Red Hat and a few others have known about
> the bug for months and the upstream kernel devs are scrambling to build
> mitigations and still don't support the microcode mitigations. Google
> apparently patched their stuff in September, while everyone else is left out
> in the cold. This is a real mess. Surely there must be a better way than
> telling a few select companies?

I think Colin's take[0] was interesting and insightful there, especially in
the light of him having handled a very similar issue a decade earlier:

> The way these issues were handled was a mess; frankly, I expected better of
> Google, I expected better of Intel, and I expected better of the Linux
> community. When I found that Hyper-Threading was easily exploitable, I spent
> five months notifying the security community and preparing everyone for my
> announcement of the vulnerability; but when the embargo ended at midnight
> UTC and FreeBSD published its advisory a few minutes later, the broader
> world was taken entirely by surprise.

> Contrast that with what happened this time around. Google discovered a
> problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go
> around contacting all of the operating systems which would need to work on
> fixes for this? Not even close. FreeBSD was notified the week before
> Christmas, over six months after the vulnerabilities were discovered.

> […]

> To make things worse, the Linux community was notified _and couldn 't keep
> their mouths shut_. Standard practice for multi-vendor advisories like this
> is that an embargo date is set, and _nobody does anything publicly prior to
> that date_. People don't publish advisories; they don't commit patches into
> their public source code repositories; and they _definitely_ don't engage in
> arguments on public mailing lists about whether the patches are needed for
> different CPUs.

(emphasis his)

[0] [http://www.daemonology.net/blog/2018-01-17-some-thoughts-
on-...](http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-
and-meltdown.html) final section on vulnerability disclosures

~~~
Sir_Cmpwn
I think lots of people (particularly security people) tend to fundamentally
misunderstand how the Linux community works. It's not a single organization
calling the shots where they can have a meeting behind closed doors about the
vulnerability and internally incubate a patchset. That's simply not how Linux
development works.

Linux is a _loosely organized_ community of developers who collaborate in
public. They don't have any ubiquitous communication channels established
_other_ than mailing lists. If you attempted to establish a private security
mailing list, the list of people who get to go on it is (1) hard to define (2)
very large and (3) frequently changing. The affected subsystems have lots of
disparate maintainers and interested parties who need to get their eyes on any
patches that go into them.

It might not be ideal from a security person's perspective but from everyone
else's perspective this model has been the driving factor behind the most
successful collaborative project of all time.

~~~
gregkh
Linux has a security team/list that does handle disclosures and fixing of
reported problems. It usually does this by just dragging in the responsible
parties, but many times it just fixes the problems themselves and submits the
patches to the proper part of the kernel.

For details on how this works, please see the kernel documentation itself:
[https://www.kernel.org/doc/html/latest/admin-
guide/security-...](https://www.kernel.org/doc/html/latest/admin-
guide/security-bugs.html)

~~~
Sir_Cmpwn
Thanks for clarifying, Greg.

>It usually does this by just dragging in the responsible parties, but many
times it just fixes the problems themselves and submits the patches to the
proper part of the kernel.

I think this is the key - once the patches are submitted they go through the
normal workflow and aren't any different from a typical patch, right?

~~~
gregkh
Yes, once the patches are generated, they get submitted just like any other
kernel change, sent to the responsible subsystem and maintainer for inclusion
like any other kernel change.

~~~
Sir_Cmpwn
I think this is where the problems becomes apparent. The change then has to go
through the subsystem's patch flow (in public) then into the merge window (in
public) and sit in the release pipeline (in public) until the window closes
and all of the rc's are through.

Personally I don't think this is a huge deal, but it's where the disconnect
between the security person's ideal worldview and the reality of how Linux is
built colllide.

~~~
qznc
Distros could speed this up. If the security team notifies RedHat, Ubuntu, etc
and they can apply the patch for their kernels immediately. They probably need
to backport it anyways, because no distro (well, maybe Arch?) just uses the
latest Linux release.

~~~
SSLy
FWIW, during the last year Fedora frequently had newer kernels than arch.

------
tomxor
If you also want to know if intel has released a microcode update for your CPU
yet (regardless of whether or not it's available in your distro yet):

Download the latest archive from Intel at
[https://downloadcenter.intel.com/download/27431/Linux-
Proces...](https://downloadcenter.intel.com/download/27431/Linux-Processor-
Microcode-Data-File) (currently microcode-20180108.tgz) then:

    
    
        tar -xzf microcode-20180108.tgz
        sudo apt install iucode-tool # or whatever your package manager is
        iucode-tool -tb -lS ./intel-ucode/*
    

This will crosscheck the microcode binaries with the signature of your CPU and
output something like:

    
    
        001: sig 0x00010676, pf mask 0x01, 2010-09-29, rev 0x060f, size 4096
        002: sig 0x00010676, pf mask 0x04, 2010-09-29, rev 0x060f, size 4096
        003: sig 0x00010676, pf mask 0x10, 2010-09-29, rev 0x060f, size 4096
        004: sig 0x00010676, pf mask 0x40, 2010-09-29, rev 0x060f, size 4096
        005: sig 0x00010676, pf mask 0x80, 2010-09-29, rev 0x060f, size 4096
    

As you can see Intel hasn't released any updates for my old CPU (latest is
from 2010)

~~~
mahrain
So this microcode goes in /lib/firmware/intel-ucode (in ubuntu) will this then
firmware update the CPU with the latest microcode, or only use this like a
driver leaving the CPU alone?

~~~
tomxor
Microcode is loaded into some on-die volatile RAM every boot by the CPU...
there is a CPU instruction to do it. AFAIK:

First the CPU loads an on die ROM of the microcode it was released with on
it's own. Second the BIOS/UEFI may load an update into the CPU before boot if
it has a newer one. Third linux and some other OS will load any newer
microcode into the CPU at the bootloader stage before the kernel loads.

This is also commonly the case for other modern hardware, not just CPUs...
although I expect it doesn't need to happen before the kernel loads and is
probably bundled into their respective drivers.

------
Santosh83
Am I correct in stating that the microcode update recently released by Intel
was buggy and therefore pulled back by Linux distros and Intel themselves, and
also a fixed version of the microcode update has not yet been released by
Intel for Linux as well as other OS like Windows?

~~~
j_s
I am curious to know more details on the status of the microcode update
specifically when it comes to Windows Update.

~~~
newman314
I do not believe the Windows patches include microcode at this time.

------
caio1982
What a mess this has been so far... I appreciate all the effort kernel folks
had put in it until now but the whole tier 1 companies cabal thing is just
disgusting and it made the problem worse with the early disclosure (not to say
"leak" if you count LKML diffs flying around). If at least all the cabal
parties were equally covered and protected by now, but nope nope nope. If I
were a kernel developer not working for any cabal party I would be very pissed
off.

~~~
TorKlingberg
How would you do it then? Publish as soon as it's discovered, and let everyone
scramble for fixes as its being exploited?

~~~
caio1982
0\. fucking put the pressure on Intel and other chip makers to have microcode
updates ready by NYE, like, you know, 6 months after the first report

1\. make sure all tier 1 companies involved are equally patched and on the
same page as far as mitigations go before EOY, then involve tier 2 companies
(see news on "the impromptu war room") a full month before the embargo
deadline

2\. not commit or patch anything or fly commented diffs around in the freaking
public LKML; git branches to be tested and secondary mailing lists can be
private if only for a brief period of time

That's definitely not what happened, and probably the dirty details have not
surfaced yet. I can't wait for this to be reasonably over so we can start to
read post mortem on the whole process.

~~~
euyyn
You can't really force a different company to have patches and updates by a
certain deadline, can you. The only hard leverage is the officially planned
disclosure date.

------
eigengrau
I’m confused… The article led me to recheck whether I’m loading the newest
Intel ucode update during early boot, and it turns out I am, but the latest
ucode revision for my Ivybridge machine is from 2015. Didn’t Intel push
microcode updates for all affected systems yet, in preparation for more
spectre-related mitigations?

~~~
remael
Lenovo mentions [1] a "target availability" of a bios update for my x230 with
ivybridge for 2nd of February. Intel didn't update the microcode for those
chips yet.

[1]
[https://datacentersupport.lenovo.com/de/de/products/storage/...](https://datacentersupport.lenovo.com/de/de/products/storage/lenovo-
storage/v3700v2/product_security/ps500151)

------
blauditore
So, apparently my system is unpatched. What is the proper thing to do now?

Should I immediately update my BIOS (would this also patch my CPU's firmware?)
or just wait for mitigations arriving with OS updates?

~~~
cesarb
> Should I immediately update my BIOS (would this also patch my CPU's
> firmware?)

The current recommendation is "no", since some of the microcode updates are
known to cause stability problems. It's better to wait for Intel to fix that
first, then make sure your BIOS update has a microcode version without the
stability problems, then update. Yes, it's a mess.

> or just wait for mitigations arriving with OS updates?

Even after you update the microcode, you still need a OS update to enable the
microcode mitigations, so you have to wait for a OS update anyway.

~~~
danieldk
_The current recommendation is "no", since some of the microcode updates are
known to cause stability problems. It's better to wait for Intel to fix that
first, then make sure your BIOS update has a microcode version without the
stability problems, then update. Yes, it's a mess._

Note that the microcode update does not have to be delivered through a BIOS
update. Linux can also update the microcode during early boot:

[https://wiki.archlinux.org/index.php/microcode](https://wiki.archlinux.org/index.php/microcode)

------
drofmij
This command does not show anything on my ubuntu 16.04 lts machine.

grep . /sys/devices/system/cpu/vulnerabilities/*

is there an alternate command for ubuntu?

~~~
drofmij
for ubuntu command check here: [https://askubuntu.com/questions/992137/how-to-
check-that-kpt...](https://askubuntu.com/questions/992137/how-to-check-that-
kpti-is-enabled-on-my-ubuntu)

Here is one of the commands recommended in that thread:

grep -q "cpu_insecure\|cpu_meltdown\|kaiser" /proc/cpuinfo && echo "patched
:)" \ || echo "unpatched :("

~~~
the_af
That line worked for me, but do note the first command listed in the answer
you linked to is NOT reliable and is not always consistent with the next two
commands.

------
jo909
You can find a "shared by a subscriber" non-paywall link of the lwn.net
article mentioned in the article here:

[https://news.ycombinator.com/item?id=16177525](https://news.ycombinator.com/item?id=16177525)

~~~
eddyg
LWN is worth supporting.

~~~
jo909
Sure it is. They allow the link to be shared nevertheless.

[https://lwn.net/op/FAQ.lwn#slinks](https://lwn.net/op/FAQ.lwn#slinks)

Where is it appropriate to post a subscriber link?

Almost anywhere. Private mail, messages to project mailing lists, and blog
entries are all appropriate. As long as people do not use subscriber links as
a way to defeat our attempts to gain subscribers, we are happy to see them
shared.

~~~
LeifCarrotson
And as this is a blog entry, I think it's a place where the author should have
used the subscriber link.

> _As always, [lwn.net covers the technical
> details]([https://lwn.net/Articles/744287/](https://lwn.net/Articles/744287/))
> about the latest state of the kernel patches to resolve the Spectre issues,
> so please go read that to find out that type of information._

The non-subscriber link in the blog entry results in a paywall. I think LWN
would prefer that Greg used a subscriber link in this case. You may be being
downvoted because people think you're trying to post a paywall workaround, but
I agree that this would have been an appropriate place to use a subscriber
link.

~~~
tytso
You _can_ use a subscriber link, but Greg _chose_ not to. Jon Corbet is a
mutual friend, and LWN is essentially a two person shop (Jon and Jake do all
of the work, and they are not getting rich off of what is essentially a labor
of love).

So if someone decides they want to use a somewhat more proactive way of
encouraging people to subscribe to LWN, personally I think it's appropriate
and its his choice. Just as someone who wants to post a subscriber link to
Hacker News is also appropriate. (But you should seriously think about
subscribing, even at the Starving Hacker level, if at all possible. I
subscribe at the highest tier even though my company has a corporate
subscription, Just Because.)

------
eis
> And yes, it is behind a paywall for a few more weeks, you should be buying a
> subscription to get this type of thing, go do that!

LWN has tons of great content and deserves any support they can get.

But I strongly suggest that in this case, there shouldn't be a paywall. The
current state of Meltdown/Spectre fixes is of high public interest. And people
who are normally not into kernel development should be able to read this
information without having to pay (or find workarounds).

