
Linux Kernel Developer Criticizes Intel for Meltdown, Spectre Response - CrankyBear
http://www.eweek.com/security/linux-kernel-developer-criticizes-intel-for-meltdown-spectre-response
======
hansendc
I worked on the Meltdown mitigations, at Intel, during the embargo. I still
work on Linux at Intel.

Intel told quite a few members of the Linux community about this well before
late December. Some got told because they work for traditional distributions
and others were contacted directly because they are community maintainers or
subject matter experts.

Unfortunately, Greg was not one of those folks that was told early. It's
pretty clear at this point that it would have been a _lot_ better had he and
folks like him been involved earlier. This is especially true for Greg since
he plays such a crucial role in stable kernel maintenance, which is how a lot
of the world consumes their kernels (including distros like Debian).

I'm glad Greg thinks "Intel has gotten better at this." I'd like to think so
too.

~~~
ibotty
That does not address the stupid embargo on involved linux developers to speak
about it to each other.

~~~
mjevans
It shows that legal and marketing (PR spin) were prominent on the security
decision tree rather than security minded individuals.

~~~
MaxBarraclough
It's a real pity that _Company caught putting PR before customers ' interests_
so rarely causes a backlash, even when it's plain as day.

~~~
hansendc
Disclaimer: I work on Linux at Intel.

FWIW, I don't think PR had anything to do with this. Without arguing the
merits of embargoes, the goal of them is to give the good guys a head start
before the bad guys. But, if you tell too many good guys, the assumption is
that the bad guys find out. So, you try to tell as few good guys as possible.

This was (from my blatantly biased perspective) an honest, but imperfect
attempt at keeping the bad guys from finding out. A lot of lessons have been
learned since this, and the group of good guys involved in recent issues has
been much more comprehensive than with the original Spectre/Meltdown bits. I
think Greg was alluding to this when he said "Intel has gotten better at
this."

~~~
blihp
The part that I find amazing about this is that Greg and Linus weren't at the
top of the list of people contacted to ask 'this is a very sensitive issue and
we'd like to work with you to figure out who we need to get involved in this'
from a Linux standpoint. Intel has had people working on Linux support for
many years now so it's not like they didn't have people on the payroll who
knew the landscape and couldn't readily figure out who to contact. This reeks
of incompetence at the senior management levels of Intel.

~~~
cookiecaper
Greg and Linus are also some of the most visible participants, and as far as I
understand, neither is considered particularly expert in the platform
intricacies that would be involved in mitigating something like Spectre.
Disclosing vulns to high-profile targets is a risky practice; their activity,
both public and private, attracts a lot of attention. It makes sense to avoid
that risk if possible.

------
cthalupa
>"The majority of the world runs Debian or they run their own kernel," Kroah-
Hartman said. "Debian was not allowed to be part of the disclosure, so the
majority of the world was caught with their pants down, and that's not good."

Is there any actual statistics to back this up? I feel like RHEL and to a
lesser extent CentOS have a stranglehold on the big enterprise-y environments,
and I see Ubuntu basically everywhere else, and Canonical does their own
kernels.

Edit: To be clear, I am aware Ubunbtu is a Debian derivative, but since we're
talking specifically about who was informed for kernel level mitigations, and
Canonical does their own kernels, it seems weird to talk about how Debian
wasn't informed and thus people were affected, when Ubuntu being updated
wasn't reliant on Debian being updated.

~~~
dijit
I can speak to the companies I've worked for: US companies (and, this includes
the UK because apparently we love to be like america) use CentOS
predominantly, but mainland europe (Sweden, Germany, Finland) seem to prefer
Debian.

I can say that I've personally administered roughly a 1:1 ratio of
CentOS:Debian despite coming from a country that's servers tend to be CentOS.

Of course, this is anecdotal, but don't undersell debian.

~~~
bluedino
Is SuSE still popular in Europe? (Apologies if I got the capitalization wrong)

~~~
pjmlp
Besides SAP deployments I guess not much.

Here in Germany it is mostly Red-Hat/CentOS or Ubuntu, on the projects I have
been involved.

So anecdote data.

------
2trill2spill
> "That's a long time, and we only heard rumors because another very large
> operating system vendor told Intel to get off their tails and tell us about
> it."

I wonder which operating system vendor pressured Intel to tell the Linux dev
community, especially because it sounds like it was a non Linux OS vendor.
Whomever it was, good job!

But it seems like Intel has angered the Linux community as well as the various
BSD operating systems. You would think Intel would be doing whatever it can to
please all operating system vendors especially now that AMD is getting
competitive again.

~~~
judge2020
My guess is Microsoft seeing how recently they're pouring a lot of resources
into the Linux subsystem and how they're trying to seem more "developer-
oriented" overall in recent months.

~~~
2trill2spill
That was my guess as well, also doesn't Microsoft run some Linux in their
Azure cloud? So that adds yet another incentive for Microsoft to help Linux
folks out.

Edit: I found a link confirming that Microsoft uses Linux for networking in
their Azure data centers [1].

[1]: [https://www.datacenterdynamics.com/news/microsoft-runs-
azure...](https://www.datacenterdynamics.com/news/microsoft-runs-azure-
networking-on-linux/)

~~~
notimetorelax
I work for Microsoft. We build customer facing business applications on Linux,
Java, Go, etc. Microsoft is Big!

~~~
2trill2spill
I'm sure Microsoft is doing this now, but wasn't that frowned upon before
Satya Nadella took over as CEO or was building on non Microsoft products okay
before?

~~~
mikekchar
I worked for a Corel at the turn of the century (ha ha, feels weird saying
that). We worked very closely with MS from about 2000 (when MS injected about
$130 million into the company). While Corel had their Linux distro, when I
worked there we were forbidden from using free software (and specifically GPL
software) other than on the Linux code. I heard it was due to an agreement
with MS (hearsay, but I believe it). I had my knuckles wrapped a few times for
disobediently editing my code using Emacs -- eventually I got used to Visual
Studio (and even learned to appreciate it to a certain degree).

Things started shifting around 2002 or 2003 and by the time Vector bought out
Corel in 2004, I was happily using Emacs and SVN with nobody batting an
eyelid. We also worked with MS to implement the shared source version of .Net,
which probably nobody remembers. It was supposed to be an "open" reference
implementation of .Net. Mono really took on that role, I think mainly because
the shared source reference implementation had a completely useless license.
You could see the wheels turning in the heads of the MS people on that
project. They were actually doing _really_ good work, but everyone knew that
the project was going to be meaningless.

IMHO, it was never a kind of binary switch from "open source is cancer" to "we
can make money through open source". It was more of a slow internalisation
that open source was a better business model for a lot of stuff that MS was
doing. It may be that the switch in CEO helped that transition, but it was
clear (at least from my perspective) that the wheels were already in motion
for a long time prior to that.

I'm sure there are a lot of people still in MS that are rabid about having to
control every last scrap of their "IP", but as MS starts to solve some of
their revenue problems with open source solutions I think these people will
lose relevance. I don't know if the corporate culture of MS (or even most
large organisations) will ever get to the point of embracing free software
ideals, but at least they seem to see the advantages of engineering
collaboration in some circumstances.

~~~
int_19h
It was a slow transition even before Satya - remember the Ms-PL license, and
ASP.NET MVC being open sourced under that with great fanfare? Or IronPython
and IronRuby? The difference before and after was that before, it felt like
carefully controlled and _contained_ experiments, that were still treated as
decidedly different and unusual on the inside, and on which the plug could be
pulled at any moment.

After, it was like a flood with the gates opened. We went from spending a year
to get approval to use Boost (and failing!), to a streamlined approval system
for pretty much any piece of OSS out there. It's very visible in Visual Studio
if you look closely at the files installed - compare, say, VS 2012 to VS 2015,
and note how many more OSS bits are in the full install of the latter.

The same goes for releasing under OSS. You might notice that a lot more
developer tools are OSS these days - even many new bits written for closed-
source products like VS. Hell, I shipped some Microsoft code under GPLv2 a
couple years ago - and it was easier to do than getting Boost approved under
the old regime!

So I think it's fair to say that Satya and his cadre of execs did drastically
re-imagine the company in that regard, rather than just finalizing an existing
process.

------
snaky
BTW

> Experts called for a new generation of secure-by-design computers at the Hot
> Chips conference here. In small steps in that direction, Microsoft and
> Google described their separate but similar hardware security architectures.

[https://www.eetimes.com/document.asp?doc_id=1333616](https://www.eetimes.com/document.asp?doc_id=1333616)

~~~
y-c-o-m-b
I wonder how many NSA back-doors will go into those.

~~~
snaky
The vast majority of regular users would _voluntarely_ install a dozen of NSA
backdoors on their computers if it guarantees them full protection from script
kiddies, phishing, malware and ransomware.

~~~
TheJoYo
I feel like you just described Google.

~~~
snaky
I feel that Apple would fit the description much better.

------
AdmiralAsshat
Greg's response is understandably frustrated, though seemingly less so than
the OpenBSD devs. Why are they being repeatedly left out of the loop?

~~~
amaranth
OpenBSD has been accused of breaking embargoes in the past. They are pretty
open about their policy of pushing their fix as soon as it's ready and not
doing anything to obfuscate what they're fixing and why.

~~~
benchaney
> They are pretty open about their policy of pushing their fix as soon as it's
> ready and not doing anything to obfuscate what they're fixing and why.

This isn't true at all. Please stop spreading misinformation.

~~~
occams_chainsaw
Neither of you have provided citation or reasoning. You basically just said
"nah uh"

~~~
chris_wot
Perhaps, but it rather behooves the one making the accusation to substantiate
it with evidence. I don't see how the one claiming that they don't leak can
provide evidence of not leaking!

~~~
euyyn
What is the reason then? (And what has led people to have that perception?)

------
rrix2
> "Intel has gotten better at this," he said.

Someone should let the BSD folks know and see what they think...

~~~
cperciva
My understanding is that they've gotten better. FreeBSD has had advance notice
of some issues. Last I heard they offered to let OpenBSD in too, but hadn't
found anyone willing to sign an NDA.

~~~
sverige
I'm reasonably certain that OpenBSD has never agreed to any NDA from anyone as
a matter of principle. It's one of the things that makes me love the project
so much.

~~~
cperciva
I don't know if "never" is accurate, but certainly they are very NDA-averse.
That's their right, but it means they're going to get left out of things like
this; it's simply not possible to organize _coordinated_ disclosure of issues
if the participants don't agree to not blab ahead of the agreed disclosure
date.

~~~
dsymonds
An NDA is a legal agreement. It's entirely possible to organise coordinated
disclosures without a legal agreement. The folks pushing NDAs, however, don't
seem to be interested in other sorts of agreements.

~~~
bloomer
The alternative would be a "gentleman's agreement"? An NDA would seem to be
much more transparent with everyone understanding what was agreed to rather
than something agreed upon over cigars and cognac. Refusing to sign NDAs as a
matter of principal doesn't seem like a very mature way to conduct business.

~~~
dsymonds
It doesn't have to be a handshake and a nod. Things can still be clearly
written down. But formal contracts with consequences take it up a notch. And
this isn't about how you "conduct business"; that's a very business-oriented
view of what's going on.

------
cperciva
While I agree that Intel's response was far from ideal, I find it a bit rich
for Linux kernel developers to be criticizing them. Remember, the completely
uncoordinated disclosure happened because Linux kernel developers started
discussing the vulnerability -- while under NDA -- on a public mailing list.

~~~
NullPrefix
You sure those devs were under NDA?

------
dschuetz
Because Intel is _that_ good in security and hardening, why not use Intel's
newly minted own special secure Linux distribution?

[https://01.org/blogs/imad/2018/letter-
industry](https://01.org/blogs/imad/2018/letter-industry)

This is almost like satire.

~~~
smhost
Not sure what you mean. This just looks to me like they're admitting they're
awful at security and they're pleading/threatening the global community to
help secure their ecosystem or else you're all going to be in a ton of trouble
because the robots/cars are going to kill you and your workers and what'll you
do then?

~~~
benchaney
> This just looks to me like they're admitting they're awful at security and
> they're pleading/threatening the global community to help secure their
> ecosystem

That post seems to make it pretty clear that they are trying to position
themselves as the experts rather than the people asking for help. That is the
part that is absurd.

------
0xFFC
I was in the room. Greg specifically said “off the record”.

~~~
TillE
That's not a magic phrase, it's something you negotiate with reporters in
advance. You can't really make "off the record" comments in front of an
audience at a conference.

~~~
0xFFC
Did I say it is a magic phrase? It used to say when you want something not
reported. You should definitely get familiar with journalism terminology.

~~~
wtallis
> Did I say it is a magic phrase?

Pretty much, yeah. You did. Pretending that declaring "off the record" during
a presentation like that falls anywhere near normal journalistic practice is
essentially the same thing as imputing magical properties to that phrase.
Confidentiality needs to be negotiated ahead of time, usually with an NDA or
embargo agreement before journalists are given any detail. When presenting at
a conference that lets journalists attend for free, it's unreasonable to
expect any confidentiality for the content of your presentation unless your
presentation is part of a closed session that the free media passes don't
grant access to.

------
docker_up
What is the current status of this? The last time I heard, the OS fixes would
impact CPU performance by 30%. Is this still the case? Will new iterations of
Intel CPUs be immune to this, or is this an ongoing issue going forward
because it's inherent with the architecture?

~~~
hansendc
Disclaimer: I work on Linux at Intel.

Future hardware will be mitigated against side-channel issues. There's a nice
table showing how things are mitigated on future processors here:

[https://www.tomshardware.com/news/intel-cascade-lake-
details...](https://www.tomshardware.com/news/intel-cascade-lake-details-
spectre-meltdown,37674.html)

But, not everything is mitigated in the silicon or microcode. The mitigation
for Spectre / Variant 1 / Bounds Check Bypass, for instance, will continue to
be in software.

------
pmontra
> While there have been many patches made in Linux, he strongly advised users
> to update with Intel's microcode fixes as well, as they provide an
> additional layer of protection beyond what an operating system can provide.

I found this page explaining how to do it
[https://www.cyberciti.biz/faq/install-update-intel-
microcode...](https://www.cyberciti.biz/faq/install-update-intel-microcode-
firmware-linux/)

I checked and my microcode is from 2018-01-21. Being on a 2014 laptop it means
it got updated by the package manager.

------
2bluesc
Any videos of the presentation?

~~~
mikeyouse
No video as far as I know, but he did post the slides with some backup
materials on his GitHub:

[https://github.com/gregkh/presentation-
spectre](https://github.com/gregkh/presentation-spectre)

One note from his talk though, his slides say that Foreshadow was fixed in
April -- but it's clearly supposed to be August. He mentioned he was fixing
the updates even on the flight out to OSSNA.

------
bigDeal
Does anyone honestly buy into the idea that this is something other than a
back door, carefully plotted by certain nation-state actors?

For real.

~~~
bArray
The creators aren't always aware of the monsters they create. I think Intel
was caught with their trousers down - AMD and ARM in many respects too.
Hopefully this will lead to automated testing procedures for Intel, other
processor manufacturers and security researchers.

The reason these flaws were found in the first place were because of security
researchers, hopefully they can now make their cases to get large research
grants and bring about a more secure future.

On the other hand, if you want to talk about tin-foil hat "CPUs working
against users", only look so far as Intel ME and the equivalents on other
processors. Closed-source and highly privileged code running all the time
inside the CPU - it's not unthinkable that Intel would bend the knee to some
state actors, especially if that leads to higher profits (i.e. a tax break or
entry into a new market ( _cough_ China)). It's enough of a concern that a lot
of reverse engineering effort has gone into preventing it from running.

~~~
acdha
> Hopefully this will lead to automated testing procedures for Intel, other
> processor manufacturers and security researchers.

They have tons of testing already. The problem is that this is the kind of
problem which is easy to miss with tests, especially automated ones, because
everything worked correctly and no real code would ever have been affected by
the side effects.

~~~
bArray
This is what I meant and didn't say - it should read:

Hopefully this will lead to _better_ automated testing procedures for Intel,
other processor manufacturers and security researchers.

The emphasis on "better" as there is obviously automated testing in existence
already.

~~~
acdha
I’m sure it will but I wouldn’t underestimate the difficulty of finding
significant unintended state changes in something on the order of complexity
of a modern CPU.

