
Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower - zrm
https://www.phoronix.com/scan.php?page=article&item=linux-420-bisect&num=1
======
schoen
In case people don't like the headline style (leaving out the reference to the
answer), the answer given at the end of the article is new Spectre
mitigations.

We new that speculative-execution mitigations would sometimes reduce
performance, and evidently they do. :-(

------
bonzini
This was known.
[https://lwn.net/Articles/765837/](https://lwn.net/Articles/765837/) reports
21% performance impact and the plan was to handle STIBP as an opt-in security
feature. I am not sure why that was not furthered in the patches that have
actually been committed to Linux, as I did not follow them closely.

~~~
rwmj
Apparently the way to turn this and other mitigations off is the following
mouthful:

    
    
        pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier
    

Would it make sense to have a single flag to "run insecure but fast" that we
can use on pure development machines, test servers and the like? My Intel
development server only runs code I choose.

~~~
pixl97
Hey boss, I don't understand why the code works in devel but fails in
production?

~~~
CyberShadow
You're implying (with unnecessary snark) that:

1\. Development machines should be configured identically to production
machines. (Do you install a GUI and development tools on your production
servers?) Occasional differences in behavior between development/production
are par for the course, and is why staging environments are commonly used.

2\. The mitigations affect the execution result of code likely to be
developed/executed at a typical software shop. AFAIU the attacks are timing-
based, and won't affect valid code that's not specifically looking to exploit
them.

~~~
manquer
try convincing a corp sec dept in large org to sign off on disabling anywhere.

------
xbmcuser
Intel was supposed to have 20-30% advantage on single thread performance over
Amd. Now I guess single or multiple thread AMD is better on Linux. I don't
know why it makes me giddy thinking of the coming consumer wars between the 2.

~~~
rdtsc
This is how it should be. And this is going to net positive thing for AMD.

If performance takes a hit by about 20% it is going to be noticeable. Next up
I think people will start looking to upgrade to a faster CPU just to get back
to last week's performance metrics. They might now consider AMD.

If AMD finds this affects them less they should sponsor security researchers
writing proof of concept exploits to convince people that that turning off the
mitigation is an absolute no-no, so it will for force people to take a
performance hit and go shopping for CPUs.

------
neopallium
Is there a comparison table of CPU vulnerabilities? I would like to know which
CPU has the least vulnerabilities or the best performance after patching.

From what I have read so far is seems that AMD CPUs have had the fewest
vulnerabilities/slowdowns? But I can't be sure since I haven't seen a complete
comparison (including these new vulnerabilities).

~~~
roblabla
I don't know of any comparison table. There are two family of vulnerabilities
at play when talking about Spectre and co:

\- The general Spectre family, which affects most (if not all) CPUs built with
speculative execution.

\- Meltdown and L1TF, which only affects Intel CPUs due to them delaying
security checks until after speculation has taken place.

AMDs, ARMs, etc. that use speculative execution are going to be vulnerable to
at least some variant of Spectre (there are 4 variants known right now). ARM
published a table[0] explaining which of their CPUs are vulnerable to which
variants. I'm not aware of any such table for Intel or AMD.

Microsoft published some interesting tables[1] explaining which mitigation
protect against which Spectre variant, and under which thread models they
operate.

[0]: [https://developer.arm.com/support/arm-security-
updates/specu...](https://developer.arm.com/support/arm-security-
updates/speculative-processor-vulnerability)

[1]:
[https://blogs.technet.microsoft.com/srd/2018/05/21/analysis-...](https://blogs.technet.microsoft.com/srd/2018/05/21/analysis-
and-mitigation-of-speculative-store-bypass-cve-2018-3639/) (scroll down for
the tables)

------
pella
Latest test ( 17 November 2018 ) :

"The Spectre/Meltdown Performance Impact On Linux 4.20, Decimating Benchmarks
With New STIBP Overhead"

[https://www.phoronix.com/scan.php?page=article&item=linux-42...](https://www.phoronix.com/scan.php?page=article&item=linux-420-stibp&num=1)

------
Aardwolf
According to this [1], the coffee lake refresh has hardware mitigation to
meltdown variant 3 and variant 5 (which are only a few of all the things).

Would that hardware mitigation reduce this performance loss?

[1]: [https://www.anandtech.com/show/13400/intel-9th-gen-
core-i9-9...](https://www.anandtech.com/show/13400/intel-9th-gen-
core-i9-9900k-i7-9700k-i5-9600k-review/2)

------
kim0
Is Intel already selling CPUs which don't need these patches?

~~~
anticensor
Not in x86 architecture.

~~~
eugeniub
Then in which architecture?

~~~
smitty1110
They still sell Itanian, I don’t think Spectre/Meltdown affect that product
line.

~~~
hermitdev
*Itanium, and they would likely suffer from the same attacks (maybe not due to being a completely different architecture, but being from thw same company, not sure if there isn't crosspolination between the two), but dont think anyone has looked at them due to small market pressense.

~~~
lmkg
I would expect the opposite. Meltdown and Spectre both make use of peeking at
implicit state that chips hang on to in order to optimize microcode; the
entire goal of Itanium was to offload this optimization from the chip to the
compiler.

A little Googling found an analysis from someone more knowledgeable than
myself that backs this up: [https://secure64.com/not-vulnerable-intel-itanium-
secure64-s...](https://secure64.com/not-vulnerable-intel-itanium-
secure64-sourcet/)

tl;dr Meltdown takes advantage of out-of-order execution, which Itanium simply
doesn't have. Spectre makes use of speculative execution, which Itanium only
has in a version too limited to support the attack.

I'm not an expert in this field so I can't attest to the credibility of this
analysis. The Spectre part in particular sounds a little hand-wavy but it
might just be over my head. But it aligns with my intuition, which is that the
simpler and more explicit architecture doesn't have as many places where data
can accidentally end up.

------
paavoova
This is somewhat unfortunate for older-gen hardware. The difference between
chugging-along but usable vs 20% reduced performance is crippling.

~~~
jammygit
I was thinking that it might be an overall large boon to Intel due to all the
new computers everyone will buy...

------
_cs2017_
1.3-1.4x slowdown is a lot more than I expected (I know it's for synthetic
benchmarks but still...)

Can someone explain (or link to an article) how a tweak to HT branch
prediction heuristic can have such a huge impact on performance?

~~~
BeeOnRope
The impact is big enough that one would suspect the microcode simply disables
indirect branch prediction, so you pay a 16-20 cycle penalty per branch.
Indirect branches just aren't frequent enough to explain such a regression via
say a simple reduction in prediction resources.

I can test it once I get the new firmware.

~~~
jcelerier
> Indirect branches just aren't frequent enough

aren't vtable calls / function pointer calls indirect branches ?

~~~
BeeOnRope
Yes, they are (at least when the compiler cannot devirtualize them) - but they
make up a fairly small fraction of the total instructions in a typical program
- and probably very small in something like cinebench, which also showed a big
regression.

~~~
jcelerier
> but they make up a fairly small fraction of the total instructions in a
> typical

but if their cost increases by a large factor... besides, in any large
compiled program, the core would certainly be based around some kind of
programmable pipeline, and these would generally be implemented like this
unless they wrote their own JIT compiler.

~~~
BeeOnRope
Yes - but for their cost to increase by such a large factor, the only obvious
thing I can think of is that their prediction is disabled.

I didn't follow your comment about a "programmable pipeline". I don't think
many or any of the Phoronix benchmarks are based on a pipeline with indirect
branches at their core.

~~~
jcelerier
> I don't think many or any of the Phoronix benchmarks are based on a pipeline
> with indirect branches at their core.

I think a bunch are. e.g. for instance FFMPEG / libavfilter which is basically
a node graph set up at runtime. Don't know for cinebench since it's closed
source, but Blender present in the benchmarks is also based around a nodal
rendering architecture. Stuff like PHP / CGI also heavily depend on function
pointers for their behaviours - PHP with its plugin architecture, and CGI
where all web requests go through FPs : [https://github.com/php/php-
src/blob/master/main/fastcgi.c#L8...](https://github.com/php/php-
src/blob/master/main/fastcgi.c#L880).

~~~
BeeOnRope
Right, I think I understand what you are saying about "pipelined"
implementations. Sure, I can believe that at a high level there are some
indirect branches to implement some kind of processing pipeline: but you'd
have thousands or millions of instructions doing the heavy lifting for each
chunk of data that passes though the pipeline, for every branch that you need
to take to get to the next stage.

So I still doubt that that indirect branches are "dense" in those benchmarks:
it just doesn't make sense since the core work they are doing are highly tuned
encode/decode/whatever kernels, even if there is a control layer over top of
that using indirect branches.

------
dev_dull
The lede is literally the last line in the article:

> _But why is it slower? More work on f &#!_#(# Spectre! _

~~~
orf
There is a second page:
[https://www.phoronix.com/scan.php?page=article&item=linux-42...](https://www.phoronix.com/scan.php?page=article&item=linux-420-bisect&num=2)

------
esotericn
No mention of the fact that AMD seem unaffected?

~~~
mkl
It's mentioned in two different places in the article.

~~~
esotericn
Indeed, though the headline seems kind of odd to me given that it's
specifically an Intel problem as far as I can tell, it'd be nice to have
_some_ mention in there (even if people following this can guess anyway).

------
rayiner
I don't like how these Scepter mitigations are being rolled out with no
cost/benefit analysis. On a client machine, I'd rather just disable Javascript
than pay a 30-50% performance penalty for mitigating these vulnerabilities.

~~~
CamperBob2
I would rather wait for reports of dangerous JavaScript exploits in the wild,
myself. Considering that no real-world exploits of Spectre and related attacks
have actually been reported, the odds that I'll fall victim to such an exploit
without hearing about it in plenty of time to mitigate it are in the millions
to one.

Instead, everyone has to pay the performance tax proactively, regardless of
their individual threat model. That is not how this sort of thing should be
handled.

~~~
frostburg
Would you like a nuclear powerplant to be ran with this kind of approach to
safety? Given the amount of possible targets and how unreliable people can be
at assessing what risk they're exposed to, having mitigation active by default
is obviously the only sane answer.

~~~
NewsAware
Somehow I doubt nuclear power plants run (unmodified) Linux kernels released
in the last 5 years

~~~
64738
Exactly... they're running on XP so it's all good

~~~
rleigh
When I went to Sellafield they had Win3.1 on the fuel rod chopping machine in
the reprocessing plant (admittedly in '97!)

------
userbinator
If you could turn off speculative execution completely I bet that would
eliminate all the security concerns, but performance would be abysmal. The
Linux kernel really can't be a "one size fits all" type of thing --- an
environment where no mutually untrusting code is run has very different
security requirements from shared cloud hosts, for example. Personally I think
it's all a bit overblown, probably due to owners of the latter environments.

------
ToFab123
Why is so much code being added to a kernel?

<terms of lines of code changed with more than 354 thousand lines of new code
added at the end of October when this merge window opened.

~~~
azahk
Someone once had the great idea that drivers should be in the same codebase as
the kernel. Crazy, I know.

~~~
umanwizard
It doesn’t seem crazy to me that something that runs in kernel mode and uses
kernel APIs should be in the kernel codebase. What are the downsides?

~~~
quotemstr
Why should drivers necessarily run in the kernel? With appropriate abstraction
for interrupts, DMA, etc. much of the work could be done in userland.

~~~
umanwizard
That’s true, but it’s AFAICT a separate question.

The question I was responding to was: “accepting that, rightly or wrongly,
Linux drivers run in kernel mode, does it make sense for them to be in the
source tree?”

------
maccio92
Big win for AMD!!!

~~~
beatgammit
It's times like this when I'm glad I went with a Ryzen chip instead of
whatever Intel has on offer. Not only do I get a ton of cores, but now my chip
performs better than the competition!

The only real problem I've had has been with soft lockups, which I was able to
solve by disabling a power saving feature on the CPU. If AMD doesn't mess
things up, they can really catch up to Intel in the next couple years while
Intel is working on fixing their issues.

------
seiferteric
Think I will wait at least 1-2 more years for all this to shake out before I
buy a new computer.

~~~
ams6110
You'll get a much better deal anyway if you do that. I never buy the current
generation of hardware. Much better value per dollar to buy one or two
generations old. Unless you absolutely need bleeding edge performance.

------
tedunangst
Does the article not actually mention the reason? (I mean, I know it's STIBP,
but it never says that?

~~~
detaro
it's on page two. Why a short article like this has pages though...

~~~
richrichardsson
I saw the social buttons and assumed end of article, very bad design.

~~~
jccalhoun
me too!

------
detaro
previous:
[https://news.ycombinator.com/item?id=18474289](https://news.ycombinator.com/item?id=18474289)

------
azahk
I assume this is not a problem if I disable hyperthreading?

~~~
vbezhenar
Probably you'll lose more performance.

~~~
drb91
As always, this depends on your workload.

~~~
azahk
As of today still most of the software I use seems to get stuck in one core of
my CPU when calculating stuff so I'd argue that by enabling HT I'm _losing_
performance.

~~~
vbezhenar
That's not really clear from the article, but I suspect that performance drop
is from multithreaded software. And sure, if you are observing performance
degradation with hyper-threading, there's 0 reason to keep it enabled. OpenBSD
guys suggest to turn it off after all.

~~~
hermanradtke
The article claims a 30% perf slowdown with PHP. The majority of PHP workloads
are single threaded. That causes me to suspect this impacts single core
workloads too.

~~~
bonzini
Yes, on some processors basically it's _as if_ all userspace was recompiled to
use retpolines. Enabling STIBP promises that one thread cannot influence the
indirect branch predictor's operation when the sibling runs; on those
processors the indirect branch predictor is completely disabled, which is an
inefficient but valid way to implement STIBP semantics.

------
bronzeage
I'm just sitting here waiting for all the security fiasco to stop ruining
performance for ridiculous unpractical attacks. Just a matter of time, the
pressure for better performance in a world where moore's law is getting stuck
will eventually cause every sane person to throw these ridiculous patches out
of the window.

If the choice is between unsecure but fast to secure and slow, the default
should be fast. Let applications who care about side channel deal with it
their way, why the fuck should everything be so slow just so that few apps
which actually care about side channel will be secure?

~~~
gizmo686
And I'm sitting here waiting for the industry to start taking security
seriously. These are not ridiculously unpractical attacks. These are attacks
that have been hypothesized for years and are only now getting attention
because people finally started to create (public) demonstrations. I would be
shocked if there were not already non-public exploits based on these
vulnerabilities.

The only reason these attacks even have the appearance of being unpractical is
because there are so many other areas of our computing systems that are even
easier to attack.

The only reason we are in this current Spectre et. al. mess is that CPU
vendors choose to prioritize performance over security, and we now have to
live with their bluff getting called for another hardware generation or two. I
suspect that the performance impact will be far less when the fixes are done
during the hardware design process, instead of figuring out how to bolt them
on after the fact.

Ultimately, I think we are overdo for a new computer architecture (both in
terms of security and performance, we are being encumbered by the need to keep
new architectures backwards compatible with old ones).

