
Disable SMT/Hyperthreading in all Intel BIOSes - carlesfe
https://marc.info/?l=openbsd-tech&m=153504937925732&w=2
======
walterbell
Does this mean AMD hyperthreading has a performance + security advantage over
currently shipping Intel processors?

Edit: [https://www.amd.com/en/corporate/security-
updates](https://www.amd.com/en/corporate/security-updates)

 _> 8/14/18 – Updated: As in the case with Meltdown, we believe our processors
are not susceptible to these new speculative execution attack variants: L1
Terminal Fault – SGX (also known as Foreshadow) CVE 2018-3615, L1 Terminal
Fault – OS/SMM (also known as Foreshadow-NG) CVE 2018-3620, and L1 Terminal
Fault – VMM (also known as Foreshadow-NG) CVE 2018-3646, due to our hardware
paging architecture protections. We are advising customers running AMD EPYC™
processors in their data centers, including in virtualized environments, to
not implement Foreshadow-related software mitigations for their AMD
platforms._

For those on AMD platforms, how do you disable software mitigations for
Foreshadow? Is this automatically done by browsers, operating systems and
hypervisors?

~~~
tareqak
With the newer AMD processors having as many real cores as they do, does the
cost-benefit analysis of HT/SMT change? I read in a comment here a few weeks
ago that turning it off on the newer AMD CPUs can yield better performance
because of improved cache-coherency on some workloads (My memory of what I
read might be totally wrong).

~~~
eklitzke
I'll defer to whatever the benchmarks say of course, but I don't see why HT
would affect cache coherency for normal workloads. If you disable HT you'd
still have the same number of threads/processes running on the system, so you
still have to schedule the same amount of work and do the same number of
context switches.

~~~
barrkel
Suppose you have 8 runnable processes and a 4 core / 8 thread system.

With 4 execution units, 4 processes run at a time, and they all get swapped
out every scheduling tick (losing all their cached lines). OTOH each process
gets a full measure of cache to use during its slice.

With 8 execution units, 8 processes "run" at a time, they interleave based on
stalls and CPU resources, and the OS doesn't need to reschedule anything every
tick (so they hopefully keep their cache lines hot). But each process gets a
half measure of cache to use.

In reality, code tuned to use a full measure of cache will be better off
matching the number of processes to the number of execution units available,
so you'd run half the number of processes with HT disabled. And cache-tuned
code tends to fall off a performance cliff when it exceeds cache available, so
it may easily run more than twice as fast, depending on the work.

The win from HT depends on most code not being tuned to full measures of
cache, and having a lot of memory stalls or other heterogeneous work that
other work can fit into. And most code is like that. Cache tends to have a
declining marginal return - you have to add exponentially more cache to avoid
cache misses -
[https://en.wikipedia.org/wiki/Power_law_of_cache_misses](https://en.wikipedia.org/wiki/Power_law_of_cache_misses)
.

------
exrook
For anyone (understandably) confused about the attacks and mitigations related
to L1TF, I've found the linux kernel documentation on the mitigations[0] to be
a great resource.

One interesting thing is that to mitigate L1TF hyperthreads only need to be
disabled if you are running VMs, the userspace mitigations are effective
regardless of HT status. However, there's a catch, you can leave
hyperthreading enabled if you disable the Extended/Nested Page Table
virtualization feature. However it is noted that this will result in a
significant performance impact.

[0] [https://www.kernel.org/doc/html/latest/admin-
guide/l1tf.html](https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html)

However this does not mean that HT with VMs is totally secure, as there may be
more vulnerabilities relating to HT yet to be disclosed/released as alluded to
by Theo. (For context, see the previous discussion [1] around the Lazy FPU
switching vulnerability where Theo made the decision to enable mitigations in
OpenBSD[2] prior to the public disclosure of the bug based (Theo/OpenBSD was
_not_ party to the embargo))

[1]
[https://news.ycombinator.com/item?id=17304233](https://news.ycombinator.com/item?id=17304233)

[2] [https://marc.info/?l=openbsd-
cvs&m=152818076013158&w=2](https://marc.info/?l=openbsd-
cvs&m=152818076013158&w=2)

------
reacharavindh
We have disabled Hyper Threading(HT) on all public facing servers(running
OpenBSD). However, our compute nodes running Linux kernel are benefiting about
80 to near 100% boost for specific scientific workloads. So, we run our
INTERNAL NETWORK ONLY compute nodes with HT on. In places where security is
not primary concern, why not make use of HT for extra efficiency?

Think and plan before you blanket disable HT on all servers running intel
CPUs...

If an attacker is able to run any code on these private servers, I have bigger
problems to deal with than HT as attack vector..

~~~
hermitdev
If you fully trust the software you're running, I see no reason to disable HT.
At this point, I don't think I'd have it running on anything publicly facing,
though. That said, I still have it enabled on my work PC & home PC.

~~~
firethief
Do you fully trust all the JavaScript you run?

~~~
hermitdev
Nope, which is why I generally have JS disabled.

------
Roritharr
Does that mean hyperthreading is effectively unpatchably insecure?

Cloud Providers are gonna have a bad time if this is true.

~~~
geofft
You can give both hyperthreads in a physical core to the same tenant, no?

Scheduling different VMs to run on the same hyperthreaded core at once seems
like it can't be good for either VM's performance, even if there were no
security concerns. Hyperthreading is much more useful for running multiple
threads of the same app, accessing similar instruction caches etc.

(There's also a question of safety _within_ the VM, but a huge number of cloud
users are running effectively one user within their VM.)

~~~
Roritharr
The latter question is very important indeed. If you for instance render
websites in your vm they, if i understand correctly, can potentially read
secrets from other processes, like db credentials and other stuff...

If the only real solution is to turn off HT/SMT that, seen positively, should
net us a lot faster VMs then...

~~~
gruez
>If the only real solution is to turn off HT/SMT that, seen positively, should
net us a lot faster VMs then...

you also doubled the cost of each VM (in terms of cpu), but you didn't double
the performance of each VM, so it's a net negative.

~~~
Roritharr
It might be Intel in the end having to pay that cost...

------
geofft
> _SMT is fundamentally broken because it shares resources between the two cpu
> instances and those shared resources lack security differentiators._

I thought the root of one of the Foreshadow problems was that caches are
shared _across cores_ , and therefore even with hyperthreading disabled, you
still gain information about a process on another core. Am I misinterpreting
it?

It does seem like the paranoid thing to do is that each _socket_ gets to be
used by only a single user. (I half-jokingly suggested at work that we replace
our internal cloud with a Beowulf cluster of Raspberry Pis...)

It also seems like you could design OSes in a way which is more robust to
this, e.g., certain cores are only for the kernel and processes running as
root, and system calls are inter-processor interrupts, so privileged kernel
(or userspace root) data doesn't go into untrusted caches at all.

~~~
wmf
Foreshadow is caused by the L1 cache which is not shared across cores. It may
be only a matter of time before L3 attacks are discovered but I don't know of
any today.

~~~
geofft
Oh - I forgot the L1 cache isn't shared across cores. That makes sense,
thanks.

------
orblivion
Scary looking headline on a discussion forum with instructions to perform a
task that the average user would not really understand, with no explanation of
attack vector or even consequences for any user who doesn't want to take the
time (and energy, frankly at this point) to follow security news.

I'm pretty close to not caring anymore. I hope somebody figures out how to at
least fix the security news infrastructure, if fixing security is still a ways
off.

EDIT: Scratch that, I assume attack vector is a browser since they mentioned
JavaScript.

~~~
bcaa7f3a8bbc
This is not security news at all. This is Theo de Raadt's personal E-mail sent
to the OpenBSD development mailing list for system developers. It is never
intended for the consumption by the general public.

~~~
orblivion
Hah, okay. I'm not familiar with OpenBSD so I didn't know who Theo was. Well,
that would be good to know, let's say, on the HN headline.

Thanks!

------
jonny_eh
Does anyone know the best way to disable hyperthreading in OSX? The only thing
I found was this:
[https://www.whatroute.net/cpusetter.html](https://www.whatroute.net/cpusetter.html)

~~~
jamroom
I'm not sure if you are running OSX on a multi user server, but is this
something we need to worry about on our laptop? It's not clear to me what
needs to be done in a single user situation.

~~~
walterbell
The OpenBSD post references possible JavaScript (browser) attacks.

------
seiferteric
Maybe this is what finally gets me to upgrade from my ~2012 i7-3770. Not
because of performance improvements, but to avoid performance degradation from
all these security patches...

~~~
shittyadmin
I'm not so sure I'd bother, more of these attacks keep coming out, odds are
you'll just buy a CPU that'll be vulnerable to the next one. Maybe AMD would
save you from some.

------
wiz21c
FTA

>>> We are having to do research by reading other operating systems.

Si Intel cooperates with business partners like apple/windows and not with
open source. Does it mean that Apple and Windows can claim to be more secure
because they have access to the information needed to fix Intel's issues ?

~~~
wmf
Intel cooperates with organizations that obey embargoes and don't badmouth
their partners in public, like Red Hat, Canonical, and probably the Linux
Foundation. Intel does not cooperate with OpenBSD.

~~~
abcdef123xyz123
OpenBSD has never disobeyed an embargo. Argued for them to be reduced,
criticized them, but not disobeyed them.

~~~
DannyBee
i'm pretty sure this is not true: The most recent example i remember is:
[https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbh...](https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbhnfz)

~~~
capt8bit
This is common misinformation. Even in this case, OpenBSD did not break the
embargo. After protesting, they received the permission of the researcher to
publish:

    
    
      Note that I wrote and included a suggested diff for OpenBSD already, and that
      at the time the tentative disclosure deadline was around the end of August. As
      a compromise, I allowed them to silently patch the vulnerability.
    

([https://www.krackattacks.com/#openbsd](https://www.krackattacks.com/#openbsd))

------
bonsai80
There are some comments here talking about the possibility of getting better
performance without HT. Here's an article from a test (on Intel only) of that
theory: [https://www.phoronix.com/scan.php?page=article&item=intel-
ht...](https://www.phoronix.com/scan.php?page=article&item=intel-ht-2018)

In the end, the conclusion is: "Long story short, Hyper Threading is still
very much relevant in 2018 with current-generation Intel CPUs. In the threaded
workloads that could scale past a few threads, HT/SMT on this Core i7 8700K
processor yielded about a 30% performance improvement in many of these real-
world test cases."

------
fyfy18
Will be interesting to see what Apple does about this in their next software
update. I can’t imagine many people will be happy if the next software update
forcibly disables hyper threading.

(For those who aren’t familiar with Apple devices, Apple don’t expose settings
like this to a user, which are usually available in the BIOS on a PC)

~~~
nine_k
Will the loss of HT have apparent consequences on most Intel-based Apple
hardware? Very few of them are servers under constant multithreaded load,
throughput-oriented. I suppose almost all Macbooks and most Mac Pros will not
visibly slow down.

~~~
Marsymars
I would expect disabling HT on dual-core systems to have a noticeable
performance impact on the desktop.

------
kevin_thibedeau
It's not clear that he's using "SMT" to refer to AMD specifically as he goes
on to talk about "Intel CPUs" and disabling it in "Intel BIOS". Does the Zen
architecture have the same issue?

~~~
gaius
I don't think AMD is impacted by Foreshadow

------
swixmix
[ marc.info not responding for me, found post linked elsewhere. Edited due to
markdown.

URL: [http://openbsd-archive.7691.n7.nabble.com/Disable-SMT-
Hypert...](http://openbsd-archive.7691.n7.nabble.com/Disable-SMT-
Hyperthreading-in-all-Intel-BIOSes-tp349646.html)

Here it is: ]

\---

Title: Disable SMT/Hyperthreading in all Intel BIOSes

Posted by Theo de Raadt-2 on Aug 23, 2018; 11:35am

Two recently disclosed hardware bugs affected Intel cpus:

\- TLBleed

\- T1TF (the name "Foreshadow" refers to 1 of 3 aspects of this bug, more
aspects are surely on the way)

Solving these bugs requires new cpu microcode, a coding workaround, _AND_ the
disabling of SMT / Hyperthreading.

SMT is fundamentally broken because it shares resources between the two cpu
instances and those shared resources lack security differentiators. Some of
these side channel attacks aren't trivial, but we can expect most of them to
eventually work and leak kernel or cross-VM memory in common usage
circumstances, even such as javascript directly in a browser.

There will be more hardware bugs and artifacts disclosed. Due to the way SMT
interacts with speculative execution on Intel cpus, I expect SMT to exacerbate
most of the future problems.

A few months back, I urged people to disable hyperthreading on all Intel cpus.
I need to repeat that:

DISABLE HYPERTHREADING ON ALL YOUR INTEL MACHINES IN THE BIOS.

Also, update your BIOS firmware, if you can.

OpenBSD -current (and therefore 6.4) will not use hyperthreading if it is
enabled, and will update the cpu microcode if possible.

But what about 6.2 and 6.3?

The situation is very complex, continually evolving, and is taking too much
manpower away from other tasks. Furthermore, Intel isn't telling us what is
coming next, and are doing a terrible job by not publically documenting what
operating systems must do to resolve the problems. We are having to do
research by reading other operating systems. There is no time left to backport
the changes -- we will not be issuing a complete set of errata and syspatches
against 6.2 and 6.3 because it is turning into a distraction.

Rather than working on every required patch for 6.2/6.3, we will re-focus
manpower and make sure 6.4 contains the best solutions possible.

So please try take responsibility for your own machines: Disable SMT in the
BIOS menu, and upgrade your BIOS if you can.

I'm going to spend my money at a more trustworthy vendor in the future.

------
ianai
Makes sense from the adage that more complexity means more exploits.

------
bcaa7f3a8bbc
Does anyone know the best way to disable hyperthreading on Linux?

~~~
bcaa7f3a8bbc
Okay, I found it. A SMT knob was added alongside in the L1TF fixes.

    
    
        /sys/devices/system/cpu/smt
    
        /sys/devices/system/cpu/smt/active
    
        /sys/devices/system/cpu/smt/control
    
        active:  Tells whether SMT is active (enabled and siblings online)
        control: Read/write interface to control SMT. Possible
    
        values:
    
        "on"		SMT is enabled
        "off"		SMT is disabled
        "forceoff"   	SMT is force disabled. Cannot be changed.
        "notsupported"	SMT is not supported by the CPU
    
        If control status is "forceoff" or "notsupported" writes are rejected.

------
mehrdadn
I would love to but how in the world do I do this when I'm using Windows and
Lenovo doesn't give me an option in the BIOS?

~~~
JBiserkov
One way to achieve something similar would be via a software tool which would
set the process affinity to only run on real cores.

Or you could only run Chrome (untrusted JavaScript) on core 2 and 3, and run
the app that has your secrets on core 0 and 1. (It is my understanding that 2k
cores are real, and 2k+1 is their matching, "virtual" core) This way you get
both hyperthreading and security. I'm not a security expert though.

[https://bitsum.com/docs/pl/Using%20the%20GUI/using_the_gui.h...](https://bitsum.com/docs/pl/Using%20the%20GUI/using_the_gui.htm#default_affinities)

~~~
mehrdadn
I'm not sure it would be that easy since I believe e.g. I/O can go through the
System process (or other processes even), which has full affinity. We'd likely
have to set thread affinities for all processes/threads. But then it would
clash with manually-set affinities, and I'm also not sure if it would have
worse performance than actually disabling hyper-threading or not.

Right now I'm looking at what making a UEFI application to disable HT before
boot might involve... not sure if that's too late in the boot process or not.

------
AdmiralAsshat
So, realistically, how much performance did your average OpenBSD server just
lose from following this mitigation?

~~~
ams6110
Performance is not a top priority for OpenBSD.

If you read
[https://www.openbsd.org/goals.html](https://www.openbsd.org/goals.html), the
word "performance" does not appear.

~~~
kibwen
I think this may be missing the point of the grandparent comment; rather than
interpreting it as an accusation of OpenBSD sabotaging its users' performance,
I think we're all just curious at the relative importance of hyperthreading
for real-world workloads, on any OS, in grim anticipation of the potential
worst-case scenario where hyperthreading's security woes continue to worsen
and worsen.

------
messe
Does anyone know if disabling SMT has had an effect on vmd(8) performance in
-current?

------
codedokode
Is not it better just to disable Javascript instead of cutting your CPU
performance?

~~~
brianwawok
Some of us like using websites.

~~~
gpderetta
Many actually improve when you disble JS

~~~
Lev1a
But a non-zero amount will completely break or not show anything at all (even
though the actual content to be displayed is just static text and doesn't need
_any_ JS)

------
JudasGoat
Can I ask, Is Intel going to sue us if compare benchmarks with SMT to no SMT?
Like they mentioned about the microcode benchmarks. Intel is in trouble.

------
purplezooey
Eh hyperthreading seems to be questionable anyway. On all the application
benchmarks I've run except for a few, it slows things down.

~~~
int0x80
Care todo share ir provide more specific info?

------
stcredzero
I've long felt that there's something less than half-baked about the multi CPU
architecture we're currently using. The hacky contortions HFT coders have come
up with to avoid things like False Sharing strike me as a big red flag.

[https://mechanical-sympathy.blogspot.com/2011/07/false-
shari...](https://mechanical-sympathy.blogspot.com/2011/07/false-sharing.html)

How about an architecture more like Erlang's, where you have independent
processes with their own CPU core, where each has their own memory, but where
you have much faster communications supported at lower hardware levels? Why
not have a multi-processor architecture designed for direct support of Hoare
CSP-inspired languages?

Hypercube topology:
[http://web.eecs.umich.edu/~qstout/pap/IEEEM86.pdf](http://web.eecs.umich.edu/~qstout/pap/IEEEM86.pdf)

~~~
wmf
That's like the Cell processor. For every year that you program for Cell you
need at least two years of therapy.

~~~
stcredzero
If you're going to change the substrate or paradigm, then you need to do a
dynamite job of supporting your users. Sony did not do that.

