
OpenBSD disables Intel's hyperthreading due to security concerns - mereel
https://www.mail-archive.com/source-changes@openbsd.org/msg99141.html
======
jimrandomh
> We really should not run different security domains on different processor
> threads of the same core. Unfortunately changing our scheduler to take this
> into account is far from trivial.

This suggests a long-term compromise solution where threads within a process
can use hyperthreading to share a core, but threads in different processes
can't. Given that hyperthreads share L1 cache, this might also be better for
performance.

~~~
classichasclass
I'm not sure that would necessarily fix the problem definitively. Say you had
a browser running web-exposed JavaScript on a thread. You could still finagle
a Spectre-type information leak that way by having the JavaScript thread snoop
other browser threads, assuming no other mitigations.

~~~
endianswap
Don't most browsers run one process per page/tab nowadays?

~~~
greglindahl
Chrome does, Firefox does not (I've got 5 processes for a billion tabs.)

~~~
valarauca1
Firefox process per tab is behind a feature flag as it’s in testing still

~~~
timvisee
I don't think the plan is to ever enable this in the comming few years. The
current approach with a few tabs is much more memory efficient, which is why
they've chosen it.

~~~
mtgx
And it's a mistake.

Just recently I noticed that when Firefox loads multiple tabs of the same
wordpress site, it starts hanging not unlike Firefox always used to hang.
That's likely because it groups all of those same site pages under one
process.

I've never experienced that with Chrome. This is why I hope Firefox eventually
(ASAP) switches to one process per tab, too. I can handle the browser using an
extra GB of RAM. I can't handle it hanging on me and frustrating me.

Instead of pushing for 30-40% lower memory than Chrome, I say they should push
for 10% lower memory with the same sanboxed process per tab model.

~~~
Amezarak
Chrome does not use one process per tab. In fact, it does something very
similar to what you say Firefox does.

[http://dev.chromium.org/developers/design-
documents/process-...](http://dev.chromium.org/developers/design-
documents/process-models)

FWIW I do not have the problem you describe and I don't want Firefox wasting
any more of my scarce memory, or for that matter, CPU.

~~~
Dylan16807
> very similar

Not really. Chrome uses a _lot_ of processes for isolation. Firefox uses about
four so it can take advantage of multiple cores.

------
Scramblejams
I've never trusted hyperthreading for workloads I haven't tested. Sometimes
it's faster, often it's slower. Beyond that, I've been suspicious of its
security implications from day one. My first trip through the BIOS on a
personal machine always includes turning it off.

~~~
ailideex
Can you give example of where it was slower for you with HT enabled?

~~~
Scramblejams
Running finite element models with MSC NASTRAN, basically heavy matrix math.
Matrices were NxN, N was around 10 million. This was on a server with 36 cores
and a half terabyte of RAM, purchased in 2014.

Also seen Erlang workloads where you could get a bit of throughput increase
with your VM scheduler scheduling more threads than your physical cores (so
starting to use HT) but the latency would spike and become very unpredictable,
which was a bad tradeoff for the use case.

------
keldaris
So... they "strongly suspect" (but don't know and haven't shown) there may be
a Spectre-class bug enabled by current HT implementations and improving their
scheduler is hard, so they'll pre-emptively disable HT outright on Intel CPUs
now and others in the near future?

I'm not an OpenBSD user (and glad for it, if this is anything to go by), but
I'm curious - is this really how they operate, or does this decision stand
out?

~~~
2trill2spill
> I'm not an OpenBSD user (and glad for it, if this is anything to go by), but
> I'm curious - is this really how they operate, or does this decision stand
> out?

I'm not a OpenBSD user either, I use FreeBSD whenever possible. However from
listening to OpenBSD devs, via blogs, conferences, HN, etc, it seems that
OpenBSD is an operating system built mainly for OpenBSD developers, their
goals support this[1]. OpenBSD being useful for non OpenBSD developers is more
of a secondary goal compared to how FreeBSD or Linux or any other OS handles
it. Also OpenBSD is much more of a research operating system then other large
successful OS(Linux, Windows, MacOS, FreeBSD, etc). Meaning OpenBSD cares way
more about developing features and novel security mitigations then trying to
maintain backwards compatibility like other operating systems do.

> So... they "strongly suspect" (but don't know and haven't shown) there may
> be a Spectre-class bug enabled by current HT implementations and improving
> their scheduler is hard, so they'll pre-emptively disable HT outright on
> Intel CPUs now and others in the near future?

The OpenBSD devs strongly suspected another Intel hardware bug a week or two
ago, implemented a mitigation and deployed it. Turns out they were right[2].

[1]: [https://www.openbsd.org/goals.html](https://www.openbsd.org/goals.html)

[2]: [https://www.bleepingcomputer.com/news/security/new-lazy-
fp-s...](https://www.bleepingcomputer.com/news/security/new-lazy-fp-state-
restore-vulnerability-affects-all-intel-core-cpus/)

~~~
bigato
> Also OpenBSD is much more of a research operating system then other large
> successful OS(Linux, Windows, MacOS, FreeBSD, etc). Meaning OpenBSD cares
> way more about developing features and novel security mitigations then
> trying to maintain backwards compatibility like other operating systems do.

This is not the feeling I get from OpenBSD at all. They don't act like
research. They aren't keen on implementing new features just for the sake of
it, or just to try it out. A better description would be that they put
correctness, security and maintainability first, and simplicity often comes as
a nice side effect. Deprecating old, unused features is just a consequence of
striving to decrease complexity by trimming your code base. OpenBSD is one of
the few OS where the number of lines of code is not skyrocketing to
unmanageable numbers.

------
GrayShade
There are some Linux HT benchmarks here:
[https://www.phoronix.com/scan.php?page=article&item=intel-
ht...](https://www.phoronix.com/scan.php?page=article&item=intel-
ht-2018&num=1)

------
mehrdadn
Do you get the exact same performance characteristics by ignoring the extra
virtual cores as you would have gotten if you could actually disable
hyperthreading in the CPU via the firmware setup? Or does it result in some
CPU resources becoming unusable that would otherwise be usable if HT were
truly disabled?

~~~
notaplumber
Operating systems can't disable HT/SMT in the same way as the BIOS/firmware
can, but presumably it will be fine if the kernel only schedules the idle
process on HT "cores", it will spend much of, or all its time in a lower power
state (MWAIT? C-states?), presumably the CPU is smart enough to handle that.

~~~
mehrdadn
I guess figuring out whether it's only "presumably" or actually "actually" was
why I asked the question in the first place.

~~~
notaplumber
Hmm. This thread on the OpenBSD lists suggests it may be adequate, of course
disabling in the BIOS is certainly a better option, if you can.

[https://marc.info/?t=152938773300027&r=1&w=2](https://marc.info/?t=152938773300027&r=1&w=2)

------
Someone1234
Ouch. I will say though, Hyper-Threading is a lot less valuable these days
than it was when it was first introduced (except for the few dual core CPUs
still available).

When you have four-six-eight or more cores, there's less value in doubling
that number. The gain is lower.

~~~
derekp7
On the other side, a hyperthreaded CPU used to be about 10 - 30% gain, but in
tests I've ran on recent hardware (HP DL380 Gen 10) hyperthreading gives
around 70% more performance (the test I used was running pigz [parallel gzip]
on a large file).

~~~
greglindahl
That's a great example of how hyperthreading's performance effects are
extremely workload dependent.

------
classichasclass
The implication seems to be that other architectures are also soon to have SMT
disabled by default. That would definitely hurt POWER, for example.

~~~
mrpippy
I think the only other OpenBSD architecture that supports any SMT chips is
sparc64 (like the US T1/T2). Unless an actual vulnerability is found, I don't
see other OSes following this lead

~~~
temprature
An "actual vulnerability" has been found. It's amazing that even after the
lazy FPU fiasco, people think OpenBSD did this on a complete whim.

~~~
mrpippy
I stand corrected. From the commit message this seemed much more speculative
than the FPU vulnerability (where Theo admitted to being tipped off by someone
under the embargo), but clearly it's more than just speculation.

[https://www.blackhat.com/us-18/briefings/schedule/#tlbleed-w...](https://www.blackhat.com/us-18/briefings/schedule/#tlbleed-
when-protecting-your-cpu-caches-is-not-enough-10149)

------
equalunique
I was going to submit this news from the source I learned it from, which has
the novel peculiarity of coming from a site that's name is similar to this
one: [https://thehackernews.com/thn/2018/06/openbsd-hyper-
threadin...](https://thehackernews.com/thn/2018/06/openbsd-hyper-
threading.html)

------
tynecomputers
Does anyone know when they are going to patch this or is it a permanent fix?

------
epynonymous
i didnt see this posed in the comments, but it was certainly tops on my mind.
is this the same issue for linux kernel?

~~~
petee
If they are using Hyper Threading, then yes, unless they already have a
different architecture:

 _" We really should not run different security domains on different processor
threads of the same core. Unfortunately changing our scheduler to take this
into account is far from trivial."_

~~~
aade
The (recent) SPARC Hypervisor does a fair job at this. Fujitsu has an
interesting implementation. But it would be conceivably difficult to do this
with time sharing on Intel chips without exposing side channels. That kind of
control should be supervisory and in control of the chip. I haven’t yet seen
that on Intel, but I’ve heard there are some hardware manufacturers that are
looking to do something like that.

------
kojon99
They should make it easier to find the diff behind all openbsd emails. I can’t
find this one.

~~~
foodstances
[https://github.com/openbsd/src/commit/96c11352863a7f6240b4e5...](https://github.com/openbsd/src/commit/96c11352863a7f6240b4e5e388052f414b75f95b)

------
DSingularity
Ouch. Huge hit for performance.

~~~
Forbo
From the commit message, it sounds like that might not necessarily be the
case: "Note that SMT doesn't necessarily have a posive effect on performance;
it highly depends on the workload. In all likelyhood it will actually slow
down most workloads if you have a CPU with more than two cores."

~~~
AHTERIX5000
Wonder why, because of poor SMP scalability and coarse locking?

I've encountered some cases where SMT made performance worse such as with very
optimized HPC libs but in general SMT can really help. Compiling projects got
a nice boost when enabling HT on Intel's recent arch for example (all of this
on Linux though, last time I checked OpenBSD its SMP perf was abysmal)

~~~
Uberphallus
For I/O and memory bound processes, it makes it worse, saturating further
buses that are already saturated. For regular. For CPU bound, it may help or
not, depending on many factors, like cache/memory contention, nature of the
operations...

Compilation can get boosts because while some threads are waiting for I/O
others are crunching source files. Also the variety of computation is high
enough so multiple threads don't overlap too much on functional unit usage. If
you try to build from a filesystem in memory, you'll find way a less
impressive speedup (if any).

~~~
AHTERIX5000
Yeah I've spent my time looking at perf counters and I'm aware how cache
access patterns affect. But the statement was drastic enough to make me
suspect there is something more OpenBSD specific going on.

------
creo
What scares me is that they do OS wide change based of wording "This can
make", "And since we suspect" and "In all likelyhood" instead of doing actual
tests. I know that open systems doesn't have required workforce, but doing
changes based on subjective reasoning is slippery slope.

~~~
flurrything
They care about making OpenBSD secure, not about producing security exploits.

Many OpenBSD devs are security researchers in academia. If they hear whisphers
over beers that there are new Spectre attacks coming that exploit this or
that, they might not be able to reproduce the exploit without putting a lot of
work into it (it's research after all), but they might be able to prevent it
by making a simple change, like disabling hyperthreading.

OpenBSD cares more about security than basically any other trade-off in OS
design (performance, usability, ...), so it makes sense to me that they went
this way. If you want a balance of security and performance, OpenBSD is not
for you any ways.

------
gerdesj
FFS: so far I've seen shit loads of "oooo - stuff <wave hands>" here from
people who are clearly not experts or even understand the issues properly in
this. Neither am I.

OP (and environs) has names on it that I have seen before and respect as
knowing what the hell they are on about.

