
CPU utilization is wrong [video] - startupflix
https://opensource.com/article/18/4/cpu-utilization-wrong
======
DonbunEf7
This nigh-useless blurb links to a Youtube video:
[https://youtu.be/QkcBASKLyeU](https://youtu.be/QkcBASKLyeU)

~~~
ModernMech
Yeah, can someone quick summarize this? Can't watch where I am.

edit: I found his blog post on this very topic. This should be the source:
[http://www.brendangregg.com/blog/2017-05-09/cpu-
utilization-...](http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-
is-wrong.html)

Also, now that we have the blog link, we have some previous HN threads:
[https://news.ycombinator.com/item?id=14301739](https://news.ycombinator.com/item?id=14301739)

~~~
decebalus1
[http://www.brendangregg.com/blog/2017-05-09/cpu-
utilization-...](http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-
is-wrong.html)

~~~
ModernMech
Beat you to it but thanks :P

------
gnode
To save you the time of watching the video: Spectre / Meltdown mitigations
(page table isolation) cause execution inefficiency because of TLB cache
misses.

------
dragontamer
So first of all: the lightning talk is certainly "worth it". Brendan Gregg is
very knowledgeable and I can certainly recommend watching the talk. Its a good
overview into the mindset of low-level optimization.

However, the core premise, the "headline" if you will of this talk, is utterly
wrong. I get it: the headline needs to be a bit provocative to draw eyes, but
its closer to "clickbait" than a good title of the talk.

The TL;DR of the talk is: Spectre/Meltdown caused a slowdown. But said
slowdown is coming up as a ~+20% "CPU Utilization" under Linux performance
measurement (ie: top or ps). This slowdown is because of a severe increase in
TLB-misses.

Which... doesn't seem "wrong" to me. If the CPU is stalling because of TLB-
buffer misses (kinda similar to an L2 cache miss or whatnot), then that's
certainly CPU-time that is being wasted on something. Perhaps it isn't using
power since the CPU is fully stalled, but the CPU is certainly "utilized"
during that timeframe. Aside from hyperthreading: you can't really do
something during the stall.

\----------

Anyway, my criticism aside, the talk is certainly worthwhile to watch. And
that's the important bit. So go ahead and watch the talk, even if I have a few
pedantic qualms about the phrasing of the headline.

The real benefit of the talk is how Brendan Gregg diagnosed this performance
problem, which eventually led to his conclusion about TLB-misses + Spectre.
Demonstrating this skill and walking through a system to be able to draw this
kind of conclusion is incredibly important.

I just don't... really like the title of the talk I guess. A rather minor
concern all else considered.

~~~
ModernMech
I think the problem Brendan is pointing out is that if we look at a task that
appears CPU-bound, we might think that one solution to make things go faster
is just make the CPU go faster. But beefing up the CPU won't help, because
it'll just spend the same amount of time waiting for e.g. the TLB to make the
round trip lookup.

I think this is really a discussion about the Von Neumann Bottleneck (VNBN).
The slowdown here is caused by the VNBN, but %CPU utilization is "disguising"
slowdowns caused by this bottleneck as a CPU bottleneck.

The VNBN is the primary reason Intel spends so much of their die space on
caches -- it's really holding us back. Intel is basically a glorified SRAM
vendor, selling vast blocks of memory attached as closely to a processor as
possible, because it's just so damn expensive to fetch it from anywhere else,
thanks to the VNBN.

> Aside from hyperthreading: you can't really do something during the stall.

This is true, but hyperthreading doesn't truly help us because the unit of
concurrency -- the thread here -- is too coarse, and everyone is competing
over the same resource: memory bandwidth (again, thanks VNBN), so any time you
spend on another task is going to increase return time of the original task
you were waiting for.

But we can do something here --- use a different architecture. Imagine a
dataflow machine, which sidesteps the VNBN entirely. Now if your processor is
stalled waiting for a TLB return, there’s a vast pile of work that can be
freely executed. Sure we have other resources at play now (like whatever
message passing bandwidth we have), but we no longer have to worry about the
VNBN.

~~~
dragontamer
Well, this particular slowdown was caused by the Spectre / Meltdown patch.

In particular, the Linux KPTI ([https://en.wikipedia.org/wiki/Kernel_page-
table_isolation](https://en.wikipedia.org/wiki/Kernel_page-table_isolation))
which prevents Meltdown from affecting the Kernel. The hardware fix is simple:
buy AMD Epyc instead. At least, in the nearterm. Intel's "Icelake" system will
have the hardware fix so that KPTI can die sometime in 2019. Its not normal to
flush the TLB on every systemcall. But its a necessary precaution as long as
the CPU is vulnerable to Meltdown.

~~~
paulmd
There's also the option of not running KPTI at all... database servers are the
most heavily impacted, and since they only run trusted code there's not much
point in patching them.

~~~
dragontamer
Hmmm... I don't like the idea of disabling security for performance.

I'm not an expert on this, but I'd turn on Huge Tables (1GB) first. Big
databases / datasets across RAM can be manipulated in 1GB chunks by the OS
instead of 4kB (!!!) chunks. The Skylake CPU can only cache ~4 Huge Tables,
but an additional set of Large Tables (2MB) should also help performance. If
properly configured of course.

I would hope that those changes would mitigate most of the KPTI issues with
regards to databases. But I really don't know.

~~~
paulmd
Do you mean Huge Pages?

------
billysielu
So is the CPU available to do something else or not?

~~~
Tuna-Fish
The particular thread that is waiting for data is not, but in a SMT system the
other thread has access to the full execution resources of the CPU.

------
gfnord
That's great and I agree with him. We have to pay attention to the stale.

