
Understanding Linux CPU Load - when should you be worried? - colinprince
http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages
======
molo_
There are two contributions to the load factor: number of processes/threads on
the ready-to-run queue and the number blocked on I/O. The processes blocked on
I/O show up in the "D" state in ps and top and also contribute to this number.

This article entirely ignores the number of processes blocked on I/O. A load
average exceeding the number of CPUs (cores, whatever) does not automatically
mean the CPUs are overloaded.

~~~
caf
Not all processes blocked on I/O are in D state - for a common example,
processes blocked on I/O to a network socket or terminal will simply be in the
S state, and not count towards load.

~~~
duskwuff
In practice, processes typically go into D state ("uninterruptible sleep")
when they're blocked on access to a local disk, whether that's explicit I/O
(read/write) or implicit (paging). Not coincidentally, this is also the one
type of blocking I/O that you can't get knocked out of by a signal.

~~~
freiheit
Actually, they'll get blocked on access to NFS or other network-based disk,
too. (but you can configure the NFS mount to allow signals to interrupt) I've
seen NFS problems lead to loads over 100.

------
kbob
Three comments.

1\. I can't comment on the original article. Are comments closed, or am I
dumb?

2\. The author seems to have assumed a web server responding to bursty
traffic. Several people have pointed out workloads to which the 0.7 heuristic
doesn't apply - compute servers, I/O bound servers, compile jobs, desktops. He
should have stated that assumption up front.

3\. Hyperthreads. For purposes of load monitoring, you should be counting the
number of threads, not the number of cores. Yes, hyperthreads are slower than
cores, but that doesn't matter. The load average is the ratio of work
available to work being done (oversimplified, I know), and, as such, it's
scaled to the actual throughput of the threads available.

Fortunately, the author suggested counting CPUs by reading /proc/cpuinfo, and
/proc/cpuinfo lists threads, not cores. So those two errors cancel out. (-:

~~~
scott_s
Point 3 depends on the workloads. Most SMT [1] implementations replicate
integer functional units - otherwise the threads would stall on basic things
like computing addresses - but they don't replicate floating point units. So
if you have lots of floating point heavy work, then you're limited by the
number of cores, not the total number of SMT contexts provided by all of those
cores.

So it's not that SMT pipelines are _slower_ , it's just that they share
resources with the other SMT pipelines.

[1] Simultaneous multithreading (SMT),
<http://en.wikipedia.org/wiki/Simultaneous_multithreading>, is the generic
name for what Intel calls hyperthreading.

~~~
kbob
You are correct. I should have had two oversimplification disclaimers in that
sentence. (-:

------
acabal
I love reading stuff like this. As a kind-of sysadmin by need rather than by
choice, I'm often confused and intimidated by systems that other sysadmins
seem to be born knowing about. It's always refreshing to read a
straightforward explanation for one of those important concepts that seems to
be common knowledge for everyone but me, and never seems to be explained
anywhere.

------
sciurus
Load average is an easy number to monitor, so lots of people focus on it.
However, it doesn't provide you with much information. When your load is high,
you have to examine other values (e.g. CPU time spent in user mode, system
mode, and iowait) to determine why the load is high before you can start to
resolve the problem. If you monitor and alert directly on those other values,
you'll save time.

~~~
keithnoizu
Exactly, looking at the load curves over time is useful to gauging how well
you are doing overall and for spotting potential trouble issues but when it
comes to actually dealing with these issues or predicting an eminent collapse
i tend to look at mysql threads and performance, the slow query log,
concurrent users, etc. for determining what needs to be dealt with and what
will hit us on the head in the near future.

------
DrJ
(if you don't know this you should read the link):

If (# CPU Cores / Load) > 1, shit has hit the fan

I disagree with 0.7 being the starting point for investigation on extraneous
load, but you should be more worried about changes in 1st or 2nd moments in
the load (velocity and acceleration), which as analogy on the link, you don't
care too much about steady traffic, it's when traffic starts bursting at the
seams.

Having a machine running at 0.75 load for a shared machined (say a development
database) might actually mean your resources are actually being consumed
regularly. Albeit seeing that average load climb slowly towards ~1.0 means you
need to fix it before the pipes clog shut.

~~~
syedkarim
Would there be any reason that perceived performance would decrease when the
cpu load is 50% of the total number of cores? We have an X5660 with 24-cores
and once the one-minute average gets over 12, pageload times increase
dramatically.

~~~
mrich
make sure you are looking at physical core count, not hyperthreaded cores
(which should be a 2x difference for your CPU).

~~~
syedkarim
Am I using the wrong command to count physical cores (I'm guessing so? grep
'model name' /proc/cpuinfo | wc -l What should I use to count physical cores?

~~~
mrich
/proc/cpuinfo are hyperthreaded cores, as exposed to the OS. For basically all
the modern multi-core Intel Xeon CPUs you can divide that by 2. It seems you
can also find out by looking at physical ID and "cpu cores". On a 64
(hyperthreaded) cores machine, I see physical ID 0..4 and cpu cores 8 in this
case, which would mean 8*4=32.

~~~
mrich
small correction: "physical ID 0..3"

------
aaronharnly
Incidentally, on a Mac, this will give you your number of cores, along with
other handy stuff:

system_profiler SPHardwareDataType

Hardware:

    
    
        Hardware Overview:
    
          Model Name: MacBook Pro
          Model Identifier: MacBookPro5,1
          Processor Name: Intel Core 2 Duo
          Processor Speed: 2.4 GHz
          Number of Processors: 1
          Total Number of Cores: 2
          L2 Cache: 3 MB
          Memory: 4 GB
          Bus Speed: 1.07 GHz
          Boot ROM Version: MBP51.007E.B05
          SMC Version (system): 1.41f2
          Serial Number (system): [snip]
          Hardware UUID: [snip]
          Sudden Motion Sensor:
              State: Enabled

------
jff
Nicely written... but looking at the last section "Bringing it Home" I'd like
to point out that if you were to, say, do a make -j<#cores> in the Linux
source tree (or any other bloated GNU monstrosity) you'll get a 15-min load of
well over the 70% desired :) But it's not a bad thing... it just means Firefox
will run like crap for a while. Also, don't do that on your web server, which
is probably what he was talking about anyway.

~~~
jerf
"it just means Firefox will run like crap for a while."

Put an "ionice -c 3" on that job and you probably won't notice the performance
effect on Firefox anymore. (You probably don't need conventional "nice"
because compile jobs tend to get their priorities dropped anyhow because they
are using a lot of CPU without yielding, but dropping the scheduler that hint
can still be helpful in some cases.)

(Annoyingly, unlike nice, ionice requires the specification of the class; I
wish it would just default to -c 3 like nice has a reasonable default.)

~~~
simcop2387
IO Nice will just affect the io scheduling (obviously), but you can also do
the same thing to the CPU schedulers.

    
    
        schedtool -B -e ionice -c3 make -j10
    

That's the idiom I commonly use on long large compile jobs. means that
anything else will always get the cpu or io time (maybe with some increased
latency, which usually isn't too bad) that it needs while all idle time is
taken up by the larger compile job. This makes for a very happy desktop system
when upgrading (gentoo user here).

~~~
sciurus
If you don't have schedtool available on your distro, you can use 'chrt
--batch 0' instead.

------
afhof
I had heard that the load averages were the size of the scheduler's ready
queue. If that is correct, wouldn't a load of more than 1.00 on a multi
processor machine still be bad, since processes are ready to fire but are
waiting for the next jiffy?

~~~
seiji
Here's a more thorough treatment (with maths and all):

<http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf>

<http://www.teamquest.com/pdfs/whitepaper/ldavg2.pdf>

------
Create
<http://video.google.com/videoplay?docid=-8002801113289007228>

ftp://crisp.dyndns-server.com/pub/release/website/dtrace/

------
quantumhobbit
So how does this change for logical/nonphysical cores. Should a hyperthreaded
dual core system be considered full at a load of 2.00, 4.00 or something in
between like 3.00?

