
NUMA Deep Dive Part 1: From UMA to NUMA - wila
http://frankdenneman.nl/2016/07/07/numa-deep-dive-part-1-uma-numa/
======
frankdenneman
Thanks all for visiting the articles. Long time HN reader, never expected to
appear on HN! WOW!

~~~
angry_octet
Great article. Are those diagrams from Omnigraffle?

Obviously you are focusing on x86, but it might be worth mentioning where many
of these ideas developed. For example, SGI Origin hardware
([http://www.sgidepot.co.uk/origin/isca.pdf](http://www.sgidepot.co.uk/origin/isca.pdf)).

------
av500
I was expecting more Dirk Pitt in this post

~~~
icc97
I was going to ask that - but assumed I'd get down voted into the err..
Pacific Vortex. I'll down vote myself for that one.

------
praveenster
Nicely written article with good detail about the problems in the older
architectures. One thing that I'm still a little fuzzy about is on the Linux
kernel side of things in the context of multi core multi socket. Which CPU
does the kernel actually run on and how does it dispatch the process/threads
on the different cores/sockets?

~~~
wila
Yes, Frank has a very nice series going there. More parts are expected to
follow.

As to your question on what CPU the kernel normally runs on? AFAIK, it
normally starts at core 0, processor 0. You can see for yourself where
processes run.

For example using the ps command.

It will print kernel threads in square brackets and it can also list cpu and
core number behind a /.

Eg, example output of a linux VM with 2 processors.

    
    
      $ ps -ax
       PID TTY      STAT   TIME COMMAND
         1 ?        Ss     0:03 /sbin/init splash
         2 ?        S      0:00 [kthreadd]
         3 ?        S      0:00 [ksoftirqd/0]
         5 ?        S<     0:00 [kworker/0:0H]
         7 ?        S      0:15 [rcu_sched]
         8 ?        S      0:00 [rcu_bh]
         9 ?        S      0:00 [migration/0]
        10 ?        S      0:00 [watchdog/0]
        11 ?        S      0:00 [watchdog/1]
        etcetera.. removed the rest for readability
    

The CPU scheduler then handles the rest to see on which thread/core to run a
new process.

This post might also help:

[http://superuser.com/questions/389161/what-do-the-mean-in-
ps...](http://superuser.com/questions/389161/what-do-the-mean-in-ps-and-how-
do-you-create-a-process-like-it)

------
frankdenneman
Updated the series with ESX NUMA constructs:
[http://frankdenneman.nl/2016/08/22/numa-deep-dive-
part-5-esx...](http://frankdenneman.nl/2016/08/22/numa-deep-dive-part-5-esxi-
vmkernel-numa-constructs/)

------
rconti
I'll have to devote some time to reading these. I'm running some 8-socket
Oracle boxes and I've got NUMA disabled due to the sheer complexity of
maintaining/understanding the configuration. Processor pinning, and so on.

I need to educate myself to understand the consequences of this decision, what
the payoff would be to enabling NUMA, what my configs should look like, if
there's an "easy way out", etc.

~~~
angry_octet
When you say 'NUMA disabled' do you mean NUMA _aware_ scheduling and memory
allocation is disabled? Because it is going to be NUMA no matter what you do.

~~~
rconti
Yes. Supposedly Oracle recommends such a configuration, but I haven't dug into
it much beyond that.

~~~
angry_octet
[https://kevinclosson.net/category/oracle-
numa/](https://kevinclosson.net/category/oracle-numa/)

------
bogomipz
What a fantastic series. The author really put a lot of work into these. Thank
you Frank.

~~~
frankdenneman
My pleasure! Im aiming to release some of the articles focussing on ESXi level
scheduling this week. Stay tuned!

------
ythl
I wonder how NUMA compares to MAYA HEAP as far as memory performance goes?

~~~
detaro
What is MAYA HEAP?

