You need to reduce the parallelism in your build. If it is possible in your build setup to be runing 144 concurrent builds you are trashing your caches. This isn't an optimization. You need to reduce the number of projects being built concurrently (or even the number of total projects.) While I advocate a bit of over allocation, you have it in the extreme and this is really not optimal at all.
Also you need to get using the precompiled headers if you can -- sometimes this requires structuring your projects right to avoid lots of little projects.
Even with large projects I usually have really low compilation times for C++.
Here is a blog post I wrote about the right way to do this:
A standard test for scheduler responsiveness in Linux is to fire up a kernel compile with unlimited jobs[0]. If responsiveness is lost, the scheduler isn't ready for prime time. You think Microsoft wants the answer to this issue to be "we can't even match Linux"?
[0] Well, it used to be. I assume they've moved on to more demanding tests now, since hardware has far outstripped kernel build complexity.
The issue isn't responsiveness during build for me (that isn't usually an issue, my build machine is generally responsive), it is compile performance from start to finish. If you over-allocate to the extreme you will trash your CPU caches and this will slow everything down. You want to minimize CPU context switches (thread switches) while also ensuring that your CPUs are running at nearly 100% -- it is a difficult balance, but unlimited jobs and focusing on the responsiveness of another task is not the solution to maximizing start-to-finish compile time for a large project.
I think the article author is quite aware that too much parallelism will slow down the build, but still wanted too investigate why it makes his computer so unresponsive.
Speed of build isn't the issue, responsiveness is. Throughput is commonly sacrificed in favor of interactive performance, otherwise we'd all be running some modern equivalent of punchcard batch jobs.
Sounds a bit like apples and oranges. If in your test GCC happened to stall because of a bug in some shared library in user mode, would you blame the scheduler?
Maybe this isn't exactly a great argument since I know Linus has been known to take the side of not ever breaking user mode wrt kernel changes (probably rightfully so) but it is possible to write buggy code, including hangs, that runs on Linux. That would be the analogous scenario.
They're not testing for GCC stalling, they're testing for user interaction stalling -- the same thing the OP is complaining about.
Strictly speaking, the direct cause of OP's problem appears to be outside the kernel, but it is system-level. Microsoft has created a stack in which low-priority tasks can trivially destroy the responsiveness of interactive tasks unrelated to the build process.
Is this the same thing as with Visual Studio compiling 12 independent projects using 12 threads each? Does the kernel compile with unlimited jobs keep the number of concurrently running jobs below the processor limit, or are additional jobs in any way blocked on cheap primitives, such as waiting for previous jobs with a mutex?
The issue here alone is that he was trying to compile 12 projects simultaneously each with every logical processor he has. There might be some slack in the system, but is each project 92% wait time?
> Does the kernel compile with unlimited jobs keep the number of concurrently running jobs below the processor limit, or are additional jobs in any way blocked on cheap primitives, such as waiting for previous jobs with a mutex?
How do you think threads were implemented by the OS back when we all had one processor with one core?
I don't understand your question. I'm asking if the Linux kernel compile with unlimited jobs oversubscribes compute resources in the same way that compiling 12 independent projects does.
He's asking whether the number of jobs is artificially limited to the number of processors, not what happens when the number of jobs is greater than the number of processors.
I think you're missing the point. The problem described in the article is the expected behavior of the indirectly loaded component (mshtml.dll) which creates message loop (COM STA requirement), and because compiler runs as a low priority process, this message loop causes priority inversion which affects entire system. There are many wrong things here:
1. COM uses message loop and hidden windows to serialize calls. This is pretty stupid IMO, but I'm pretty sure Raymond Chandler would find millions of good reasons why it's done so.
2. Layering problem (XML parser uses mshtml.dll etc etc).
3. Why use config in XML file? Isn't XML so 1990s?
I think Bruce already knows plenty about making builds fast ;-)
The linked article demonstrates some ignorance as well by noting that command-line MSBuild builds are slower than from Visual Studio. I'd hazard a guess and say that they didn't find the /m option.
I wrote the linked article, so I take responsibility for any ignorance. It was some time ago (three years) so it maybe that we were not using the /m option at the time with MSBuild.
To my knowledge VS caches the MSBuild projects (.targets etc) while it's running. So it's possible that, even if VS and MSBuild have the same degree of parallelism, not having to parse the MSBuild XML gives VS an edge.
> You need to reduce the parallelism in your build.
I looked at your blog post and it recommends using /MP and setting num. parallel project builds to num. cores.
In other words, you recommend exactly what I was doing!
We do use precompiled header files. And we get great build times. It's only with /analyze builds that the system locks up.
Yes, 144 parallel compiles is excessive, but as your article says, 12-way parallel compile and 12-way parallel project builds are both necessary to expose all possible parallelism. Otherwise there will be times when many CPUs are idle. Until VS gets a global scheduler to avoid over subscribing the CPUs there is no way to get full parallelism without sometimes over subscribing.
I'm surprised that you find SSDs that helpful. With enough RAM they should offer only modest improvements to compilation, because source files and header files get cached.
Also you need to get using the precompiled headers if you can -- sometimes this requires structuring your projects right to avoid lots of little projects.
Even with large projects I usually have really low compilation times for C++.
Here is a blog post I wrote about the right way to do this:
http://exocortex.com/blog/speeding_up_cpp_builds_on_windows