
GCC Automatic Parallel Compilation Viability Results Help Up to 3.3x - rbanffy
https://www.phoronix.com/scan.php?page=news_item&px=GCC-Auto-Parallel-Compilation
======
xxpor
I regularly work with multi-thousand line source files, but compilation speed
has never really been an issue there, at least for me. The issue has always
been the link time afterwards (especially if you turn LTO on). I guess its
probably more of an issue for C++.

Is anyone working on adding parallel linking to GCC like clang has?

~~~
rrss
I'm not familiar with clang's support for parallel linking, so maybe this
isn't what you mean, but GCC supports parallel ltrans for LTO, and IIRC GCC 10
made multithreaded ltrans the default behavior.

~~~
xxpor
Yeah, I could just be behind. I try to at least know what GCC is up to, but
for one of the platforms I work with we're still stuck on GCC 4.9 :(

------
MaxBarraclough
Very impressive work for a student, the GCC codebase isn't known for being
approachable.

Can we expect Clang to implement the same trick in a few months time?

~~~
jcranmer
No.

LLVM is not amenable to using multiple threads to compile a single TU. The
use-lists of global values (such as functions, global variables, or constant
expressions) include all uses from all functions, so parallelizing on a per-
function basis requires acquiring locks (or some sort of lock-free data
structure) to add, remove, or iterate these lists, which would considerable
overhead on a relatively common operation.

If you want more details, you can read the recent thread on llvm-dev
discussing this: [http://lists.llvm.org/pipermail/llvm-
dev/2020-March/139606.h...](http://lists.llvm.org/pipermail/llvm-
dev/2020-March/139606.html)

------
matt42
Great that C++ compiler teams are working on reducing C++ compile time. The
next great feature I think would be a game changer is caching the result of
compilation (templates and binary code). A fine grained cache acting at the
level of a line (or a function if it's too hard) would save an incredible
amount of time to C++ devs, while also simplifying build systems. This would
also stop engineers from losing days trying to speedup their build.

~~~
jcelerier
Such a compiler exists, it's called zapcc (was a fork of clang 3.something).
Sadly it has been abandoned and never merged even though it was opensourced.

~~~
appleflaxen
The github page states:

> When was the source last merged with LLVM trunk?

> This open-source release was last merged with LLVM 325000 on 2018-02-13.

------
gnufx
This was being touted for the Fedora build system recently, and Jeff Law made
typically sensible remarks to temper the enthusiasm. I don't have a link to
the devel archive to hand.

------
rowanG077
What is automatic here? It's clear from the article what this means.

~~~
acqq
Imagine you have one source file with 100 of functions which you want to
compile. Traditionally it could be compiled only using one core, processing
function after function. Assuming we are processing already preprocessed
information, there's no theoretical reason why more cores couldn't be used to
compile the functions: e.g. with 4 cores, each could get 25 functions. The
practical reason why not is not only that it increases code complexity in the
trivial case (all the functions take the same time) but that in non-trivial
case you can end up trying to do a lot of what the OS and make system together
do with separate processes doing the separate compilation of many files,
dynamically deciding what to do at which point.

This experiment tries to make a case that something in that direction is
doable and worth doing. I missed the good proof for the later, based on the
analysis of state of the existing projects, however. I would personally rather
consider the completely opposite direction:

In the big C++ projects, most of the functions are already present in small
compilation units, and the total of the build process mostly spends time in
processing the same huge set of headers for every compilation unit. Often, the
whole process would be faster if more compilation units would be compiled as
one(!) when the headers aren't used in a way that the semantic of them changes
depending on the way in which they are used (I've heard that, of all, LLVM
actually does that unfortunate thing, but I haven't spend time analyzing that
myself).

~~~
thebiss
Compiling entire units as one was standard practice dealing with the (defunct)
IBM VisualAge C++ compiler. On one of our larger projects, updating the
makefile to concatenate all .cpp's in each component, then compiling those
.cpp's, reduced build times from 15-20 hours down to 1-2 hours.

~~~
jcelerier
With CMake it's just a matter of setting CMAKE_UNITY_BUILD=1

~~~
saagarjha
Well, I would assume you'd also have to make sure your code is safe to build
under that kind of inclusion.

------
lawwantsin17
Google: Let's start all of our projects on a foundation built by an intern.

~~~
saagarjha
Google Summer of Code participants are not interns, they are paid a stipend to
work on open source projects. And–isn't it good if an intern project ends up
being used?

