
Warp, a fast preprocessor for C and C++ - jamesgpearce
https://code.facebook.com/posts/476987592402291/under-the-hood-warp-a-fast-c-and-c-preprocessor/
======
WalterBright
Walter here (author of Warp). AMA about Warp.

~~~
jedbrown
> warp is currently able to preprocess many source files, one after the other,
> in a single command. [...] warp is set up/torn down only once for the entire
> set of files, rather than once for each source file.

I'd like to learn more about this. I spend a fair amount of time building on
HPC systems. Frustratingly, compiling on a $100M computer is typically 50x
slower than compiling on a laptop due to the atrocious metadata performance of
the shared file system. Configuration is even worse because there is typically
little or no parallelism. Moving my source tree to a fast local disk barely
helps so long as system and library headers continue to reside on the slow
filesystem. A compiler system that transparently caches file accesses across
an entire project build would save computational scientists an enormous amount
of time building on these systems.

~~~
mrich
So your build speed is limited by metadata file accesses to network storage?
Your network/storage must be really bad then. How about mirroring all the
needed headers locally before building? You could establish that as a makefile
rule.

~~~
_delirium
On large HPC clusters a common setup is to have a distributed cluster file
system, rather than disk set up as "local" and "network" volumes. All the
cluster machines' disks are integrated into a distributed volume running a
filesystem like
[https://en.wikipedia.org/wiki/IBM_General_Parallel_File_Syst...](https://en.wikipedia.org/wiki/IBM_General_Parallel_File_System)
or [https://en.wikipedia.org/wiki/GFS2](https://en.wikipedia.org/wiki/GFS2)

------
zenbowman
Must say facebook has quite the lineup of all-star programmers:

\- Carmack

\- Abrash

\- Alexandrescu

\- Kent Beck

Cannot say I'm not a little jelly of those who get to spend time with these
fine gents.

[http://www.southparkstudios.com/clips/babjj8/jelly-
school](http://www.southparkstudios.com/clips/babjj8/jelly-school)

~~~
aristidb
I find it a bit weird that the top comment in this story fawns over several
star programmers, but ignores the guy who actually wrote the tool that the
story is about.

~~~
astrodust
Those are the greats today, and with a talent pool like that it's likely that
many _future_ greats will be created there. You have to do more than write
good code to be legendary.

Maybe Facebook might end up being the Xerox PARC of our modern times.

~~~
eropple
I'm not sure you know who Walter Bright is. He's done a...few...things.

~~~
astrodust
There was a time at Xerox PARC when you couldn't throw a book without hitting
someone who was or would become an important pioneer.

I'm not trying to discredit what Facebook is doing, or the people there, but
it'll take time to build up to that level of talent. Having a few remarkable
individuals is a great start. Having an entire department filled with them is
going to be hard work.

------
mfonda
Great line from the interview:

> WB: I can guarantee that you are wrong about where your code is spending
> most of its time if you haven't run a profiler on it.

Definitely a good thing for all programmers to think about!

------
dllthomas
So, to be clear, this is a reimplementation of cpp?

~~~
WalterBright
I have never looked at the source code to cpp.

But Warp is designed to be a drop-in replacement. It doesn't produce char-by-
char exactly the same output as cpp, as the whitespace differs, and the
decisions about when/where to produce linemarker records are different.

But the output is functionally identical (any differences are bugs in either
Warp or cpp).

~~~
haberman
Interesting, I did not realize the standard left room for preprocessors to
make implementation-defined decisions!

------
jevinskie
Are precompiled headers and the performance increase from switching to
clang[0] not enough?

[0]:
[http://clang.llvm.org/features.html#performance](http://clang.llvm.org/features.html#performance)

~~~
pjmlp
Surely not.

I know of a project that takes around 3 hours to build with Clang when you do
a "make all".

~~~
deadfa11
Sure, but is that using PCHs? And how much of that time is preprocessing? I
don't think the parent question is whether clang can compile any project in
less than 3 hours. I think the interesting question is if warp is more
compelling than clang from a preprocessing perspective.

~~~
pjmlp
Yes with PCH and full build not just preprocessing.

------
wehadfun
Could any of this be used to improve comiling speed in other languages?

~~~
pmr_
Which other languages use the C/C++ preprocessor? I've seen it used for
generating data or other source code when a preprocessor comes in handy, but
never as a full-fledged component of another language.

I also think that a modern language that uses includes instead of modules is
just outright insane.

~~~
plorkyeran
Objective-C(++). I don't think obj-c changes anything about how the
preprocessor works, so presumably it'll work fine for that.

Using the C preprocessor in a non-superset-of-C language would be a pretty odd
choice.

~~~
duaneb
Objective-c introduced `#import`, which guards from double including by
default. I would suspect and performance gains are tempered with a "pure" objc
project.

------
yetanotherphd
One thing that really sped up my compiler was eating a ketogenic diet, and
eliminating preprocessed foods.

------
aDevilInMe
Does Andrei have a sticky D button on his keyboard? He seems to mention it in
every other sentence, where as I thought this post was about C and C++.

I would ask if there are any improvements between using wrap or just using
clang.

~~~
DannyBee
I would be shocked if there are improvements over clang (and if there were,
clang would fix them) in the general case.

clang has a pretty optimized preprocessor (It even has things like using SSE2
to do some neat things with character processing).

GCC has a fairly modern one, but it's still beatable (and clang beats it
handily).

~~~
pjmlp
Well, as I mentioned on a parallel thread, I know of a project taking around 3
hours to compile, that doesn't look very optimized to me.

~~~
DannyBee
You know of a project that takes 3 hours of time to _preprocess_ with clang?

I have serious doubts. Overall compilation time is kind of irrelevant to this
discussion, because Warp is just a preprocessor.

While you can get some X speedup to gcc by replacing the preprocessor, X, as a
factor of overall compilation time, is usually 0.2-0.5 in most cases,
depending on size of file.

I expect the gains warp gets over gcc overall from preprocessing to be similar
to those clang gets over gcc overall from preprocessing.

(Though it depends on the size of files being compiled, etc).

Most companies that want actual fast overall compilation and have the
resources, build caching distributed compilation infrastructure (Google,
Facebook).

As mentioned, if warp is really that much faster than _clang 's_ preprocessor
that it mattered, clang would be fixed :)

~~~
pjmlp
Faire enough, I meant full build times.

