
Testing GCC - dmalcolm
https://developers.redhat.com/blog/2017/02/13/testing-testing-gcc/
======
boris
Interesting to see complex pieces of software including features that are
primarily used for testing. I guess this is a natural next step after having
testing micro-formats[1].

One thing I don't like about GCC's testing framework is the number of files.
If you need to test 10 related things (say, diagnostics for a particular
language construct), then you will have 10 different test files. I would much
prefer for them to be in a single file. This is one of the goals of
Testscript[2].

[1] [https://blog.nelhage.com/2016/12/how-i-
test/](https://blog.nelhage.com/2016/12/how-i-test/)

[2] [https://build2.org/build2/doc/build2-testscript-
manual.xhtml](https://build2.org/build2/doc/build2-testscript-manual.xhtml)

~~~
dmalcolm
[author of the post]

Thanks for the links; a lot of interesting material in there.

FWIW, DejaGnu gives some flexibility over how granular the test files can be.
For example, here's part of the test case for the -Wmisleading-indentation
warning I added in gcc 6: [https://github.com/gcc-
mirror/gcc/blob/master/gcc/testsuite/...](https://github.com/gcc-
mirror/gcc/blob/master/gcc/testsuite/c-c%2B%2B-common/Wmisleading-
indentation.c)

i.e. that's one 700 line test source file, expressing most of the test
coverage for one feature.

I realize now that the post should have had an example of what such a test
case has; see the "dg-warning" directives in that file.

So that's one "single file" test case. That said, I had to put some of the
other test cases for that feature into different test files: some of them
interact with the preprocessor in "interesting" ways, so it was simplest to
split them out.

However, when a test starts failing, it's handy if the failing test is small
(especially when stepping through the latter in the debugger...).

So in our testing we have a mix of both big test files covering a lot of
material, and small test files, covering very specific corner cases.

Another aspect of granularity is how long the testsuite takes to run: each
test source file in the DejaGnu suite involves exec-ing the compiler, which
adds a fixed amount of overhead. It parallelizes well, but it's still
advantageous to stuff more testing into fewer files (with the obvious tradeoff
w.r.t. debugability if something starts failing). This was another reason for
adding the unit-testing suite: this part is all done in-process, and so these
~30,000 tests run in less than a second.

------
saurik
> So, in GCC 7, we’ve extended the C frontend so that we can embed fragments
> of GIMPLE and RTL dumps as the bodies of functions.

This additionally sounds extremely interesting for reasons that have nothing
to do with testing (though, I bet most of the things I'd want to do this with
this will continue to require the expanded platform support for
__attribute__((__naked__)) that has blocked a lot of my use cases for inline
assembly and which does not seem to be something that is "wanted" by GCC,
though I should verify it isn't just due to no one providing a patch... it
isn't as if there aren't other people asking for it).

------
namelezz
> As a relative newcomer to the project, one of my “pain points” learning
> GCC’s internals was the custom garbage collector it uses to manage memory

It's interesting that GCC has a custom GC.

~~~
greglindahl
I'm be surprised if it _didn 't_ have something related to memory allocation
and free. Open64 (previously the SGI compiler) has an "arena" memory allocator
to improve memory locality and make freeing scratch memory easier. Looks like
LLVM uses arena allocation, too.

~~~
pizlonator
Some people swear by arenas, some people swear by GCs, some swear by stacks,
and others swear by malloc. There are probably people who swear by things I
didn't list. Compiler writers are people, too, and so they swear by many
memory allocation styles.

When I first started writing compilers, I learned the craft from people who
happened to also be GC zealots, and they insisted that it was not possible to
write a compiler any other way.

I remember once talking to a VM hacker who wrote compilers in C++ and was
shocked that it was even possible to write a compiler in Java.

Then I saw LLVM, which mostly just relies on new/delete and an intuitive
ownership model. I've done that ever since.

But as a memory management nerd I have to say that there are some advantages
to every approach. Arenas are cheap and intuitive. GCs mean you don't have to
care about lifetimes. Malloc/free minimizes object drag. Pick what you like
and don't let anyone tell you that any one of these approaches is inherently
better than the others.

------
raverbashing
Well, besides the big testing of compiling an entire linux distribution, yes,
it is good to have specific tests for some particular issues in the compiler
(manageable pass/fail tests that can identify a problem quickly)

~~~
greglindahl
gcc has always had a huge unit test suite. Not a new thing.

~~~
sanxiyn
gcc had a huge _integration_ test suite, which is different from unit test
suite.

------
wolf550e
So is GCC adding a toolchain similar to the one LLVM has around its bitcode?

In this instance, a way to write a text representation of an internal data
structure that represents code, run a single optimization pass over that data
structure, dump the result, compare the result with serialized text
representation of "expected value"?

Will they add the other parts, to extract an llvm-like core out of gcc and
implement the C and C++ compilers and link time optimizer as users of that
core?

~~~
ori_b
> So is GCC adding a toolchain similar to the one LLVM has around its bitcode?

Not as far as I can tell.

> In this instance, a way to write a text representation of an internal data
> structure that represents code, run a single optimization pass over that
> data structure, dump the result, compare the result with serialized text
> representation of "expected value"?

That's already existed for ages. See the -fdump-tree-* family of functions. It
also doesn't do anything llvm-like.

~~~
wolf550e
So all that hasn't existed for ages. Only dumping the intermediate
representation has existed for ages, but not loading it and not running the
middle-end of the compiler on the loaded IR.

~~~
DannyBee
Right, the historical problem with loading GIMPLE was that it still required
all sorts of frontend specific bits.

Also, tons of things reused tons of bits for different purposes :)

Way the heck back when (pre-c++), i started splitting up just the decl
hierarchy in trees to try to get things to stop reusing bits, and to expose
what would be required to make it independent of the frontendy bits.

------
drfuchs
Great, but now how will you find out that a particular optimization pass
hasn't been completely obviated by earlier passes, since you won't be looking
for C code that actually triggers it?

~~~
greglindahl
I never got around to actually implementing the following in the PathScale
compiler, but my idea was to add logging in all the places where allocations
do something, and then create lists of optimizations that I expect to have
successfully fired in various functions or programs. This would be especially
handy for checking that new optimizations didn't disable important things for
various SPECcpu benchmarks.

~~~
greglindahl
Sorry, "the places where _optimizations_ do something". Didn't notice until
after the edit window was long passed. I blame autocorrect :-)

------
pizlonator
I'm super impressed by the size of their test suite. I wish I had a test suite
like that!

------
monochromatic
Took forever to load, then gave me "Page Does Not Exist." Anybody else having
issues?

~~~
ParadoxOryx
Here's the Google cached version -
[http://webcache.googleusercontent.com/search?q=cache:https:/...](http://webcache.googleusercontent.com/search?q=cache:https://developers.redhat.com/blog/2017/02/13/testing-
testing-gcc/)

~~~
egwynn
For me, that version refreshes indefinitely. I recommend reading the plain-
text version:
[http://webcache.googleusercontent.com/search?q=cache:https:/...](http://webcache.googleusercontent.com/search?q=cache:https://developers.redhat.com/blog/2017/02/13/testing-
testing-gcc/&num=1&prmd=ivns&strip=1&vwsrc=0)

