
“gcc will quietly break nearly half of all the packages that it compiles” - reader_1000
http://www.metzdowd.com/pipermail/cryptography/2015-October/026841.html
======
zzalpha
Headline is misleading.

What's going on here is that the GCC developers feel free to change the
generated code for behaviour undefined in the C spec. This means they might
alter the behaviour of code since 40% of all packages in Debian (the "half of
all packages" mentioned here) contain code whose behaviour is undefined.

But the poster seems to define any change in behaviour as "breaking" code,
which is ridiculous. How much of that 40% is rare edge cases or similar, such
that in reality the behaviour of the code doesn't change in practice?

The follow-up email raises this exact point, noting that in the specific
example of pointer overflow:

 _In executions where overflow does not happen, the gcc produced binary will
match the behavior of the abstract machine in the C spec._

Which means that the cited static analysis, while correctly identifying code
containing UB, may not actually have an issue at runtime as the condition may
never occur in practice.

The real problem is any software that truly relies on undefined behaviour for
correct operation (which, I'll bet, is far less than the 40% cited here). That
code is fundamentally non-portable to other compilers specifically because
each compiler may produce semantically different output.

~~~
barrkel
_The real problem is any software that truly relies on undefined behaviour for
correct operation (which, I 'll bet, is far less than the 40% cited here)._

I'd bet it's well into double digits relying on undefined behaviour, and over
50% relying on unspecified behaviour. It's hard to write interesting code in C
that doesn't, where interesting means code that couldn't just as profitably be
written in a different language with stronger memory safety.

Programmers have a mental model of many things - in particular, 2's complement
signed numbers, and the idea that pointers are just numbers indexing memory -
that are not guaranteed by the C spec. Much of the reason for programming in a
lower level language is to take advantage of lower level machine
characteristics, like tagged pointers, unsafe unions, structs with blobs of
data appended to avoid an indirection, custom memory allocators, etc., but the
C abstract machine doesn't necessarily give all the guarantees required,
without a lot of care and attention to details.

~~~
makomk
Even C compiler developers can't always get it right. There have been a few
bugs where, for example, one compiler pass optimised a struct initialisation
into a single write that wrote into the slack space after the end of the
struct, and a subsequent pass detected this undefined behaviour and removed
the initialisation altogether. (These optimisation tricks inherently make
compiler development more risky, because they mean that composing safe
compiler passes is likely to create something unsafe.)

------
deng
Now that's some flawed logic here:

\- Some paper says that 40% of Debian packages have undefined behavior in
them.

\- gcc's optimizer is sometimes unforgiving w.r.t. undefined behavior (see
also: strict aliasing), changing the intended meaning of the code.

\- Therefore, it breaks 40% of packages.

And boom, there's your clickbait headline...

------
copsarebastards
The only reasonable thing to say about this was already said upthread of the
page, and quoted here:

> _I have worked on many programs, and whenever I found such a problem
> (typically called a "portability problem"), where the code was assuming
> something that the language did not guarantee, I fixed the program rather
> than bitching about the compiler._

~~~
geofft
IMO, it's a little subtler than that. It's not the compiler's fault, it's the
_language 's_. Plenty of languages have no undefined behavior that can be
written by accident. Go and safe Rust, for instance, have just about no
undefined behavior at all, and are both performance-competitive with C. (Go
has UB if you cause race conditions, and Rust has UB within `unsafe` blocks
analogous to C's UB.)

A C compiler, meanwhile, has to aggressively take advantage of undefined
behavior to get good performance, and the C specification has been keeping
behavior undefined for the benefit of compilers.

You can hope that you find all such problems in C (which you might not) and
"fix the program", but you can also "fix the program" by switching to a better
language.

~~~
EdiX
Yes, the C language specification is a bit shit, it leaves too much leeway to
compilers so that C compiler for broken, niche architectures can be written.
However, I disagree with you. All this badness in the C specification did not
stop people from writing reasonable C compilers for reasonable architectures
for decades.

The real problem here is competition. Gcc is in a competition with clang to
produce fast code which makes the gcc developers feel justified when they
exploit undefined behaviours for marginal optimizations.

This is a case of following the letter of the law (in this case the C
standard) while disregarding its spirit: all the undefined behaviour was so
that C compilers could accomodate for odd architectures while remaining close
to the metal, not so that compiler programmers could go out of their way to
turn their compiler into a mine field.

~~~
copsarebastards
> This is a case of following the letter of the law (in this case the C
> standard) while disregarding its spirit: all the undefined behaviour was so
> that C compilers could accomodate for odd architectures while remaining
> close to the metal, not so that compiler programmers could go out of their
> way to turn their compiler into a mine field.

Computers don't have spirits; they work as you tell them to work, to the
letter, and if you're remaining close to the metal, your language will
indicate that fact. Optimizing undefined behaviors doesn't make C a minefield;
low-level programming for different architectures just is inherently a
minefield. C was a minefield before these optimizations were added.

Rust is extremely impressive because they've found so many ways to do high-
level programming while maintaining low-level performance. But they can only
do that because they have the benefit of the 4 decades of programming language
research that have occurred since the basics of C were designed.

~~~
EdiX
>Computers don't have spirits

But standards committee do.

>Rust is extremely impressive because they've found so many ways to do high-
level programming while maintaining low-level performance. But they can only
do that because they have the benefit of the 4 decades of programming language
research that have occurred since the basics of C were designed.

I doubt rust could be ported to a 8bit PIC microcontroller, or to a 6502
keeping reasonable performance characteristics or letting the programmer take
advantage of the platform quirks. It's not just "4 decades of programming
language research" it's also that it's intended to work only on "modern"
processors.

~~~
geofft
> I doubt rust could be ported to a 8bit PIC microcontroller, or to a 6502
> keeping reasonable performance characteristics

I don't believe this is correct. Most of why Rust avoids UB is that it uses
static types much more effectively than C does. Static types are an
abstraction between the programmer and the compiler for conveying intent, that
cease to exist at runtime. So the runtime processor architecture should be
irrelevant.

For instance, in C, dereferencing a null pointer is UB. This allows a compiler
to optimize out checks for null pointers if it "knows" that the pointer can't
be null, and it "knows" that a pointer can't be null if the programmer
previously dereferenced it. This is, itself, a form of communication between
the programmer and the compiler, but an imperfect one. In Rust, safe pointers
(references) cannot be null. A nullable pointer is represented with the
Option<T> type, which has two variants, Some(T) and None. In order to extract
an &Something from an Option<&Something>, a programmer has to explicitly check
for these two cases. Once you have a &Something, both you and the compiler
know it can't be null.

But at the output-code level, a documented compiler optimization allows
Option<&Something> to be stored as just a single pointer -- since &Something
cannot be null, a null-valued pointer must represent None, not Some(NULL). So
the resulting code from the Rust compiler looks exactly like the resulting
code from the C compiler, both in terms of memory usage and in terms of which
null checks are present and which can be skipped. But the communication is
much clearer, preventing miscommunications like the Linux kernel's

    
    
        int flags = parameter->flags;
        if (parameter == NULL)
            return -EINVAL;
    

Here the compiler thinks that the first line is the programmer saying "Hey,
parameter cannot be null". But the programmer did not actually intend that. In
Rust, the type system requires that the programmer write the null check before
using the value, so that miscommunication is not possible.

There are similar stories for bounds checks and for loops, branches and
indirect jumps and the match statement, etc. And none of this differs whether
you're writing for a Core i7 or for a VAX.

> or letting the programmer take advantage of the platform quirks.

I'm not deeply familiar with that level of embedded systems, but at least on
desktop-class processors, compilers are generally better than humans at
writing stupidly-optimized code that's aware of particular opcode sequences
that work better, etc.

(There are a few reasons why porting Rust to an older processor would be
somewhat less than trivial, but they mostly involve assumptions made in the
language definition about things like size_t and uintptr_t being the same,
etc. You could write a language with a strong type system but C's portability
assumptions, perhaps even a fork of Rust, if there were a use case / demand
for it.)

~~~
EdiX
> I don't believe this is correct.

Did rust find a way to defeat the halting problem and push all the array bound
checks to compile time? How well does rust deal with memory bank switching
where an instruction here makes that pointer there refer to a different area
of memory?

~~~
geofft
> Did rust find a way to defeat the halting problem

I don't understand why the halting problem is relevant to this conversation.
Just about all practical programs don't care about the halting problem in a
final sense, anyway; see e.g. the calculus of inductive constructions for a
Turing-incomplete language that lets you implement just about everything you
actually care about implementing.

The halting problem merely prevents a program from evaluating a nontrivial
property of another program _with perfect accuracy_. It does not prevent a
program from evaluating a nontrivial property (bounds checks, type safety,
whatever) with possible outputs "Yes" or "Either no, or you're trying to trick
me, so cut that out and express what you mean more straightforwardly kthx."

This realization is at the heart of all modern language design.

> How well does rust deal with memory bank switching where an instruction here
> makes that pointer there refer to a different area of memory?

This problem boils down to shared mutable state, so the conceptual model of
Rust deals with it _very_ well. The current Rust language spec does not
actually have a useful model of this sort of memory, but it would be a
straightforward fork. As I said in my comment above, if there was interest and
a use case for a safe language for these processors, it could be done easily.

------
joosters
The article is just complaining about undefined behavior, and what compilers
do when they encounter code not written to spec.

Rather than rehashing the arguments for and against it, I'd really recommend
anyone interested to read these articles:

[http://blog.llvm.org/2011/05/what-every-c-programmer-
should-...](http://blog.llvm.org/2011/05/what-every-c-programmer-should-
know.html)

[http://blog.regehr.org/archives/213](http://blog.regehr.org/archives/213)

------
the_mitsuhiko
Inknow people are quick to complain about programmers relying on UB here but
this really is a long standing disagreement with the gcc folk. They are
language lawyers of the worst sort and do not consider security implications
being a point of discussion :(

~~~
copsarebastards
> _On two occasions I have been asked, — "Pray, Mr. Babbage, if you put into
> the machine wrong figures, will the right answers come out?" In one case a
> member of the Upper, and in the other a member of the Lower, House put this
> question. I am not able rightly to apprehend the kind of confusion of ideas
> that could provoke such a question._ \--Charles Babbage

The modern version of this seems to be:

"Mr. Babbage, I put the wrong figures into the machine and the wrong answers
came out! Please fix it this, this has security implications!"

You can't reasonably expect the compiler to make your insecure code secure.

Calling them "language lawyers" is some entitled crap. GCC commits to
implement the specification of the language. Expecting them to maintain some
huge number of undefined behaviors is literally expecting them to do something
they never said they would do and couldn't do even if they said they would.

~~~
makomk
The problem is that it's often perfectly clear, reasonable code on all the
systems it was intended to run on. For example, on all Unix-like systems,
pointer arithmetic is simply arithmetic and behaves like it. (C's predecessor
didn't even have separate pointer and integer types.) So prior to compiler
optimisations, this series of operations is safe and well-behaved on all
architectures Linux supports even if a is NULL:

    
    
      int *b = &a->something; // pointer arithmetic, doesn't dereference a.
      if(a == NULL) return 0;
      else something_critical = a->somethingelse;
    

However, some non-Unix address models that Linux doesn't support don't permit
pointer arithmetic on NULL pointers. So the ANSI C standards committee
declared it undefined. Which means that gcc can - and eventually did -
eliminate the NULL pointer check. This has resulted in privilege escalation
vulnerabilities in Linux that didn't exist until gcc decided to optimise the
code, some of them quite well-hidden.

~~~
copsarebastards
I understand the problem, I'm saying that it's not GCC's problem. If you don't
want undefined behavior, don't put undefined behavior in your code. The code
you wrote _isn 't_ clear or reasonable, because it relies undefined behavior.
It's a valid criticism that this code does appear to be straightforward when
it isn't, but that's not a criticism of GCC, it's a criticism of ANSI C. If
you don't like it, use a better language. C was designed 4 decades ago; and
they can't possibly have forseen every problem that we've discovered in that
time.

~~~
MaulingMonkey
> If you don't want undefined behavior, don't put undefined behavior in your
> code.

I'd quip that this is statistically impossible for a sufficiently large
codebase.

> it's a criticism of ANSI C. If you don't like it, use a better language.

This is my basic stance. However, if I'm e.g. in a situation where I have a C
or C++ codebase I can't afford to rewrite from scratch, I'd like to use a
"Better C" compiler, where "Better C" is a slightly less bad version of "ANSI
C" \- some undefined behavior removed, for example.

As shorthand, I'll generally refer to compilers for "Better C" as "Good C
Compilers".

GCC is not trying to be a Good C Compiler. They've decided these things aren't
their problem. Which is... fair. That's their choice. I do not for one minute
pretend to understand that choice however - and it gives me yet one more
reason to switch to a Good C Compiler.

~~~
copsarebastards
Good luck with that. I suspect the only good C is not C.

~~~
MaulingMonkey
I don't disagree - but there's value in harm reduction, no?

------
DanWaterworth
_Since compilers (and by "compilers" I mean gcc mostly) quietly break your
code behind your back, you have no way of telling whether you really fixed
things or not._

Compile your test suite with -fsanitize=undefined.

~~~
deng
That will catch a _tiny_ part of undefined behavior.

~~~
MichaelGG
Why doesn't the compiler emit a warning for all UB it finds while compiling?
Or do regular programs rely on this too much to be feasible? It must be
something like that, or there'd not be much performance gains to be had by
exploiting UB right?

~~~
lmm
Any time you add two (signed) integers, that's potentially undefined
behaviour.

------
jacquesm
Bad craftsmen blame their tools.

~~~
Dylan16807
[https://c4.staticflickr.com/8/7226/7095238893_5000f6e57d_b.j...](https://c4.staticflickr.com/8/7226/7095238893_5000f6e57d_b.jpg)

------
babuskov
Since when is 40% "nearly half". Can we change the title to something like:

"40% of Debian packages might break if GCC changes the way it handles
undefined behavior"

~~~
tedunangst
When has 40% not been nearly half? What is the minimum threshold for a
quantity to quality as nearly half?

~~~
GTP
"Nearly half" is a subjective concept so different people can give different
answers if you ask them wich value "nearly half" has. I also think that it
strongly depends on the absolute number we are talking about, not only
percentage.

