
Three proposals for the C standard WG14 - mannykannot
http://www.yodaiken.com/2018/06/17/three-modest-proposals-for-the-c-standard-wg14/
======
anderskaseorg
Your proposed statement “conforming implementations cannot transform code on
the basis that some behavior left undefined by the standard is impossible”
would have no meaning in the standard, even if it were written somewhere other
than a non-normative note.

The standard doesn’t describe transformations; it describes behaviors. You
can’t just say that undefined behaviors can’t be “transformed”, because there
was nothing to transform them _from_. If you want some behaviors to be
possible and some behaviors to be impossible, then you need to define those
behaviors.

For C, that’s not something we can realistically do in every case. There’s no
way to reasonably constrain the set of allowed behaviors after a program does
some crazy pointer operation like * (int *) rand() = rand(), because any
implementation of those constraints would more or less require an expensive
run-time check at every pointer access.

What you could meaningfully ask is for particular situations (like negative
shifts) to be considered unspecified or implementation-defined rather than
undefined. That way you don’t give up the ability to constrain the behavior of
programs after those situations arise. There are already many unspecified
behaviors in the standard, such as reading the value of padding bytes in
structs.

Such proposals could be considered on their merits, but they’d have to be
discussed individually. There’s no way to remove all undefined behavior from C
and still have something that would be recognizable as C.

~~~
vyodaiken
>The standard doesn’t describe transformations; it describes behaviors

That is not remotely true - as you can see by looking up the example I
reference.

>For C, that’s not something we can realistically do in every case. There’s no
way to reasonably constrain the set of allowed behaviors after a program does
some crazy pointer operation like * (int *) rand() = rand(), because any
implementation of those constraints would more or less require an expensive
run-time check at every pointer access.

That's such a weird response. There is nothing in the proposal that requires
run-time checks. The proposal simply limits the ability of the
compiler/translator to make counter-factual assumptions or otherwise violate
the C semantics.

>What you could meaningfully ask is for particular situations (like negative
shifts) to be considered unspecified or implementation-defined rather than
undefined.

I don't want to do that: I want to clarify the meaning of "undefined behavior"
to curtail its use as a license to do arbitrary code transformations. One
implementation may trap on a negative shift, one may generate negative shift
instructions that the architecture treats as no-ops or whatever they do, one
might do sufficient static analysis to detect negative shifts and generate an
error message - but the compiler is not free to assume that the shift operator
has a negative value.

I'm not trying to remove undefined behavior from C. I am trying to correct a
mistaken analysis of what undefined behavior should mean. C is intended to
allow for non-portable operations, for example. These are "undefined" in the
standard because they depend on the implementation, not because the C language
should be specified so as to allow compilers to introduce hidden booby traps
or counter-factual assumptions into the implementation

~~~
obl
You should read the post you are replying to again, as it is well written and
precise.

Basically, the intent of the proposal is understandable but it does not make
sense as written.

The spec does not explain how to compile C code, it gives you a set of rules
so that you could in theory take some piece of C source and a full execution
trace and verify whether that trace was a conforming execution for that C
code.

You take the example of the negative shift again, and it is addressed in the
post you respond to : it's a perfectly fine proposal to demote negative shifts
from UB to something less permissive to the compiler.

However, if you do so, of course, you have to frame what behaviors are
accepted for a program doing such a negative shifts, just as you did in your
post !

So let's say : arbitrary (but consistent) values are fine, termination is
fine. That's great because it means we can still compile a shift to a CPU
shift and there's no UB anymore. Ok.

Same goes for signed overflow.

However this is a case by case process, and you will absolutely not be able to
ascribe meaning this way to many forms of UB. For example, as the GP said,
storing through an invalid pointer.

If you want to compile a pointer store to a store instruction without runtime
checks, then possible observable behavior include : other variable changing
values, function call not returning to their calling context, and even of
course running a full ROP chain exploit !

It's no surprise that in that case the specs says that the set of allowed
observable behaviors is anything.

~~~
jstimpfle
> It's no surprise that in that case the specs says that the set of allowed
> observable behaviors is anything.

And yet UB is frequently described as this scary monster that will eat your
hard drive.

Sure there are more UBs in C than there would need to be if the standard
restricted more on contemporary hardware. I think it still allows ones'
complement representation for example? But I guess it shouldn't be so bad as
long as compilers do not exploit the freedom to do anything that comes from
UB, more than they need to on a particular architecture?

~~~
cesarb
Compilers do not "exploit the freedom to do anything that comes from UB", they
just assume that the program does not have UB. This might or might not lead
the compiler to behave as if it was doing anything it wants, but it's not
screwing the programmer on purpose.

As for "UB is frequently described as this scary monster that will eat your
hard drive", setting aside that UB could be exploited by an attacker, it's not
hard to imagine, for instance, a recursive file deletion routine that is
affected by UB misbehaving and erasing more than it should.

~~~
jstimpfle
> As for "UB is frequently described as this scary monster that will eat your
> hard drive", setting aside that UB could be exploited by an attacker, it's
> not hard to imagine, for instance, a recursive file deletion routine that is
> affected by UB misbehaving and erasing more than it should.

I'd figure it's much more likely to happen as a result of a logic error
introduced by the programmer, without any UB involved.

~~~
cesarb
There's also this infamous example of UB:
[https://kristerw.blogspot.com/2017/09/why-undefined-
behavior...](https://kristerw.blogspot.com/2017/09/why-undefined-behavior-may-
call-never.html) (explanation: [https://blog.tchatzigiannakis.com/undefined-
behavior-can-lit...](https://blog.tchatzigiannakis.com/undefined-behavior-can-
literally-erase-your-hard-disk/))

~~~
jstimpfle
Thanks, that's interesting. And still, it's artificially constructed, so
concerning the point I made (actual logic bugs are more likely) I think it
stands.

IMHO this is clearly a compiler bug. Yes, it's technically allowed to do this
because calling a NULL function pointer is UB, but nobody wants such an
optimization (which is my first point from earlier). If the compiler makes
such a decision based on UB, shouldn't it be able to at least issue a warning?

Furthermore, why should this be optimized in the first place? If the user
decides to make a function pointer instead of a hard-wired inlineable call, he
is aware of the performance implications (which are negligible in 99.9% of the
cases). Where function call efficiency matters, nobody ever used a function
pointer.

Shouldn't the compiler architecture be able to avoid making this optimization,
simply by avoiding propagating the UB? It should simply not say "Calling NULL
can never happen". This way it would never make optimizations that make the
situation even worse. I think this would even be less optimization code!

Instead, when the UB happens, the programmer should get what he deserves
(which is, well, calling NULL, and most likely resulting in a segfault, which
is the right thing).

On the other hand, if the compiler can infer the value of a function pointer
based on propagation of values alone, I'm fine with hard-coding the address,
even though it's probably not worth the effort anyway.

~~~
cesarb
That's a "devirtualization" optimization. Suppose you have a C++ class with a
virtual method (or something similar coded in C), which only has a single
concrete implementation. Clearly, every place which calls that virtual method
through any pointer to that class can only be calling that single concrete
implementation, so the compiler can call the function directly instead of
through the function pointer, perhaps even inlining the called function.

Yes, that EraseAll example is silly, but how can the optimizer distinguish it
from non-silly examples? Perhaps there are other alternative functions which
also set that same function pointer, but they were all hidden by some #ifdef;
in that case, devirtualizing when there's only one option left will be a
performance win, and can expose other optimization opportunities.

~~~
jstimpfle
> devirtualization

Hmm, shouldn't it be possible to do that in some other way? After all, a
virtual function is something different than a function pointer (the vtable
holding a function pointer is only an implementation detail, isn't it?).

Anyway, I'd rather not have that optimization. When we don't want to pay a
small performance hit, we shouldn't make a virtual function in the first
place. It's good to see how advanced optimization tricks can combine in
dangerous ways with complicated language features!

> distinguish it from non-silly examples?

Yes, probably not possible. There is a real risk, but non-silly examples are
less likely than constructed ones.

> all hidden by some #ifdef

If you do it with the preprocessor, there is no need for function pointers in
the first place. Just make a #define instead of a variable.

Thanks again for the example, very interesting!

------
raphlinus
This sounds like Regehr's "Proposal for a Friendly Dialect of C" [1]. I
believe that was a well thought out argument, but ultimately did not make
headway [2].

[1]
[https://blog.regehr.org/archives/1180](https://blog.regehr.org/archives/1180)
[2]
[https://blog.regehr.org/archives/1287](https://blog.regehr.org/archives/1287)

~~~
iainmerrick
I haven’t seen a good explanation of the arguments against this kind of thing.
Who doesn’t want to fix the UB weirdness? Are there really many people worried
it will completely break optimization? If so I’d love to hear about it.

Even if you don’t agree with this proposal, or with Regehr’s, surely a
reasonable response would be “okay, yeah, it would be good to do something
along those lines, but not that; how about this?”

~~~
jcranmer
> I haven’t seen a good explanation of the arguments against this kind of
> thing. Who doesn’t want to fix the UB weirdness? Are there really many
> people worried it will completely break optimization? If so I’d love to hear
> about it.

If you're not a compiler writer, you probably have a mental model of undefined
behavior that works like this:

    
    
        if (expr->invokes_undefined_behavior()) {
          // Yay! Silly user!
          shit_on_users_code();
        }
    

Sure, you probably don't think it's so brazenly stated, but the general
sentiment from compiler users tends to be that all you have to do is delete
that if statement and everyone's happy.

But that's not how compilers work, not in the slightest. Instead, the
undefined behavior effects tend to work like a chain: you dereferenced a
pointer, therefore it's now safe to assume on any subsequent path that it will
always be safe to dereference that pointer, therefore we can move that later
load out of that loop even though we don't know if the loop will be executed
in the first place. And the reason that logic works is because we're allowed
to optimize under the assumption of undefined behavior in one of those steps.

Now in some cases--signed integer overflow and strict aliasing come most
readily to mind--there is a single knob you can twist to disable the undefined
behavior (and most compilers let you twist that knob with command-line flags).
Arguing whether or not the standard itself should dictate a different position
on that knob is one thing, and it's by no means a bad discussion to have. But
instead arguing "let's let you keep undefined behavior, but let's try to
specify what you can do with undefined behavior" runs into the problem that
it's _very_ difficult to actually specify what somewhat-constrained undefined
behavior really means. Look up the massive email threads on LLVM that try to
tease what poison and undef actually mean.

~~~
iainmerrick
_If you 're not a compiler writer, you probably have a mental model of
undefined behavior that works like this_

I find that very patronizing.

Yes, we mere compiler users complain about specific examples; that’s because
we keep finding specific examples that break in weird ways!

But at the same time, we realize that it’s a _general_ approach that causes
trouble. As I understand it: assuming that undefined behavior _cannot happen_
and optimizing on that basis.

I’m not arguing for specific small tweaks to fix a few known weirdnesses, or
even a more general adjustment to the notions of poison and undef (as your
mention of “massive email threads” suggests).

I’m arguing, first, that the entire approach seems to be misguided and not
working well; as evidence, many real-world examples of code doing extremely
unexpected things, and the position of the Linux kernel maintainers (surely an
important group of users by any standard).

Second, that the problem seems to be largely ignored by compiler writers; as
evidence, no changes or improvements, despite several proposals from
outsiders. You can point to massive internal debate on email threads but it
amounts to nothing if nothing comes out of it.

------
zokier
What exactly is the point of this article? Despite its title it feels neither
modest, nor like a good-faith attempt of a proposal. It more feels like a
cheap snipe against the standards group, regurgitating the same tired points.
Does anyone really believe that anything productive comes out of this?

~~~
tom_
No, but perhaps if people keep regurgitating these same tired points, compiler
developers will get bored of listening, and start to do something different.

As for the modesty, it's presumably a reference to this:
[https://en.wikipedia.org/wiki/A_Modest_Proposal](https://en.wikipedia.org/wiki/A_Modest_Proposal)

------
obl
Specifying behavior is a bit more complex than that. There is a wide range of
"unspecified" and the "full UB" one is the easiest from the spec point of
view. Using the shift example :

\- allowed to assume that the negative shift does not happen (what it says
now)

\- the value of each dynamic negative shift is unspecified but it must have a
well defined value, however the same shift operation in the source code could
return a different output given the same inputs dynamically (eg in a different
loop iteration or function call)

\- each source code shift is a well defined (unspecified on negative inputs)
function

\- all shifts in the program are a well defined (unspecified) function

So, for example, the last (least surprising) version means that you have to
compile every shift to the target CPU's shift and that you also have to use
the target's behavior when you constant fold a shift at compile time.

------
loeg
I think what the author actually wants from (1) is a transformation of common
(or maybe all) UB to be more strongly specified as implementation-defined.
That could be reasonable, with specific scope. I'm having trouble determining
the proposed scope from the post, though.

(2) seems like it is already specified; the trouble is in determining what
"the semantics of the program" are. This relates to (1) although it doesn't
seem like the author has a completely solid grasp of the legalities of the
language.

~~~
jcranmer
What the author wants, essentially, is to dictate that common disable-UB flags
(e.g., -fwrapv, -fno-strict-aliasing) be folded into the C standard. Well, he
probably knew that the C standards committee isn't going to accept that, so
instead he's trying to "only" disable the bits of undefined behavior that
don't screw over the user. Except he doesn't have a good enough grasp of the
standard to actually effect proper wording:

(1) essentially disables inlining as a side effect, at least if undefined
behavior is involved. (2) either is 100% redundant or means that every
implementation must strictly execute the C abstract machine with absolutely no
optimization permitted. (3) means you can no longer access any type with a
char _. You also couldn 't access the bits of a float by casting to an int_.

~~~
vyodaiken
it's bizarre to me that people make this argument, given the obvious fact that
those flags do not prevent optimization or inlining.

------
eftychis
The first proposal is extremely welcome. Still you need to expand on it I
feel. But yes, I think it is becoming the dream and nightmare along with
aliasing (...) of a lot of engineers. It has become ridiculously impossible to
write serious code without hitting the UB land.

The second although I think I get the reasoning, seems not well-defined to me.
I am not talking about expanding into it with concrete steps. You need to
define what kind of semantic invariance to require. You might just not want
the compiler deleting sections it thinks do not affect the output; you might
not want the compiler to alter nop segments for timing reasons (e.g. when
writing a cryptographic primitive). There is a wide variety of different
aspects of semantics and it is not clear what you propose. This have been a
discussion topic among cryptographers. Or you are a kernel programmer and
actually do not want checks removed or aliasing messed with etc. (One reason
people compile their kernels with specific options -- see Linus' rants about
UB if you want to pull some extra hair of your head.)

Anyways -- I am afraid at times GCC might be too far gone to tone down
undefined behavior. Anyone working on a compiler has any input if there are
plans, in the (unlikely) case the next WG decides to go sane on undefined
behavior, how their compiler is going to address that new standard?

~~~
jabl
> Anyone working on a compiler has any input if there are plans, in the
> (unlikely) case the next WG decides to go sane on undefined behavior, how
> their compiler is going to address that new standard?

Disclaimer: I have contributed on my own time to one of the GCC frontends, so
I guess I know a little bit more about compilers and the compiler development
than the average Joe, but I wouldn't call myself an expert nor am I in a
position to do any decisions wrt this for GCC. With that out of the way:

\- You can see compiler development, particularly the optimizer stuff, as a
game, where the standard limits what you can do, and the goal is to make SPEC
CPU [1] run as fast as possible (your employer says which target CPU(s) you
should focus on).

\- The standard is the "contract" between the user and the compiler writers.
Compiler writers do take standards conformance pretty seriously.

\- If the standard changes, say to limit undefined behavior, compiler
developers adapt to the new rules, and the game continues. No big drama here.

\- AFAICT GCC developers wouldn't complain that loudly about "losing"
previously allowed optimizations. There is (again, AFAICT) sympathy towards
the viewpoint of changing the standard towards improving robustness.

\- What you won't see is GCC, or any other compiler, doing it on their own
without the standard backing them. That would just lose performance compared
to the competition, and wouldn't help users write portable code anyway.

[1] Yes, there are other benchmarks too, but SPEC CPU is the most influential
one.

------
geofft
> _6\. EXAMPLE Code that checks to see if a pointer value is null cannot be
> omitted during translation solely on the basis that the pointer is
> dereferenced before the check and that earlier dereference would produce
> undefined behavior if the pointer value was null._

Here's the thing I don't quite understand: I gather that this is advocating
for something like GCC's -fno-delete-null-pointer-checks to be part of the
spec, but, how is -fno-delete-null-pointer-checks actually helpful?

I _think_ it's because if you write code like this:

    
    
        void foo(struct foo *param) {
            int n = param->length, i;
        
            if (param == NULL) {
                return;
            }
        
            for (i = 0; i < n; i++) {
                do_something(param->data[i]);
            }
        }
    

with -fdelete-null-pointer-checks the compiler might optimize it in two
successive steps to delete the check (because param->length was already
dereferenced) and then move the dereference down to the beginning of the for
loop, and with -fno-delete-null-pointer-checks the compiler might skip the
first optimization and keep the second.

But for -fno-delete-null-pointer-checks, you're relying on the second
optimization, moving the dereference later. Otherwise you'd just have
segfaulted already (or, in a kernel context, the read might have _succeeded_
from a userspace page mapped at the null page). Right?

It seems reasonable for the compiler (or some other tool) to produce a hard
error here and prevent this code from compiling, saying, hey, this is clearly
doing something unwanted. It doesn't seem reasonable to expect the compiler to
do certain optimizations to make your code safe, _without_ which your code
would have a security vulnerability, and also get mad at the compiler for
doing more optimizations that return it to the original state of things.

> _The limitation of lvalue access by types has never made much sense and
> required a undefensible exception for character pointers._

My understanding of type-based alias analysis is that it's required for code
like this to optimize the way you'd want:

    
    
        struct string {
            size_t len;
            char *data;
        }
    
        void copy_string(struct string *dest, struct string *src) {
            size_t i;
            if (dest->len < src->len) return;
            for (i = 0; i < src->len; i++) {
                dest->data[i] = src->data[i];
            }
        }
    

Otherwise the compiler would have to worry that dest->data would overlap with
src->len and re-read it every time!

You can do this without type-based alias analysis in a language that gave you
different tools for managing variables, but doing so backwards-compatibly in C
seems difficult, and this proposed rule does not seem to help.

~~~
vyodaiken
For certain types of image manipulations or other computations, it may well be
that the pointers are supposed to refer to overlapping regions. This is the
kind of problem that, as Dennis Ritchie pointed out, bedeviled FORTRAN for
ever. I think the solution is to fix "restricted" and let programmers opt-in.
If the programmer doesn't do that, or gets it wrong, too bad.

~~~
comex
The issue isn’t just that src->data and dest->data might overlap, which TBAA
doesn’t forbid anyway (although that’s what restrict can help with). It’s that
dest->data might alias src-> _len_ , and magically change the length midloop.
I doubt there’s any legitimate image manipulation where that would happen, but
the compiler doesn’t know that...

~~~
vyodaiken
This is exactly why I don't propose to do away with UB but to limit what
compilers can do with UB. The programmer in C cannot be prevented from writing
terrible code where e.g. a loop counter is written over by an aliased pointer
in the middle of the loop - the current standard doesn't prevent it either. In
fact, strictly conforming C code can use char pointers to randomly write over
any aliased variables if I understand it correctly. The positive thing about
UB is that it makes this the programmers problem, not the compilers problem.
The negative thing is that in current interpretations, the compiler is free to
detect such an error and then instead of either failing to compile with a
warning or just letting things happen as the programmer has specified, it can
introduce global program failures because "it's impossible to have UB".

------
crooked-v
As somebody primarily focused on other languages, I've always found it
absolutely bizarre that in C-land it's considered even remotely acceptable for
"bad stuff" to result in invisible-to-the-user transformation of code instead
of actual error handling.

~~~
palotasb
I guess you're focused on languages that don't have the following two
constraints from C: (1) be as fast as possible (2) compile ahead-of-time. The
"weird" behavior in case of UB in C/C++ is a direct result of these two
constraints and the fact that the transformations you're referring to are only
affect code that already has _programming bugs._ The standard already
specifies that optimizations don't change semantics -- and really they don't
-- for bug-free code. Adding runtime error checks to that code only helps
buggy programs and pessimizes the performance of correct ones. This is a
tradeoff C/C++ does not make.

~~~
pjmlp
Using seat belts is an inconvenience that only gets in the way of good
drivers, designed to help drivers that aren't capable of handling traffic.

~~~
palotasb
That's a good analogy except that with traffic, you have factors outside your
control that can cause a crash. Avoiding bugs in code is theoretically
completely within the programmers control. There are various tools that help
with this in different languages to compensate the fact that programmers are
human: compiler warnings, best practices and linters, sanitizers, testing,
static typing, runtime checks etc. Different safety mechanisms offer different
tradeoffs both for cars and programming.

One could argue with the your analogy that UB in C is silly because it assumes
programmers who make no mistakes. In reality, writing C code can be economical
(adequate safety, low cost, high performance) -- pricing in all the human
errors -- and making the proposed would also cost performance and possibly
money for some, that's why there is pushback on such proposals.

~~~
pjmlp
In the utopian world that C developers are all 10x, have 100% control over the
complete source code of their applications and don't depend on third party C
libraries.

~~~
palotasb
What I implied by "programmers are human" as _exactly_ that they are 1x and
make mistakes. The tools help in the situations where "100%" assumptions
cannot be made. But you can't put a seat belt on everything and contain every
issue because it will cost more.

But you must also know that you _must_ trust at least part of the environment
your operating in. There will never be a language or a tool that can make a
program useful and correct when nothing can be trusted. C/C++ regarding UB is
just explicit about trusting the code that is currently being compiled for the
sake of optimizing code that actually is. I get where you're coming from, but
this is one of the possible solutions, and a very widely accepted one given
that the whole world runs on C.

Managed languages might avoid some class of bugs, but they will always retain
some, and I'm not sure it is possible to guard against all without solving the
halting problem. Some C/C++ people might on the contrary find it bizarre that
your languages do absolutely unnecessary bounds checks. None of them are
better than the other on an absolute scale, it all depends on what you want to
use the tool for.

~~~
pjmlp
Ada is not a managed language for example, yet it avoids such errors.

That is the typical C hand waving that every language has errors, so no big
issue.

Of course there are no perfect programming languages, however it is quite
different having to handle _" Logic Errors"_ or _" Logic Errors + Memory
Corruption Errors + UB Errors + Implicit Conversion Errors"_.

Even 10% reduction is a major improvement.

~~~
vyodaiken
Ada overflow semantics are precisely what programmers assumed were C overflow
semantics on architectures that roll-over on signed arithmetic (e.g. basically
every processor architetcure in current wide use). The original intent of the
C "undefined behavior" category, as far as I can tell, was to in this case
permit compilers on dissenting architectures to make another choice. For
example, on the Itanium you might want to have a trap on overflow. Because C
is comfortable with "it works differently on different processors". So it is
my intention to repair the obscurity in the language by spelling out what I am
100% sure would have been obvious to C programmers and language developers at
on time.

------
mpweiher
Yes! Please return the standard and the compilers to sanity.

------
iainmerrick
Thank you for doing this.

------
ericpauley
Since nobody has mentioned it:

[https://en.m.wikipedia.org/wiki/A_Modest_Proposal](https://en.m.wikipedia.org/wiki/A_Modest_Proposal)

------
YouAreGreat
> conforming implementations cannot transform code on the basis that some
> behavior left undefined by the standard is impossible

You may have misunderstood how rogue optimizers obtain their license to
miscompile.

It is _not_ by assuming that the "behavior left undefined is _impossible "._

As boring and tautological as it sounds, it is by assuming that the operation
with "behavior left undefined", _when it actually happens,_ has undefined (and
therefore arbitrary) behavior.

