
The Strict Aliasing Situation Is Pretty Bad - pascal_cuoq
http://blog.regehr.org/archives/1307
======
kbenson
_Although I have no evidence that it is being miscompiled, OpenSSL’s AES
implementation uses chunking and is undefined._

Oh, that's nice. :/

~~~
kazinator
It's nonsense. The function is external, called from a separately compiled
file. The pointer comes in as a char *. The code checks its alignment before
assuming it can be cast to a block. There is no way in it could be
"miscompiled".

The ivec argument could in fact have come from an object that is of type
aes_block_t. The only thing which might reveal that it didn't is wrong
alignment. In other regards, there is no way to tell.

Lastly, any cross-compilation-unit optimization which could break code of this
type is forbidden, because ISO C says that semantic analysis ends in
translation phase 7.

I'm looking at C99, not the latest, but I think it's the same.

In translation phase 7 (second last), "The resulting tokens are syntactically
and semantically analyzed and translated as a translation unit." Note the
"semantically analyzed": semantic analysis is where the compiler tries to
break your code due to strict aliasing.

In translation phase 8 "All external object and function references are
resolved. Library components are linked to satisfy external references to
functions and objects not defined in the current translation. All such
translator output is collected into a program image which contains information
needed for execution in its execution environment."

No mention of any more semantic analysis! So unless somehow the mere
resolution of external symbols can somehow break OpenSSL's AES, I don't see
how anything can go wrong.

One thing I woudl do in that code, though is to make sure that it doesn't use
the original ivec pointer. In the case where "chunking" goes on, it should
just cast it to the block type, and put the result of that cast in a local
variable. All the ememcpy's, load/store macros would be gone, and the
increments by AES_BLOCK_SIZE would just be + 1.

~~~
haberman
Citing the translation phases in the standard as evidence that undefined
behavior is ok, as long as it's divided between two translation units, strikes
me as wishful thinking.

~~~
kazinator
Undefined behavior is the absence of requirements: there not being any
requirements for some situation.

Suppose a document tells you that for some special situation X, there is an
absence of requirements. However, suppose that some other general rules
elsewhere in that document in fact imply a requirement for that situation.

That just means that the claim that situation X has no requirements is
incorrect.

For instance, ISO C says that two struct types appearing in separate
translation unit are only compatible of they have the same typed members in
the same order ... _with the same names_.

This says that if you do aliasing with otherwise identical structures that
don't have the same names, the behavior is undefined: i.e. that there is no
requirement that it work.

But, we can infer that it must work by the logical fact that during the
semantic processing of one translation unit, the translator has no clue what
the names of struct members are in another translation unit. They disappear at
translation time and turn into offsets.

I mean, we can fwrite a struct to a file, right? We can send that file over a
network. According to ISO C, every program (or at least every C program) will
have to use a structure with the correct names to fread that area of the file!
Ridiculous!

Suppose we take ISO C and add a statement to it like, "the consequences are
undefined if one of the operands of the + operator is the integer 42". The
rest of the document would still be exactly what it is, and if we strike out
that sentence with a black marker, nothing has changed. The rest of the
document continues to inform us that adding 42 to something is well-defined
(in the absence of overflow, or overflow-like issues with pointer displacement
and so on).

Basically, it's a contradiction: the document gives a description which adds
up to some requirements, but then some sentence tries to take them away.

In such a situation, we can just proceed as if the requirements apply. That is
to say, when a requirement conflicts with the claim that there is no
requirement, just let the requirement prevail.

(In a situation where conflicting requirements are asserted, it's a different
story, of course.)

~~~
xorblurb
You can't infer shit, because of the way current compiler writers are
interpreting the standard today. At one point in the 90's it was obvious for
the entire planet including compiler authors that some undefined behaviors did
not apply for a given target architecture, so obviously obvious that nobody
would even require to have it specified in the compiler doc (still nice to
have, but you would not be too much angry if it does not appear)

Now they just add their "optimizations" at the highest levels so of course
even for targets where it should makes no sense, and without asking for your
permission, and even by default, and even some they consider "aggressive". So
either you have provisions to avoid all that shit, like having a guy disabling
all the new ones each time you upgrade you compiler, and you better have some
defenses in depth, and I agree with you that using TU as boundaries is also a
good idea, if your compiler+build-sys have an option to NOT do WPO.

But it is just not in any way guaranteed by the ISO standard, and still just
an implementation detail from its pov. And honestly that's a problem. Because
compiler writers will takes more drugs and come up with new imaginative way to
break your code more, in the name of their "strictly-conformance and nothing
more" crazy ideal.

------
adrianN
See also this older post
[http://blog.regehr.org/archives/959](http://blog.regehr.org/archives/959)

------
eternalban
This was an eye opener for me. Bell Labs tech is as usual a mountain of hidden
complexity hiding under a "simple" facade.

No more excuses for me. Time to learn Rust.

~~~
aidenn0
C is actually quite simple, even the aliasing rules (I think the aliasing
rules all fit on about a quarter page). Programming in C though is anything
but.

The tension between weakly typed pointers and the desire to generate efficient
code is where there is a problem. More or less anywhere you violate the
aliasing rules you are doing something that Fortran doesn't allow at all, and
by disallowing it semantically the hope was that C programmers could have
their cake and eat it too. The reality is that all systems code should
probably be compiled with alias analysis disabled.

~~~
xorblurb
I completely agree.

A spec of a quarter page can already be incredibly convoluted and with very
hard to anticipate consequences -- even more so when crazy people are
interpreting it without caring about the consequence of their acts in the real
world. And C is not just about aliasing rules; the current situation is that
ANY undefined behavior is a landmine waiting to kill you, regardless of
whether is seems to makes sens for your target architecture. And that is
mostly because compiler writer have an insane interpretation of the standard:
the definition of "undefined behavior" is "behavior, upon use of a nonportable
or erroneous program construct or of erroneous data, for which this
International Standard imposes no requirements"

A key word here is "nonportable" and it is clearly not considered often enough
by some compiler writers, who generally prefer to see all undefined behaviors
as a licence to "optimize" your code without carrying too much about warning
you about potential bad side effects because it's "hard" according to them.

This does not make even the beginning of any sense. If it is not what many
programmers are expecting (a majority ? -- most coworkers I know including my
direct boss are not even aware of all that mess), is costly during the
translation phase, has unclear/unquantified runtime performance benefits, is
dangerous in the real world and is hard to detect when bugs are activated by
those "optimizations", then WHY they are doing them in the first place? From
an engineering point of view this is just plain insane. Correct executions and
in depth safety are extremely valuable, and only becoming more so year after
year, and when they pretend that it's not their fault that programs are
breaking they are being even more ridiculous; a compiler does not exist in a
vacuum, and neither just to reproduce itself and satisfy the curiosity of
geeks for mathematical logic.

Obviously some amount of alias analysis can be useful, and this is clearly one
of the topic really intended from scratch in the standard to address some
performance issues, but maybe it would be enough to explicitly identify what
you want to not alias. Seeing the bug when they discuss about allowing uint8_t
to not behave in _general_ as an unsigned char in regard with aliasing is just
plain disgusting and makes me lose yet again a portion of the tiny remaining
trust I had in them.

It will soon get to the point that C/C++ will not be realistic languages to
consider if you want any kind of reliability. Maybe I'm even deluded in
thinking this is not already the case.

~~~
aidenn0
I get a bit fed up when software developers complain about C compilers and
blame the developers, as though compiler developers are somehow a different
breed or something. It's all software, and compilers are actually one of the
easier things to write.

Somehow, the developers of gcc, clang, and various commercial compilers are
all crazy, while people who work on any other project in C are sane? Why
haven't the sane software developers forked an open-source compiler and
implemented sane semantics?

Blaming the state of compilers on compiler developers without understanding
their motivations for making engineering choices is intellectually lazy,
particularly for other software developers.

~~~
xorblurb
What you write is all very meta, so maybe enlighten us on their motivations
and why they should continue in that way? And on how using a language is the
same as implementing it and how it seems that the application space has
magically folded into a single point last night?

I know already too much for my taste about the reasons they gave to come with
such horribly risky designs. Example: "it's difficult to properly warn where
we are doing dangerous optims". I'm not buying it. Don't do them in the first
place or use more heuristics. Publish your data about your risk / benefit
analysis. I think they made strategic mistakes. "Everybody" in the academia
and security / safety related business is saying that. And no, that won't be
solved with dynamic checkers. They should concentrate on optimizations
minimizing the risk of new behaviors concretely appearing, and heuristics
maximizing the detection of the intent of the programmer. That sounds informal
as hell, because this is, but in the real world solving a problem does not
always mean finding the perfect solution to an equation, and good enough
approx are often the best thing to seek. Compiler writers know that. They are
just not seeking the good thing. The "smartness" they think they are creating
though their imaginative use of undefined behavior is as smart as my ass,
because on real projects those kind of sufficiently smart compilers are
indistinguishable from an adversary.

I have recently implemented some kind of compiler, though not C to machine
code. In the real world it's not easy. It's messy. It's made of blood and
tears. You have to care about all kind of little details for your users. You
have to make the junction between two domains that sometimes have quite
different semantics -- there is never a perfect solution to that. And all of
this, with the minimal risk to be misused. If you don't care for your users,
their is no point. I've the greatest respect for the authors of CLang, which I
used for the front end. It's not perfect, but it is quite good and it does the
work. But sometimes, when you feel that somebody effort really is misplaced,
you better tell him (or the community, my voice is not particularly original),
and explain your reasoning. Otherwise, you effectively won't be entitled to
complain latter, if you wait too long.

So now what do we have: two domains, C and various target architectures, that
used to be with quite some low impedance mismatch, by design, are considered
radically different. The expectations of most users have nothing to do anymore
with the expectation of how the compiler writers thinks the users should write
their code (or have written it in the past, for less maintained software).
That won't ends well. Actually, we are already in the mess, while we don't
especially needed more than we had already.

So go on and please explain your POV. But anyway, no, I won't fork and write a
"friendly C" compiler overnight. Neither will maintainers of cryptography
softwares, probably. I'm just yet another datapoint that will loose some time
in adding all the -fdisable-insanity of the day anywhere he passes. Because
somebody else somewhere else in the world thought that would be a good idea to
infer proofs using rules written for the least common denominator of all
computers to detect if you read the n1570 over and over again enough times to
obtain the privilege to get a sane translation of your code that actually will
anyway only ever run on x64_86, thank you very much.

The same debacle somehow happened about memory models (see how they have been
received for kernel work). If you only ever care about the abstract, concrete
real prospective users won't greet you as you might have expected. Rightly so.
Especially when your formalism has been proven unimplementable _and_ unsound.

~~~
the_why_of_y
I don't think it's going to be a fruitful exercise to demand that implementers
of ISO C/C++ change their implementation to guarantee certain properties of
programs that ISO C/C++ consider to be invalid, and thereby weaken their
competitive position vs. other implementations.

The root cause is the ISO C standard itself, so if you want any change in that
direction, the best approach would be to join the ISO C working group and make
proposals to replace various undesirable _undefined_ behaviors with, at the
minimum, _implementation-defined_ behaviors (starting with signed integer
overflow, perhaps). Probably there aren't any vendors of one's complement
machines and the like left to veto your proposals.

The C11/C++11 memory models for the most part don't solve a problem that
kernel developers have, because all kernels already contain tested solutions
to the same problem, so one would not expect kernels to quickly exchange all
their tested and highly performant concurrency primitives. The only benefit
for them is that the memory model effectively prohibits implementations from
doing some optimizations that would be valid only under the assumption that
there is only one thread of execution, and this benefit is _implicit_ , i.e.
you don't need to use any of the new language features or standard library
headers to benefit.

------
haberman
I thought this article was unnecessarily dire.

One section claims "Physical Subtyping is Broken", where "physical subtyping"
is defined as "the struct-based implementation of inheritance in C." I assume
this means the typical pattern of:

    
    
        typedef struct {
           int base_member_1;
           int base_member_2;
        } Base;
    
        typedef struct {
           Base base;
    
           int derived_member 1;
        } Derived;
    

The article claims physical subtyping is broken because casting between
pointer types results in undefined behavior. The article gives this example:

    
    
        #include <stdio.h>
         
        typedef struct { int i1; } s1;
        typedef struct { int i2; } s2;
         
        void f(s1 *s1p, s2 *s2p) {
          s1p->i1 = 2;
          s2p->i2 = 3;
          printf("%i\n", s1p->i1);
        }
         
        int main() {
          s1 s = {.i1 = 1};
          f(&s, (s2 *)&s);
        }
    

I agree this example is broken, but casting between pointer types in this way
is totally unnecessary for C-based inheritance. You can do upcasts and
downcasts that are totally legal:

    
    
        Derived d;
    
        // Legal upcast:
        Base* base = &d->base;
    
        // Legal downcast:
        Derived* derived = (Derived*)base;
    

So I don't think the article has proved that "Physical Subtyping is Broken."

The next section says that "Chunking Optimizations Are Broken," because code
like this is illegal:

    
    
        void copy_8_bytes(char *dst, const char *src) {
          *(uint64_t*)dst = *(uint64_t*)src;
        }
    

While this is true, such optimizations are generally unnecessary. For example,
write this instead as:

    
    
        void copy_8_bytes(char *dst, const char *src) {
          memcpy(dst, src, 8);
        }
    

If you compile this on an architecture like x86 that truly allows unaligned
reads, you'll see that modern compilers do the "chunking optimization" for
you:

    
    
        0000000000000000 <copy_8_bytes>:
           0:   48 8b 06                mov    rax,QWORD PTR [rsi]
           3:   48 89 07                mov    QWORD PTR [rdi],rax
           6:   c3                      ret
    

It says next that "int8_t and uint8_t Are Not Necessarily Character Types."
That is indeed a good point and probably not well-known. So I agree this is
something people should keep in mind. But most of this article is warning
against practices that are generally unnecessary and known to be bad C in
2016.

It's true that a lot of legacy code-bases still break these rules. But many
are cleaning up their act, fixing practices that were never correct but used
to work. For example, here is an example of Python fixing its API to comply
with strict aliasing, and this is from almost 10 years ago:
[https://www.python.org/dev/peps/pep-3123/](https://www.python.org/dev/peps/pep-3123/)

~~~
regehr
Josh, what's your opinion about this situation?

[https://goo.gl/3hz0em](https://goo.gl/3hz0em)

~~~
haberman
It's undefined. But the same thing would be undefined in C++, a language that
has inheritance built-in: [https://goo.gl/shOJi1](https://goo.gl/shOJi1)

You can't downcast to a derived type if the object isn't actually an instance
of the derived type. That seems straightforward, no?

~~~
regehr
It doesn't seem straightforward to me: you're using words like base and
derived that aren't in the C standard.

~~~
haberman
If we talk in terms of concepts that exist in the C standard, we would say
that you can't cast an object to pointer-to-X unless your pointer actually
points to an X.

The reason your example is illegal is that you are casting to pointer-
to-"struct derived", but the thing being pointed to is not actually a "struct
derived."

The "physical subtyping" pattern works because the C standard says that a
pointer to a struct, suitably converted, also points to its first member. So a
pointer-to-Derived, converted to a pointer-to-Base, points at Derived's first
member. But a pointer-to-Base doesn't point at a Derived unless that object
actually _is_ a Derived. So the downcast is only legal if the object actually
is a Derived.

~~~
regehr
Looks like your other comment hit the max reply depth so this will need to
finish up, but in any case I don't agree with your reading of the vice versa.

~~~
regehr
Replying to myself because depth limit.

Let me try to think of a good way to update the post to capture this better...

~~~
haberman
There isn't actually a depth limit (or if there is we haven't hit it yet :).
HackerNews just hides the "reply" link for 5 minutes or so to cool down
flamewars.

You can work around this by clicking on the link for the post itself (ie. "3
minutes ago") which allows you to reply immediately.

------
Pxtl
... I am so happy I don't code in C right now. That's icky.

~~~
DannyBee
C++ is just as bad :)

