

Null Pointer Dereferencing Causes Undefined Behavior - DmitryNovikov
https://software.intel.com/en-us/blogs/2015/04/20/null-pointer-dereferencing-causes-undefined-behavior

======
_kst_
The standard offsetof macro is sometimes defined as:

    
    
        #define offsetof(st, m) ((size_t)(&((st *)0)->m))
    

That expression has undefined behavior, since it dereferences a null pointer.
That doesn't mean it's an invalid definition of offsetof _for a given
implementation_.

Saying that it has "undefined behavior" means that the C language standard
says nothing about how it behaves. If its behavior _for the current compiler_
happens to satisfy the language requirements for offsetof, then it's a
legitimate implementation. Code that implements the standard library is free
to do whatever it likes, as long as the resulting behavior is correct.

Part of the problem, I think, is the way the question is phrased. Quoting the
linked article:

> The programmers' community divided into two camps. The first claimed with
> confidence that it wasn't legal while the others were as sure saying that it
> was.

The word "legal" doesn't even occur in the C standard. Some program constructs
violate constraints or syntax rules; any such violation requires a compile-
time diagnostic (which may be a non-fatal error). Other constructs have
"undefined behavior"; the standard places _no_ restrictions on what compilers
can do for such constructs. Only a `#error` directive actually requires a
program to be rejected.

~~~
userbinator
> the standard places _no_ restrictions on what compilers can do for such
> constructs.

But it does list some suggestions on what could happen, one of which is
"behaving during translation or program execution in a documented manner
characteristic of the environment", which is what most C programmers expect of
the language. Violating this implicit assumption is what makes compilers user-
hostile; choosing the worst possible behaviour, just because the standard does
not define it, is never the right thing to do.

Related reading:

[http://blog.metaobject.com/2014/04/cc-
osmartass.html](http://blog.metaobject.com/2014/04/cc-osmartass.html)

[http://thetrendythings.com/read/7057](http://thetrendythings.com/read/7057)

[http://blog.regehr.org/archives/1180](http://blog.regehr.org/archives/1180)

~~~
_kst_
That suggestion is pretty vague; the only way a compiler could violate it is
by not documenting its behavior. The behavior itself can still be completely
arbitrary.

~~~
mpweiher
Well, before the first ANSI standard, ALL behavior was "undefined", and yet we
got many working compilers and tons of working code.

~~~
_kst_
Before the first ANSI standard, we had the C Reference Manual in the back of
K&R1. The definition was merely less rigorous (and that lack of rigor often
led to complex nests of "#ifdef"s). If a compiler generated code that
evaluated 2 + 2 and yielded a result of 5, we could confidently say that
compiler had a bug.

------
kragen
This is ridiculous. The C standard was apparently written in order to allow
compilers to introduce subtle security holes into previously-working code, all
in the name of a fraction of a percent of performance increase. It’s totally
irrational, especially in an epoch where C is no longer the language you’d use
to get maximum bit-twiddling performance anyway — because it won’t run on your
GPU!

See [http://blog.regehr.org/archives/761](http://blog.regehr.org/archives/761)
for a sarcastic take on the problem,
[http://blog.regehr.org/archives/880](http://blog.regehr.org/archives/880) for
a more serious description of the problem, and
[http://blog.regehr.org/archives/1180](http://blog.regehr.org/archives/1180)
for the "Proposal for a Friendly Dialect of C" that proposes to actually solve
it.

~~~
lmm
> The C standard was apparently written in order to allow compilers to
> introduce subtle security holes into previously-working code, all in the
> name of a fraction of a percent of performance increase.

Yes it was. Or at least, that's the constituency C currently has - and
understandably C continues to serve it. There needs to be a language for that
kind of people, and that language is C.

If you don't want that, _stop using C_. It's not hard.

~~~
mpweiher
I've been using C since before there even was a standard. Why should I ditch
the language because <deleted> have hijacked the language spec?

~~~
lmm
Because they're the ones who the language is valuable to. You have plenty of
other options; they don't.

~~~
mpweiher
For the sorts of semantics they are looking for, FORTRAN seems to be a good
option. Or keep the crazy in C++ land?

What are my options if I want a simple, easy to understand language that is
close to the execution model of current machines?

------
quotemstr
This article highlights yet more "gotcha" optimizations that, IMHO, do more
harm than good. There is no reason that any compiler _must_ miscompile &x->y,
where x == NULL. What's undefined in C can be well-defined in POSIX or in the
documentation of a specific build environment. It would behoove higher-level
standards to make this construct well-defined even if C itself doesn't.

~~~
pcwalton
Being able to assume that a pointer is not null opens up some really
interesting optimizations. For example, if you know that a pointer is
dereferenceable, then you can speculatively hoist loads of it out of a loop as
long as there are no possibly-aliasable pointer stores in between. That's why
it's helpful for C compilers to be able to aggressively infer that pointers
are not null.

~~~
ufo
Just wondering: how much do these particular optimizations end up being worth
in the end? Is it something noticeable or is it like array bounds checking
where its almost always OK to leave the checks in the end?

~~~
joosters
It's impossible to give a general answer. You could have a tight loop in a
performance critical part of your code, where optimising out the NULL check
removes (say) 25% of the loop code. Or, the check might be negligible. It
depends upon the code.

That's why IMO people saying that the compiler shouldn't bother with these
'excessive' optimisations are wrong. It might be worthless for them and their
programs, but might be valuable for others. Don't force your priorities upon
other people.

~~~
pcwalton
Yes. I'd also add that loop optimizations tend to magnify in importance when
you consider that the loop might be vectorized. Especially since the state of
scatter/gather SIMD intrinsics is so poor at the moment...

------
adamtj
I'm sure he's right, but I'm not so sure he actually proved it. The real
question is what happens when you deference a null pointer with '->'. He
doesn't show us what the spec says on that, meaning there's a gap in his
logic, leaving the quoted sentence in bold unsupported. I don't know but would
bet that bolded sentence isn't even necessary. The undefinedness probably
comes from the silence of the spec on what happens when you dereference a null
pointer with '->'.

As he showed, a null pointer cannot point to any object, and since a lvalue
that does not designate an object gives undefined behavior, saying '* podhd'
would clearly give undefined behavior when 'podhd' is a null pointer.

However, instead of '* podhd' evaluating to a non-existant object, we have
'podhd->line6' asking for the object some particular offset from a non-
existant object. It's entirely possible that a sufficiently insane spec might
actually define such an operation.

What the spec says on that point is the key. That's where the undefinedness
must come from, but he doesn't talk about it or quote from the spec on '->'.
He just assumes it, which leaves his original point unproven, even if it is
correct.

So why would anybody want to define legal behavior for dereferencing a null
pointer with '->'? Well, some people start college funds for children who have
yet to be conceived. If non-existant people can have account balances, why
can't non-existant objects have properties? It's crazy, but it's not crazy
like Intecal, where syntax errors are legal statements that you actually use
for productive purposes. [http://catb.org/~esr/intercal/ick.htm#Syntax-
Error](http://catb.org/~esr/intercal/ick.htm#Syntax-Error) See also the
statement "COME FROM".

~~~
numeromancer
> As he showed, a null pointer cannot point to any object

From my reading, he didn't even show that:

>> _If a null pointer constant is converted to a pointer type, the resulting
pointer, called a null pointer, is guaranteed to compare unequal to a pointer
to any object or function._

So a null pointer could point to an object, it just wouldn't compare equal to
any pointer, even itself.

~~~
ben0x539
Null pointers comparing equal to null pointers is kind of important though.

------
notacoward
The problem comes from the assumption that NULL is the same as zero. Even the
author makes that assumption, when he gives an example that explicitly uses
zero and then talks about it as if it had used NULL. In fact, the two have
never been the same. I've worked on machines that represented NULL with a
different value/bitpattern, and yes, it smoked out a lot of these bad
assumptions. NULL can never point to valid memory, by language definition. An
explicit zero can, from a language standpoint, even if the runtime/OS might
disagree. ;) Thus, uses like that in offsetof are quite valid.

~~~
pcwalton
But NULL must, broadly speaking, be the same as zero.

C standard 3.2.2.3: "An integral constant expression with the value 0, or such
an expression cast to type void * , is called a null pointer constant."

C standard 4.1.5: "The macros are... NULL, which expands to an implementation-
defined null pointer constant."

See [1] for more info.

[1]: [http://stackoverflow.com/questions/2599207/can-a-
conforming-...](http://stackoverflow.com/questions/2599207/can-a-conforming-c-
implementation-define-null-to-be-something-wacky)

~~~
notacoward
Casting zero to a pointer yields a null pointer. That doesn't preclude other-
valued null pointers, or ensure that a cast the other way will yield zero.
That's where the "undefined" part comes from.

~~~
_kst_
Casting a _constant_ zero to a pointer type yields a null pointer. The
conversion is done at compile time, so the compiler knows that a null pointer
is the required result.

There is no requirement for an integer-to-pointer conversion to retain the bit
pattern of the integer value, any more than an integer-to-floating-point
conversion is required to do so.

In most modern implementations, conversions among integer and pointer types of
the same size are trivial, simply reinterpreting the bits as a value of a
different type.

A conforming C or C++ compiler can use, for example, the bit pattern
0xFFFFFFFF to represent a null pointer. Converting a constant 0 to a pointer
type would yield a pointer with the representation 0xFFFFFFFF. Converting a
non-constant 0 to a pointer type does not necessarily yield a null pointer;
the result is implementation-defined, and may be indeterminate garbage.

------
DominikD
I find this entire debate absurd. Pragmatic approach would be to refrain from
using construct that are known to be controversial. By controversial I mean
"there are reasons to believe that under certain conditions this may be
dangerous, even though most people agree on what reasonable behavior is in
this case". Don't write code that can bite you. The "struct foo *bar =
&baz->qux;" is not your child. Kill it by refactoring it into something that's
known to be obviously correct.

~~~
eps
Refactoring for the sake of C pedantry rather than legitimate portability
concerns is not a _pragmatic_ approach.

If this code were a part of some abstract "spherical horse in vacuum" project
that may potentially build on esoteric platforms using random C compilers,
then, yes, it might be worth re-factoring. In the context of Linux kernel
project given the toolchain used the code is perfectly fine as is.

The debate is silly though, can't argue with that.

~~~
DominikD
Linux kernel has official support for over 20 different ISAs. Some of the
cores supported are pretty esoteric: BlackFin, SuperH, Etrax. Most of the
problems with optimizing compilers - around UBs and others - come from
architecture-specific optimizations. Problem discussed here is not abstract -
people writing kernel code encounter these issues from time to time. I did, my
friends did, we will be bitten by UBs in the future.

Code my team is working on at the moment was written for a completely
different architecture than it is run on right now. Sudden platform shifts
happen, like when tsunami affected Renesas' ability to manufacture chips used
in automotive. It IS pragmatic to avoid code that can potentially introduce
problems in the future. Perhaps not for every piece of software, but low level
one? Definitely.

------
scott_s
If you want to read more than can handle about undefined behavior in C, check
out John Regehr's blog ([http://blog.regehr.org/](http://blog.regehr.org/))
and academic papers
([http://www.cs.utah.edu/~regehr/papers/](http://www.cs.utah.edu/~regehr/papers/)).
He also posts here on HN.

------
TheLoneWolfling
Ah, the wonders of C, where the obvious solution to "we can't guarantee <x>
because it may cause problems on weird hardware" is "let's cause problems on
mainstream hardware instead".

------
Animats
Of course it's undefined behavior. It's also a famous issue with GCC
optimization.

This is another reason I look forward to the day when C is a legacy language.

~~~
spoiler
As it stands now, C might outlive most, if not everyone in this thread. :-)

Although, I don't mind C. I work in it every day and I've comfortable with it.
A colleague of mine (who works primarily in Python) teased me it's because I
have developed a Stockholm syndrome.

------
dmethvin
The author may be correct, which is of course the best kind of correct. Yet I
have written code for more than 25 years that depends on the behavior
exemplified in the `offsetof()` macro referenced there. When writing low-level
code it's really handy to know the offset of a member in a struct.

~~~
madmoose
Your compiler probably has a built-in method to achieve the equivalent of
offsetof without invoking undefined behavior.

GCC has __builtin_offsetof:
[https://gcc.gnu.org/onlinedocs/gcc/Offsetof.html](https://gcc.gnu.org/onlinedocs/gcc/Offsetof.html)

~~~
masklinn
offsetof is part of C89 stddef.h, why would you call the underlying GCC-
specific __builtin_offsetof? Just call offsetof that's what it's here for,
__builtin_offsetof is the GCC implementation detail.

------
shultays
I don't get it, there is no actual dereferencing going on here, It is just
basic addition arithmetic between two values, one of them happens to be 0. Why
It is illegal or "The program running well is pure luck"?

~~~
cygx
The C standard does not specify the representation of null pointers, nor does
it allow arithmetics with invalid pointers.

Eg the conversion (void *)0 might result in a value with all bits set and
member access will overflow.

Another example would be hardware that raises a signal when encountering an
invalid value during address calculations.

In practice, the address calculation will work as expected (ie I'm unaware of
machines where it doesn't). However, an aggressively optimizing compiler migh
treat an expression like foo->bar as assert(foo != NULL), which might lead to
unintended consequences in worst-case scenarios.

~~~
masklinn
> However, an aggressively optimizing compiler migh treat an expression like
> foo->bar as assert(foo != NULL)

In fact most modern optimising compilers do exactly that, and the article does
refer to such a behaviour having had security implications in the past.

------
tempodox
This is to be taken with a grain of salt.

    
    
      sizeof(&P->m_foo)
    

does NOT return the size of `m_foo` but the size of a pointer to an `m_foo` on
the target architecture.

------
jwatte
The post hinges on the faulty assumption that calculating the address of an
lvalue "evaluates" that lvalue. This is not so.

Any compiler that generates anything other than an add instruction (or lea or
similar) is broken, for rather unsubtle reasons.

Or to put it another way: dereferencing a null pointer may be undefined, but
arithmetic on any pointer (including null) is well defined, and the code in
question is arithmetic, not dereference.

And, yet another broken blog post will make the rounds on the internet every
three years and confuse even more people. It's sad.

~~~
pcwalton
The post cites the spec. The C standard clearly seems to state that & must
reference "an object", and NULL does not reference an object. A "pointer to
any object" is distinguished from a null pointer.

Do you have links stating otherwise?

~~~
Gibbon1
An of course plain C does not have 'objects' and a _lot_ of architectures have
stuff mapped at address 0x0

~~~
nullc
You may be mistaking C for a fancy macro assembler.

It's not-- It's a high level language, even though a fairly low level looking
one. The obligation of the language is to behave as the spec says, and there
may not be a close mapping between your program and what runs on the machine,
though there often is.

An "Object" is a concept defined and used extensively in the language
specification; "a region of data storage in the execution environment,".

------
nraynaud
When you have a bunch of experts arguing on the interpretation of the language
spec to know if a construction in the most fundamental piece of software on
the computer is valid like a herd of clerics around a vulgate deciding if dead
newborns can go to heaven, you know you have a cultural problem.

~~~
jwatte
Are those who argue experts, or are they "experts" ? I know of no expert who
would age with the interpretation of the post.

~~~
nraynaud
they are strawmen, you get to pick

------
ascotan
That's why you should be passing input params by reference and not as a
pointer. The compiler should be be used to check that inputs are valid
references so you don't have to null check everything.

------
amelius
How does Rust deal with this type of situation?

~~~
lmm
By not having null pointers. If you want to have a pointer that might not
point to anything valid, you do that explicitly with an option type. Of course
if you really need to use a possibly-invalid pointer for performance you can
drop into unsafe mode.

(At least, the above is what I'd expect from a language like Rust)

~~~
yonran
But when you do need unsafe code (e.g. for defining a structure that interacts
with a C API), is behavior undefined? I’m just looking at a bit of my own
offsetof calculation and hoping it won’t break once I turn optimizations on.

    
    
        unsafe extern "stdcall" fn callback(arg0: *const IUnknown) {
            let myself: &MyStruct = &*(arg0.offset(-(&((*std::ptr::null::<MyStruct>()).as_event_handler) as isize)) as *const MyStruct);
        }

~~~
dbaupp
Yes, it generally is. Rust has similar (but not the same) undefined behaviours
to C/C++, the compiler just ensures that they can't happen in safe code.

