
Pointers Are More Abstract Than You Might Expect in C - ognyankulev
https://stefansf.de/post/pointers-are-more-abstract-than-you-might-expect/
======
gilgoomesh
This scratches the surface of why I hope C slowly fades away as the default
low-level language. C sounds simple when you look through K&R C. C lets you
feel like you understand the stack, ALU and memory. A pointer is just an
integer and I can manipulate it like an integer.

But the reality is filled with a staggering number of weird special cases that
exist because memory doesn't work like a simple flat address space; or the
compiler needs to optimise field layouts, loops, functions, allocations,
register assignments and local variables; or your CPU doesn't use the lower 4
bits or upper 24 bits when addressing.

C has no way to shield common language constructs from these problems. So
everything in the language is a little bit compromised. Program in C for long
enough and you'll hit a lot of special special cases – usually in the worst
way: runtime misbehavior. Type punning corruption, pointers with equal bit
representations that are not equal, values that change mysteriously between
lines of code, field offsets that are different at debug and release time.

When using fundamental language constructs, we shouldn't need to worry about
these ugly special cases – but C is built around these ideas. The need to
specify memory layouts, direct memory access and other low level access should
be gated by barriers that ensure the language's representation and the machine
representation don't tread on each other's toes.

Rust has a long way to go but is so much more robust at runtime. I think
there's room for other languages to occupy a similar space but they're need to
focus on no-std-lib no-runtime operation (not always the sexiest target).

~~~
nostalgeek
The following is purely my opinion.

Rust might have gone too far the other way. Yes it strives to be a modern
language, with a lot of functional programming features and "OOP" done right
(aka no inheritance, simply method polymorphism). To me Rust is more a C++
replacement than a C replacement.

A replacement for C should try to be as simple as possible while fixing C weak
typing mess with strong typing for instance, including when it comes to
pointers (for instance like in Ada where you can't just take the address of
anything you want, you need to declare it as aliased to begin with). In fact
Ada tries hard to make pointers redundant which is a great thing. This and
range types and bound check arrays( Ada has a lot lot of great ideas, and "C
style" Ada is a treat to use).

So a C replacement should keep the procedural nature of C, with closures
perhaps, but not go further in terms of paradigm. C aficionados often claim
they love C because it's "simple"(it isn't) That's what they mean, they love C
because you can't really do OO with it.

On the other hand, Rust macros are exemplary of what a good macro system
should be.

~~~
TheArcane
> OOP" done right (aka no inheritance, simply method polymorphism).

Why do you consider inheritance a bad feature of OOP?

~~~
jimmy1
On a very high level, in the vast majority of cases, you bring along lot of
baggage, both in code and mental, that you don't need, and compared to some
alternatives, such as composition, it is simply far less flexible. Inheritance
fits a much smaller subset of problems than people think (one of them is GUI
design, a problem inheritance fits with nicely). But like everything in this
industry, it tries to get shoe-horned in to solve every problem.

~~~
stevenwoo
It's surprising when I go into interviews and the people interviewing me do
not have enough experience working with object oriented code (in the languages
I know) to have come to realize this - it's one of those things it seems to
have to be relearned over and over.

------
tzs
Note that if the two pointers are passed to a function, and the comparison is
done in the function, the results are different:

    
    
      #include <stdio.h>
    
      void pcmp(int *p, int *q)
      {
          printf("%p %p %d\n", (void *)p, (void *)q, p == q);
      }
    
      int main(void) {
        int a, b;
        int *p = &a;
        int *q = &b + 1;
        printf("%p %p %d\n", (void *)p, (void *)q, p == q);
        pcmp(p, q);
        return 0;
      }
    

That is giving me:

    
    
      0x7ffebac1483c 0x7ffebac1483c 0
      0x7ffebac1483c 0x7ffebac1483c 1
    

That's compiled with '-std=c11 -O1' as in the article. The result is the same
of pcmp is moved into a separate file so that when compiling it the compiler
has no knowledge of the origins of the two pointers.

I don't like this at all. It bugs me that I can get different results
comparing two pointers depending on where I happen to do the comparison.

~~~
JoeAltmaier
Yes, I agree. It makes far more sense to make guarantees in a language, that
two values are compared using a metric that is invariant. Especially in a
language like C, where we expect it to be a very thin abstraction over the
machine.

------
bluecalm
I like the behavior of the compiler here. There is no guarantee that a and b
are next to each other in memory. That's why the comparison fails, the
alternative makes is runtime/compiler/optimization level dependent which would
be a total mess.

As usual with those C bashing articles you won't run into trouble if you don't
try very hard to write contrived code. I mean, the moment you see:

    
    
         int *q = &b + 1;
    

on your screen alarm bells should go off. Doing pointer arithmetic on
something that is not an array is asking for trouble. If the standard should
be amended in any way it should be undefined behavior right away you do
pointer arithmetic on non-array objects.

~~~
tzs
> I like the behavior of the compiler here. There is no guarantee that a and b
> are next to each other in memory. That's why the comparison fails, the
> alternative makes is runtime/compiler/optimization level dependent which
> would be a total mess

Yes, there is no guarantee that they are next to each other, but in this case
they _happen_ _to_ be next to each other, and according to the spec as quoted
in the article, two pointers are equal if:

> [...] one is a pointer to one past the end of one array object and the other
> is a pointer to the start of a different array object that happens to
> immediately follow the first array object in the address space

Note that it says "happens to" immediately follow, not "is guaranteed to"
immediately follow.

This are pointers to ints, not arrays of ints, but that should not matter
because as quoted in the article:

> For the purposes of these operators, a pointer to an object that is not an
> element of an array behaves the same as a pointer to the first element of an
> array of length one with the type of the object as its element type.

I don't see any way to read these as not requiring the pointers to compare as
equal if the compiler happens to put a and b adjacent in memory in the right
order and with no padding between them, other than resorting to something
ridiculous like claiming that "follow" or "immediately" follow are not defined
in the spec and so one object occupying that very next available address after
another does not necessarily "immediately follow". This seems to be what the
gcc developers went with to declare this is not a bug.

Also, note that if you move the "int a, b" to outside main, so a and b are on
the heap instead of on the stack, then gcc does find that the pointers are
equal.

~~~
clarry
> > For the purposes of these operators, a pointer to an object that is not an
> element of an array behaves the same as a pointer to the first element of an
> array of length one with the type of the object as its element type.

> I don't see any way to read these as not requiring the pointers to compare
> as equal if the compiler happens to put a and b adjacent in memory in the
> right order and with no padding between them, other than resorting to
> something ridiculous like claiming that "follow" or "immediately" follow are
> not defined in the spec and so one object occupying that very next available
> address after another does not necessarily "immediately follow". This seems
> to be what the gcc developers went with to declare this is not a bug.

"These operators" in the spec refer to the (additive) pointer arithmetic
operators, not the equality operator. So incrementing a pointer to int behaves
as if the pointee were in an array of length one, but that does not mean any
subsequent equality operator should consider the objects as arrays.

It's totally fine for the equality comparison to return false when the objects
in question are actually not arrays.

~~~
tzs
Then let's make them real arrays:

    
    
      #include <stdio.h>
    
      #define N 8
    
      int main(void) {
        int a[N], b[N];
        int *p = &a[0];
        int *q = &b[N];
        printf("%p %p %d\n", (void *)p, (void *)q, p == q);
        return 0;
      }
    

Results:

    
    
      $ gcc -std=c11 -O1 ar.c; ./a.out
      0x7ffd32d048c0 0x7ffd32d048c0 0
      $ gcc -std=c11 ar.c; ./a.out
      0x7ffe3f8ccd60 0x7ffe3f8ccd60 1

~~~
smolsky
BTW, things work correctly when the objects live inside a real array:

    
    
      #include <stdio.h>
    
      int main(void) {
        int arr[16];
        int *p = &arr[0] + 1;
        int *q = &arr[1];
        printf("pointers: %p %p %d\n", p, q, p == q);
        return 0;
      }
    

The result is perfectly correct and predictable, as the Standard guarantees
pointer comparisons in this case:

    
    
      pointers: 0x71b35ac790d4 0x71b35ac790d4 1
    

The original code in the article relies on UB, I think, so all bets are off.

------
foxhill
the comparison at the start is nonsense - there is no specification for the
ordering or location of stack variables. by taking the address of these
variables, you could see that they actually are the same value, and so
intuitively you’d think they might be the same, but a different compiler might
put them in different locations. or they may be elided entirely through
optimisation. it’s far safer to fail the equality test in this case - this is
what the model specifies.

this is not even the first example of this counter-initiative behaviour.
imagine two floating point values with exactly the same bit-representation. it
is possible, without any trickery for them to fail an equality check - i.e,
they are both NaN.

this is what IEE754 demands of a compliant floating point implementation. and
indeed, it’s a sane choice when you understand why it was made.

similarly, it’s perfectly reasonable for this example to fail.

~~~
jameshart
Surely the point of writing an if-block testing the equality of the pointers
is precisely because the code _isn’t_ assuming that they are the same - but
maybe it has something it wants to do if it finds itself running somewhere
where they are.

Writing code which assumed those pointers were equivalent would be bad, but
this code doesn’t do that. Instead, it looks like the compiler assumes that
the pointers won’t be the same and refuses to let you entertain the
possibility at all.

~~~
foxhill
> Instead, it looks like the compiler assumes that the pointers won’t be the
> same and refuses to let you entertain the possibility at all.

and indeed it should - it’s sane to use the address of a stack variable for
the purposes of writing something into it, but assuming that it forms some
sort of array in memory is sure to cause pain if you rely on that assumption.

------
mpweiher
clang is a bit more sane:

    
    
       > cc pointereq.c 
       > ./a.out 
       0x7ffee83163b8 0x7ffee83163b8 1
       > cc -O pointereq.c 
       > ./a.out 
       0x7ffeeeb8b3b8 0x7ffeeeb8b3c0 0
    

So without optimization, the pointers are the same and compare as equal. With
optimization, the pointers compare as not equal. At first that seemed
horrible, until I saw the pointers actually are not the same. Since I don't
recall any guarantees about stack layout, that seems perfectly fine.

    
    
       > cat pointereq.c 
       
       #include <stdio.h>
       
       int main(void) {
         int a, b;
         int *p = &a;
         int *q = &b + 1;
         printf("%p %p %d\n", (void *)p, (void *)q, p == q);
         return 0;
       }

------
tzahola
There's nothing surprising in the first example. Comparing the addresses of
stack variables is undefined behaviour.

The second one is more interesting:

    
    
        extern int _start[];
        extern int _end[];
        
        void foo(void) {
            for (int *i = _start; i != _end; ++i) { /* ... */ }
        }
    

GCC optimized "i != _end" into "true". The kernel guys fixed this by turning
"_start" and "_end" into "extern int*". I always thought [] was just syntactic
sugar over a regular pointer, but seems like I was wrong.

~~~
JdeBP
Actually, the headline article tells us that it optimized it to true.

That affected some of my code from years ago, too; and another fix is to keep
_start and instead of having another array called _end have an external size_t
giving the number of elements of start and loop while i!=_start+count .

In fairness to the compiler people there shouldn't really be anything
surprising in the second example, either. There are x86 memory models (compact
and large) where that loop never terminates in the general case, because the
two arrays are not in the same data segment and no amount of incrementing a
(far but not huge) pointer will get from one to the other. So it's not the
case that this is a _new_ pitfall. It was a pitfall decades ago.

People just thought that it didn't exist with modern tiny (a.k.a. flat) memory
models, because the old explanation wasn't applicable. It does, but with a
different explanation.

~~~
tzahola
Yup, I've mistyped "true" as "false".

If you set "_start" and "_end" to addresses of different objects, then you're
invoking undefined behaviour and all bets are off. But this should be
perfectly valid IMO:

    
    
        extern char* _start;
        extern char* _end;
    
        void foo() {
            for (char* c = start; c != end; c++) ...
        }
    
        int main(int, char**) {
            char buf[128];
            _start = &buf[0];
            _end = &buf[128];
            foo();
        }
    

Right?

~~~
JdeBP
The thing there is indeed not the validity, but the introduction of extra
memory references and variables.

The code with the arrays compiles to immediate value loads with load-time
fixups to the code; and if this is linking to an assembly language module the
necessary assembly language is just some labels against the data.

The code with the pointers compiles to loads from data memory, with (in the
more usual paradigm where the start and end are bracketing some vector of
initialized data in the program image) load-time fixups to the initial values
of the variables; and the assembly language has to be different, some
variables with these labels whose initial values are the addresses of some
other labels against the actual data.

Notice that the Linux Kernel developer in 2016 did not immediately appreciate
the need to modify the concomitant assembly language and linker script stuff
to match the change in the C language code.

The approach that I mentioned keeps the immediate value for the start, and
uses a load from initialized data memory for the count. Other approaches are
possible, but they need more work in the build process. For example: By
generating the C/C++ language declaration and definition from a script or
suchlike, one could calculate and declare the number of elements in the array,
making it possible to use sizeof.

------
nuriaion
Basically you have no guarantee where the pointer q is pointing to. Some
compiler/static code analyzer will yell at you with this code.

~~~
jerrre
isn't it guaranteed that is points sizeof(int) bytes higher than the address
of b?

Whether something useful is behind that address is another question

~~~
JdeBP
There's a guarantee relative to the address of b. But there's no guarantee
about the relative addresses of a and b _themselves_ , i.e. where they are
placed in automatic storage. So even setting aside the idea of optimizing the
test away at compile time, there's no guarantee that the comparison result
will be a specific value. a could be above or below b, and they are not
necessarily adjacent objects.

~~~
ChrisSD
From the C11 spec quoted in the article

> If both the pointer operand and the result point to elements of the same
> array object, or one past the last element of the array object, the
> evaluation shall not produce an overflow; otherwise, the behavior is
> undefined.

So although most implementations might produce an address that's relative to
b, that's not actually guaranteed by the spec unless it's `&b + 1`.

~~~
JdeBP
... _which it is_ in the program under discussion. Again, the address which
has no guarantee relative to the address of b or the value of q is _the
address of a_.

------
crehn
_Two pointers compare equal if and only if both are null pointers, both are
pointers to the same object (including a pointer to an object and a subobject
at its beginning) or function, both are pointers to one past the last element
of the same array object, or one is a pointer to one past the end of one array
object and the other is a pointer to the start of a different array object
that happens to immediately follow the first array object in the address
space._

Can someone explain to me the rationale behind this? Why not just "two
pointers compare equal if they point to the same address"?

~~~
JdeBP
Because (a) pointers do not point to addresses, and (b) now you have to define
the concept of addresses being equal in the language standard and haven't
actually reached the goal. (-:

~~~
Sharlin
Indeed; pointers intentionally abstract away memory addresses and addressing,
which, as the article states, can be much nastier business than the benign
flat address space of i386 protected mode.

------
joosteto
When I compile the example, i get:

    
    
      0x7ffd0b57ebd0 0x7ffd0b57ebd8 0
    

OK, so gcc reordered a & b; I'll fix this by chaning the initialisation of p
and q to:

    
    
      int *p = &a + 1;
      int *q = &b;
    

But when I now run the example, I get:

    
    
      gcc -o c c.c && ./c
      0x7ffe55eb0aa4 0x7ffe55eb0aa4 1
    
      gcc -O -o c c.c && ./c
      0x7ffcfeffd914 0x7ffcfeffd914 0
    

So, p==q only evaluates to 1 if optimisation is enabled.

    
    
      $ gcc --version
      gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0

~~~
psyklic
It shows the exact options the author used if you click on GCC:

Version: 8.1.1 Options: -std=c11 -O1 Target: x86_64-redhat-linux

------
oconnor663
> If we step back from the standard and ask our self does it make sense to
> compare two pointers which are derived from two completely unrelated
> objects? The answer is probably always no.

The one big counterexample I can think of is the difference between memcpy and
memmove. The latter is supposed to be able to do arithmetic on memory regions,
to see if they overlap. Is this article saying that the standard C
implementation of memmove is relying on unspecified behavior?

~~~
slrz
Memmove is frequently cited as an example of a standard library function that
can't be implemented using only standard C. It's incorrect, though: the way to
do memmove using standard C is to use malloc to allocate a temporary buffer.

But the whole question is not terribly relevant. When memmove is provided as
part of the C implementation it can rely on non-portable platform behaviour
just fine. There's no rule that you have to implement libc using only standard
C facilities.

~~~
pjmlp
And how do you implement malloc() in standard C?

------
Sharlin
> _...or one is a pointer to one past the end of one array object and the
> other is a pointer to the start of a different array object that happens to
> immediately follow the first array object in the address space._

I was not aware of this special case. What's the rationale? Is there even a
way in standard C to guarantee that two array objects are laid out in memory
like that, with no padding?

~~~
saagarjha
A two-dimensional array?

~~~
Sharlin
Yeah, good point.

------
bsder
Ummm, when I run this on gcc 7.3.0 on OS X I actually get:

0x7fff5dbd89fc 0x7fff5dbd89fc 1

Which kind of shoots the whole article in the foot ...

~~~
zubyak
Did you compile it using '-std=c11 -O1' ?

~~~
bsder
No, but that just makes me assume that this is an optimization bug. Especially
since -O4 actually coughs up _different_ pointers:

    
    
      gcc-7 -std=c11 -O4 -o test test.c
      $ ./test
      0x7fff5b5f5a08 0x7fff5b5f5a10 0
    

The following code works as expected even with your flags and the pointers are
clearly not related to one another:

    
    
      #include <stdio.h>
      #include <stdint.h>
      
      int main(void) {
          uint32_t *p = ((uint32_t *)0x7fffffffffff3eac)+1;
          uint32_t *q = (uint32_t *)0x7fffffffffff3eb0;
          printf("%p %p %d\n", (void *)p, (void *)q, p == q);
          return 0;
      }
    

So, I would either chalk this up to either 1) "compiler optimization bug"
unless a gcc maintainer _explicitly_ gave me a reason to believe to the
contrary or 2) getting your underwear eaten by weasels because you optimized
after invoking undefined behavior (attempting to probe stack layouts qualifies
as undefined behavior in a big way).

My _assumption_ would be that the compiler had to do something screwball that
it got wrong because you actually asked for a pointer to a stack object that
it had optimized to always be in registers.

------
mabynogy
Not related to the content but I find the colored links in the nav a simple
and very good idea.

------
andyjohnson0
Just a datapoint: VS2017 on Win10 x64 gives me 00AFF89C 00AFF894 0, which is
what I naively expect.

~~~
JdeBP
Actually, you _should not_ expect that, because you don't have a guarantee of
ordering of the addresses of a and b on the stack.

Now try with /O2 . (-:

~~~
andyjohnson0
Maybe I'm misunderstanding you, but it's precisely because I have no knowledge
of the stack frame's layout that I didn't expect the pointers to be equal.

With /O2 optimisation I get:

00B1F99C 00B1F9A4 0

~~~
JdeBP
That's where you are going wrong. You have no guarantee that the compiler
places a and b next to each other, let alone in a defined order with a at a
higher address than b such that one integer width beyond the end of b you find
a.

Expecting _one thing_ is wrong. You are expecting the _one thing_ that the
comparison always returns false. You should _not expect anything_. The
comparison result could be true or false.

    
    
        H:\>cl /O2 pointereq.c
        Microsoft (R) C/C++ Optimizing Compiler Version 19.14.26431 for x86
        Copyright (C) Microsoft Corporation.  All rights reserved.
        
        pointereq.c
        Microsoft (R) Incremental Linker Version 14.14.26431.0
        Copyright (C) Microsoft Corporation.  All rights reserved.
        
        /out:pointereq.exe
        pointereq.obj
        
        H:\>.\pointereq
        00CFFA20 00CFFA20 1
        
        H:\>

~~~
mpweiher
You are confusing "did not expect them to be equal", which is what the OP
wrote, with "expected them to be not equal", which is what you are railing
against.

~~~
JdeBP
Ahem!

"which is what I naively expect."

~~~
andyjohnson0
My expectation was that they wouldn't be equal.

~~~
JdeBP
I already knew. (-: Tell mpweiher.

------
_RPM
if you're adding 1 to a pointer, why would you expect it to be equal? I would
not expect it to be equal.

~~~
armitron
He's adding "1" to the pointer that points to the variable "b" which in this
case is allocated immediately below "a" in the stack. Thus adding one will
make the two pointers equal.

~~~
saagarjha
The layout of things on the stack is completely implementation-defined.

~~~
sannee
It has been a while since I read the standard, but this seems well into the
realm of undefined behavior, not implementation defined.

~~~
saagarjha
From my understanding, how the stack is laid out is implementation-defined;
attempting to actually observe this ordering in your program is undefined
behavior.

