
Why is my x64 process getting heap address above 4GB on Windows 8? - ScottWRobinson
http://blogs.msdn.com/b/oldnewthing/archive/2015/07/09/10624293.aspx
======
phs2501
This bit me in a library of mine. Some third-party code I used was using long
as memory offsets which failed on Windows 8. Unfortunately I was developing on
Windows 7, so it didn't get noticed until the customer stumbled over it. :(
(Fortunately said third-party code had an update that switched to ptrdiff_t in
all the right places, so it was an easy fix.)

I'm a Unix person, so the notion of long being less than word-length seems
pretty silly to me, but I understand why they did it, sort of.

~~~
sjolsen
>I'm a Unix person, so the notion of long being less than word-length seems
pretty silly to me

It's silly for Windows to do something a certain way just because that's not
how Unix does it?

What's silly is assuming that long and unsigned long are more than 32 bits
wide, since neither C nor C++ has ever guaranteed anything more.

~~~
michaelt
I don't write much C, so perhaps someone can educate me.

What are the advantages of having data types that behave differently on
different platforms, like C does? It seems to make code less portable and I'm
having difficulty seeing what the benefits are.

I can appreciate that, on an 8-bit processor, you'd want access to an 8-bit
data type for speed - but wouldn't that be better accomplished by explicitly
choosing an 8 bit data type?

~~~
brudgers
Undefined behavior in C standards is often not a result of technical decisions
but of practical concerns when developing a consensus during the standards
process. The people on the standards committee historically included compiler
vendors and users of diverse operating systems. When the process started in
1983, there were 8, 16, and 32 bit systems [and things like Burroughs 48 bit
word systems].

The practical decision to leave the sizes undefined was made so that nobody
was burdened unfairly. It's choices like this - not biasing stakeholders
toward forgoing a standard for clear business reasons - that makes the
adoption of standards likely. Subsequent C standards have adopted the same
undefined behavior for the same reasons [there are still 8 and 16 bit systems
developed with C] and to facilitate backward compatibility of the new
standards.

~~~
tbirdz
Bit of a nitpick, but technically the sizes of int, long, etc are
implementation defined, not undefined. Undefined behavior has different
implications in C.

~~~
brudgers
I agree. Undefined would mean declaring x a long integer could launch the
missiles. Thanks. It's my xkcd386.

------
copx
Pointer packing is popular with people wanting to write efficient interpreters
for programming languages. The memory use and performance gains are massive
and it was perfectly safe during the 32-bit era.

LuaJIT does it for example. Interestingly enough some programmers do this
without even understanding how memory allocation works. For example the author
of the newish wren language:
[https://github.com/munificent/wren](https://github.com/munificent/wren)

Somehow most programmers - including almost everyone who frequents
reddit.com/r/programming - believe that "virtual address space" means that the
addresses returned by malloc() will restart from (somewhere close to) 0x0 for
every new process i.e. that it is safe to truncate pointers if you never
intent to allocate more than ~2GB per process anyway.

~~~
beagle3
LuaJIT does something weirder. It does double packing - using 48 bits (IIRC)
of value inside a "NaN" double encoding. It is going to be a while before this
becomes a problem, even on 64-bit systems - you need more than 128TB
addressable space before this becomes a problem (and I suspect Pall will
switch to a 51-bit (1024TB=1PB) pointer when that future is on the horizon).

~~~
copx
No that is wrong. Currently the x64 port of LuaJIT is limited to about 1GB [1]
in the best case and can fail to allocate any Lua objects at all if the lower
region of the address space is already completely consumed.

[1] [http://lua-users.org/lists/lua-l/2010-11/msg00241.html](http://lua-
users.org/lists/lua-l/2010-11/msg00241.html)

~~~
beagle3
Thanks. A slightly later discussion I found is here[0]. However, this is not a
limit imposed by the tag structure used by LuaJIT, but rather the Lua/LuaJIT
GC (from a quick reading of the mailing list discussion - I might be wrong
here). Apparently, there's a GC in the works that is supposed to resolve this,
but it isn't there yet.

[0] [http://lua-users.org/lists/lua-l/2012-04/msg00729.html](http://lua-
users.org/lists/lua-l/2012-04/msg00729.html)

------
the8472
> Why are they truncating 64-bit pointers to 32-bit values?

Maybe to get more compact data structures? But in that case they shouldn't
rely on that and try detecting that behavior at runtime.

E.g. the hotspot jvm tries to map memory in the low end of address space to
compress pointers but falls back to full pointers when that fails.

~~~
cesarb
As mentioned on the blog post's comments, it's often caused either by storing
a pointer in a DWORD variable (very common on 32-bit Windows), or by storing a
pointer in a "long" variable (on most common 32-bit and 64-bit platforms, a
"long" is at least as wide as a pointer; the main exception is 64-bit
Windows).

The correct variable type to store a pointer would be intptr_t/uintptr_t for
portable software (and DWORD_PTR for Windows-only software), but the "intptr"
types didn't exist until the 1999 C standard (and, as far as I could find, the
default Windows compiler didn't add the corresponding header file until around
2010). Software older than that would often assume that a "long" (or a DWORD,
which IIRC was just another name for "long") was big enough to store a
pointer, or perhaps tried to use the C99 types and used "long" if the header
wasn't found.

~~~
TillE
> storing a pointer in a DWORD variable (very common on 32-bit Windows)

Is it? I mean, if you're lucky it won't cause a compiler warning on x86, but
the sane thing to do is to use the types which the API docs specify. Which
would usually be things like LPVOID or HANDLE.

That's basically how I managed to port a service and several drivers to 64-bit
Windows many years ago with almost no changes.

~~~
acqq
At the time the code was written the warnings you get now didn't exist. Then
the 32-bit code worked, so nobody needed to change the it. Now with windows 8
they see it for the first time that it doesn't work.

I know as I also worked on the huge codebase where the oldest pieces of the
code were written around 1995.

------
daniel02216
This is one reason why OS X enforces a hard page-zero on 64-bit programs - it
is a hard error to map or allocate anything in the lowest 4 GB of memory, so
programmers will immediately discover pointer truncation bugs when porting
32-bit code to 64-bit.

~~~
cokernel_hacker
What if you have a load command in your mach-o which asks for an address below
the 32-bit boundary? Will loading the image fail?

------
fsloth
Long was not 64 bits? This is why C standard has stdint.h which typedefs
numeric types of specific length (for 64 bit integers use int64_t, etc). It
seems Visual Studio also has it nowadays [https://msdn.microsoft.com/en-
us/library/hh874765.aspx](https://msdn.microsoft.com/en-
us/library/hh874765.aspx).

