
Ask HN: Is big-endian dead? - api
Are there any big-endian chips still in production outside the embedded or specialty market?<p>Even MIPS and PPC now support a little-endian mode and are usually deployed that way. ARM is nearly always LE or deployed that way, and RISC-V is LE.<p>Edit: bonus factoid:<p>Little-endian is slightly more confusing for humans but may be objectively better for machines. On a little-endian machine integer size casts are free -- e.g. casting a uint64_t to uint32_t just means reading the first 4 bytes of it. On big-endian machines integer size casts require pointer math.
======
rdtsc
As long as we use the traditional network protocols and socket API it's not
"dead" dead. The other name for big-endian is "network order", after all.

As way to serialize data (wire / disk format) it's becoming more common.
FlatBuffers and Cap'n'Proto are the popular ones. They reduce (completely
eliminate?) byte shuffling when de-serializing.

In one instance I was reading a spec for an industry-specific protocol. At
first I was looking at it and thinking "What the... they are padding stuff in
a strange way, and using little-endian for the data". Then it suddenly dawned
on me, that they've designed the spec to be whatever GCC on x86-64 Linux
machine would do to layout C the structures in memory.

So someone very lazy could just define a struct and cast into it as data comes
from the wire. Someone one a big endian machine, would have to do a lot more
legwork to get the thing working. But given that there aren't many of those
around, it was deemed an acceptable tradeoff.

~~~
scott00
Using the x86-64 C struct layout as the serialization format is increasingly
common in financial trading protocols. Skipping all the bit twiddling has a
huge performance impact at high message rates. I once compared two feeds
covering the same basic data, where the major difference was one had a
complicated serialization and one was C struct layout. The former needed two
16 core servers just to process I/O and deserialize the messages; the latter
used 1 core on a single server.

~~~
usefulcat
"The former needed two 16 core servers just to process I/O and deserialize the
messages; the latter used 1 core on a single server."

I'm genuinely curious; what was the 'former' protocol? Was it encoded using
FAST or zlib or something?

~~~
twic
I would guess FIX/FAST. For a concrete example, have a read of the
specifications for Eurex's market data protocols:

[http://www.eurexchange.com/exchange-
en/technology/t7/system-...](http://www.eurexchange.com/exchange-
en/technology/t7/system-
documentation/release6/Release-6.0/3088824?frag=3088828)

Specifically, compare the Enhanced Market Data Interface protocol (described
in "T7 Market and Reference Data Interfaces") and the Enhanced Order Book
Interface protocol. EMDI is FAST, a morass of tags, stop bits, presence
bitmaps, and who knows what else (i don't). EOBI is structs.

------
hsivonen
Big endian is dead on the client. WebGL exposes endianness and happened at a
time when almost all systems running browsers are little-endian. This makes
the big-endian provisions of the spec a dead letter. Web devs don't need to
test on big-endian systems and would have a hard time finding big-endian
systems to test with even if they wanted to. It's safe to assume, therefore,
that there's a mass of Web content that only works if the browser exposes
little-endian behavior to JS. It's not performance-wise competitive for a
WebGL system to present little-endian behavior on big-endian hardware.

Therefore, it won't be feasible to try to re-introduce big endian to systems
that need to be competitive at rendering the Web.

Big endian will stay alive for the time being on home routers and Sparc
servers, and probably for a long time on IBM z systems.

The cost–benefit of software accommodating big-endian systems will look
increasingly bad with legacy enterprise servers imposing a negative
externality on everyone else. (E.g. refusal to consider bitcasts between SIMD
vectors of the same bit width but different number of lanes as portable/safe
operation in language design.)

~~~
classichasclass
You can not only assume it, but I can also say with certainty it exists.
Emscripten hastened this by letting site owners compile little-endian code
into little-endian asm.js. For example, the WhatsApp Web QR code generator
won't run on a BE platform; it's an Emscriptenized blob of code backing it
that assumes the typed array it stores into has LE orientation.

TenFourFox and Leopard Webkit both emulate little-endian typed arrays on BE
Power Macs and transparently byteswap integers. This mostly works for sites
that care and has almost no overhead in JITted code for 16 and 32-bit int, but
Facebook managed to break some of our assumptions with floats (example:
[https://github.com/classilla/tenfourfox/issues/453](https://github.com/classilla/tenfourfox/issues/453)
\-- we don't byteswap floats because it's expensive and probably would break
things expecting native endian on the DOM side). Fortunately, TenFourFox
doesn't support WebGL, so we don't need to worry about that.

IMHO, this is a broken promise about how the Web was supposed to be platform-
independent. I get the feeling the general attitude on this thread is that
such platforms are unimportant, but that doesn't mean it's not a broken
promise.

~~~
userbinator
_IMHO, this is a broken promise about how the Web was supposed to be platform-
independent. I get the feeling the general attitude on this thread is that
such platforms are unimportant, but that doesn 't mean it's not a broken
promise._

I'm pretty sure many other fundamental protocols of the Web don't work so well
on a non-2's-complement, non-8-bit-byte machine either.

~~~
classichasclass
Ignoring whether asm.js is a fundamental protocol or not, the Web wasn't
supposed to depend desperately on the byte orientation of the consumer. Surely
you remember how it was supposed to lift information access above what type of
computer you viewed it on.

WebGL's endian specification was the beginning of the end, after which folks
started saying certain implementational details weren't worth caring about,
and this just finishes it off. And that's not what was being sold in the
beginning.

------
kazinator
The "casting" advantage basically just hides bugs. Because it's often __not
okay __to access just 16 or 8 bits of a 32 bit number; the value is being
truncated and that could be a bug.

However, when the number's range is small enough, such code appears to work.
Finding the bug is delayed until larger values occur, which is not until its
deployed in the field or someone thinks of writing better test cases.

Anyway, if it is okay to truncate, the right thing to do (and ISO C
conforming) is to access the underlying type as that type, and then cast the
result, not to cast the pointer. I.e. rather than:

    
    
        int x = *(int *) ptr64; // breaks on BE; not kosher with ISO C aliasing rules
    

this:

    
    
        int x = (int) *ptr64;  // works on LE and BE
    

We don't even need the casting operator in the second example. So it's less
syntax and clearer code to do the more portable thing.

If we want the bits for that int from some other part of the wider word, we
can use shifting and masking.

~~~
arnsholt
I ran into a bug in that area a while back. Our development environment has a
binding to Winsock32.dll for socket stuff, and as it turns out the bindings
were written so long ago that many of the return values are specced to be
16-bit ints. As a result, our application would work fine almost all of the
time, but fail mysteriously with "socket operation on non-socket" after about
8000 sockets have been opened. As I understand it 8000 is because file
descriptors are a kind of bump-the-pointer scheme internally and pointers are
8 bytes wide on 64-bit platform.

------
ajross
Little endian is also better for multi-word addition/subtraction. Your first
operand will be the low word, which your pointer conveniently already indexes.
This was a classic trick in the days of 8 and 16 bit systems, and assembly
code for modern modular crypto systems still uses it.

Really I think consensus is that the industry focus on big endian systems was
actually a mistake. It prioritized programmer comfort in a regime where it
never really mattered.

~~~
kazinator
OTOH, big-endian is better for comparison; you can tell when one integer is
larger than another by looking at as little as just one bit, from the most
significant end.

~~~
amelius
In the worst case, comparison is still like subtraction.

~~~
kazinator
Even if the whole datum is examined, comparison is more parallelizable; it
doesn't require carry propagation.

------
tlb
I used to use IETF's network order (big-endian) for everything on disk or over
the wire, but I switched to little-endian around 2008. If I ever have to port
to a big-endian machine I'll deal with it then. I haven't regretted it yet.

~~~
api
Big-endian is the canonical form for storage and the wire because back when
the Internet was designed most "pro" machines were big-endian: Sparc, old
MIPS, old PPC, DEC Alpha, etc. All these are dead or dying now.

It's easy enough to deal with BE files and protocols by just swapping bytes.
I'm referring to hardware architectures. Are there still any big-endian chips
out there? Does it still make sense to support big-endian chips in new code as
long as that code is never intended for embedded or very old systems?

~~~
drewg123
Small pedantic correction from a former Alpha kernel hacker: DEC Alpha was Big
Endian only for some rare Cray systems. Everything else was Little Endian.

~~~
macintux
Many, many years ago I worked for Progeny, and we contracted with HP to help
bring Debian to Itanium.

We quickly discovered that a fair bit of Linux software (or at least Debian
packaging of same) only worked properly on 64-bit DEC Alpha because the
architecture was Little Endian; 32-bit pointers would typically "just work(™)"
because the code was grabbing the first 32 bits, and the other half of the
pointer addresses was typically 0x0.

Since we were porting to Big Endian hardware (just now discovered Itanium was
selectable, not sure why it was Big Endian in our case) we were getting a fair
bit of null pointer exceptions.

Anyway, ancient history, but that was my only direct exposure to endian-
madness, so it has stuck with me.

~~~
zlynx
HP wanted to replace PA-RISC and that was big-endian. Intel of course wanted
little-endian. That's why Itanium was selectable.

------
wmf
We had this same thread 997 days ago:
[https://news.ycombinator.com/item?id=9451284](https://news.ycombinator.com/item?id=9451284)

If big endian was effectively dead back then, I guess it's really dead now.

------
tinus_hn
Let’s hope so because one of these is not better than the other but one is
much better than two.

~~~
phkahler
I was annoyed reading the RISC-V docs and seeing that they allow
implementations to go either way. Sometimes it's better to just make a choice
and tell people what it is.

------
jcadam
I was once on a contract that involved porting satellite simulation software
from SPARC/Solaris to x86/RHEL. The software was in Ada95, and it didn't take
much to get it to compile on the new hardware. However, ground-to-vehicle
comms (which our simulator had to handle as it needed to work with the real
ground control software) took place using a packed binary format (wasting as
little space as possible). Data fields frequently crossed byte boundaries (the
first 10 bits are field X, the next 6 are field Y, etc.). Fun times
rearranging and repacking bits.

Of course, dealing with all of the Endian issues had not been planned for and
was not in the project schedule :/

~~~
gbacon
Sounds like VMF. It’s such a pain to debug when you can’t even rely on marker
strings being recognizable in dump output.

One time I had to make little- and big-endian machines talk nicely was for a
message engine that ran on Intel/Linux. All the messages in the ICD were
glorified packed structs that were native on the IBM/AIX side.

I used the ICD wire format as my storage format. I defined containers for all
the primitive types with correct sizes and that swapped values on assignment
and read (except for strings, of course). I added the appropriate pragmata to
pack the underlying structs and locked down their sizes with static_asserts
that would fail at compilation time until everything fit perfectly.

These classes gave me what looked like native structs on the Linux side, e.g.,

    
    
        msg.foo = 3;
    

or

    
    
        msg1.date = msg2.date + 1;
    

and the byte swapping happened transparently all around, both on the way in
and on the way out. Yes, if latency had been a concern, this design may have
been problematic. The Linux box faked messages on behalf of three other
systems so we could command the AIX software into different modes with complex
prerequisites. Changing a few fields per cycle, I could then dump the struct
onto the wire.

A cool aspect to the story was it all worked correctly the first time out of
the box even though I did not have access to the AIX box during
implementation. When it came time to integrate with the AIX side, all that was
left to do was some low-level socket stuff to hook up to the message flow.

I’d moved to a different project full-time and worked this message engine
during evenings and weekends. We got the code unpacked, so I run make check
like I had been so frequently.

“Wow, you had time to write unit tests and everything?” asked one of the guys
from behind my shoulder.

A moment of Zen hit.

“I didn’t have time _not_ to.”

------
faragon
Little endian is genius. Not only because of size casts being "free" (because
LE is compatible among "word" sizes -lower bits are at a known location-,
while with BE you need to know the word size in order to locate the lower bits
of the word), but also because of efficient bit I/O (using CPU registers as
accumulators, and flushing with a word-size store), so variable size data can
be encoded efficiently.

------
Xeoncross
Dealing with PCM audio streams (like WAV files) you sometimes find both. After
all, there are thousands of codecs + media containers + multiple versions.

[https://en.wikipedia.org/wiki/FFmpeg#Supported_formats](https://en.wikipedia.org/wiki/FFmpeg#Supported_formats)

> When more than one byte is used to represent a PCM sample, the byte order
> (big endian vs. little endian) must be known. Due to the widespread use of
> little-endian Intel CPUs, little-endian PCM tends to be the most common byte
> orientation.

[https://wiki.multimedia.cx/index.php/PCM](https://wiki.multimedia.cx/index.php/PCM)

------
ComputerGuru
I’m presuming from your wording that you’re already aware, but for the benefit
of others, ARM is bi-endian and can be used in either of LE or BE modes.
However, most “in the field” configurations deploying real-world operating
systems use it in LE mode as that makes porting everything so much easier.

------
gumby
> Little-endian is slightly more confusing for humans...

I think you meant to write “Little-endian is slightly more confusing for
humans _who use left-to-right languages_ ” as all the R-L languages also put
the least significant digit on the right.

~~~
samat
Not true. In arabic languages you write text R-L, but numbers are L-R.

~~~
BrandoElFollito
Do you mean that you read a text from the right to the left, and when you meet
a number you jump to is leftmost digit, then read it LR, then jump agzin to
its left (where there is a space) and resume reading text?

~~~
simias
If you think about it in left-to-right languages reading numbers requires
backtracking because you have to see how many digits there are to know if
you're reading units, thousands, millions, billions...

Take 12,345,567 for instance, if you want to read it aloud you first have to
go all the way to the end to figure out that there are three groups of
numbers, so the first one is million, then thousands, then units. If you were
only given a truncated version of the number it's impossible to read it aloud.
So in a way the arabic represents number actually makes some sense and it
could be argue that _we_ write them backwards.

Furthermore IIRC from a discussion with an arabic friend (from the middle
east, not sure if it's different in NA) the most formal way of reading numbers
aloud is actually backwards (or little endian I guess) compared to the way we
do in english. So _if I understood correctly_ for 1,234 you'd say "four and
thirty and two hundred and a thousand" or something similar. Don't quote me on
that though because I know almost nothing about the arabic language and this
is from a discussion I've had many years ago.

------
wilsonnb
I believe the chips that IBM uses in their current Z series mainframes are
still big-endian.

------
monocasa
Eh, haswell added the movbe instructions. And I wouldn't be surprised if
someone in the RISCV world added big endian instructions (they're nice for
network processing).

And nearly every powerpc I've seen is big endian, FWIW.

~~~
acallan
Actually, Silverthorne (the first Atom CPU) added MOVBE.

~~~
ComputerGuru
It seems that Atom has become - ironically - a staging ground for new features
that make their way much later to the desktop/mobile x86_64 architecture.
Here's another: [https://neosmart.net/blog/2017/will-amds-ryzen-finally-
bring...](https://neosmart.net/blog/2017/will-amds-ryzen-finally-bring-sha-
extensions-to-intels-cpus/)

~~~
monocasa
Those little cores get a lot more benefit out of special case instructions.

------
comandillos
Im working for an Space Agency and we'r using SPARC for all our sats CPUs.
Basically, Big-Endian.

~~~
planteen
I worked on a code base using the LEON (SPARC, big-endian) and also ARM in
little-endian mode for space. The code was common for both endians. I followed
the sage advice of Rob Pike on it:
[https://commandcenter.blogspot.com/2012/04/byte-order-
fallac...](https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html)

------
jfries

      Little-endian is slightly more confusing for humans
    

I've heard this before, but the reason is that you view hex data and list
numbers left-to-right as if they were letters. They are not.

0x12345678 stored big-endian, numbering bytes left-to-right:

    
    
      12 34 56 78
    

Looks good, but I think that this is actually more confusing, because when you
number the bytes and bits you will see that the bytes are written left-to-
right, while at the same time the bits are written right-to-left.

The solution is to show dumps with bytes numbered right-to-left. This is
coherent with how we number bits, and also how we relatively position digits
in any other number.

0x12345678 in little-endian, but written right-to-left:

    
    
      12 34 56 78
    

Now the numbering of the bytes is consistent with the numbering of bits. You
can easy see that bits 0-7 of the bits interpreted as a 32-bit word belong in
byte 0, while bits 16-23 belong in byte 2.

~~~
phs2501
At least in Freescale's PowerPC documentation, it's convention to number the
bits left-to-right in big-endian. So the most-significant bit is bit 0, which
matches up with the most-significant big-endian byte being 0. See, for a
random example, page 1101 of [https://www.nxp.com/docs/en/reference-
manual/MPC8379ERM.pdf](https://www.nxp.com/docs/en/reference-
manual/MPC8379ERM.pdf).

Personally I prefer the little-endian representation.

~~~
kazinator
The documentation might have that numbering, but that makes no difference in
programming on the PowerPC. When we take the value 1 and shift left by 1 bit,
we get 2.

That the documentation thinks this is bit 7 going to bit 6 is immaterial.

Calling the MSB "bit 1" is a tip of the hat to serial communications. In
serial communication and networking, it is predominant to transmit the MSB
first.

If the documentation is about a wire format, then using that numbering is
correct down to the data link layer and (modulo framing considerations and
such), physical.

~~~
phs2501
Oh, I'm aware it makes no difference internally; I picked that documentation
because I am somewhat intimately familiar with it as I worked on a product
with that CPU and did a lot of driver work. My only point was to the poster I
was replying to, who wrote " _because when you number the bytes and bits you
will see that the bytes are written left-to-right, while at the same time the
bits are written right-to-left_ ". That assumes that bits are universally
numbered starting with 0 at the LSB, which isn't true.

------
mrpippy
SPARC still exists, and is big-endian

~~~
DannyBee
SPARC 64 is bi-endian. It has big endian instruction format, but you can
choose endianness both at the load-store level or at the memory page level.

------
echlebek
Maybe in processors, but not in general.

Why? Because you can use lexicographic sorting with big-endian bytes.

------
IshKebab
Funny I just asked a question about this on SO when I discovered that CBOR - a
very recent protocol - uses network byte order (big endian). A confounding
decision given that no machines today use it, and it just add extra byte
swapping at both ends.

I can't explain it other than presumable there was some guy on the design
committee who was like "but.... Unix... the 70s... network byte order...
guys!" or something.

~~~
spc476
Having implemented CBOR [1], it's really only an issue for the x86, which
_can_ do unaligned reads. On most other architectures (even little-endian ones
like RISC-V) it's less an issue because you will either have to copy the bytes
to be aligned, or just read byte-by-byte (sorry, octet-by-octet) and shift.

[1] [https://github.com/spc476/CBOR](https://github.com/spc476/CBOR)

------
rozzie
Big-endian is the new 1’s complement.

Like 6-bit bytes and octal, BCD, segments, and other such memories, it’s
simply a relief that we’ve moved on.

------
jondubois
I find the the terms big endian and little endian confusing, feels like they
should be switched around.

So the one that ends big (most significant byte last) should be big endian and
the one that ends little should be little endian. But it's actually the
opposite. It makes it difficult to remember which is which.

~~~
userbinator
Here's an alliterative mnemonic:

    
    
        Little endian puts the
        Least significant part at the
        Lowest address(offset)

------
viraptor
> On a little-endian machine integer size casts are free -- e.g. casting a
> uint64_t to uint32_t just means reading the first 4 bytes of it. On big-
> endian machines integer size casts require pointer math.

Is it really that different? It matters on x86 if the value is in some
register already, since there's no subdivision of the high part, but it
shouldn't matter for memory. Most instructions support math in addressing, so
whether `(char)*x` gets compiled to `[eax]` or `[eax+3]` seems not very
different.

------
grawprog
The 24-bit virtual machine I'm working on is little endian. The 'processor'
reads instructions and addressess in little endian and instructions are stored
in memory in little endian. But when programming it, using assembly, values
and addresses are written with the least significant byte on the furthest
right to make it easier for people to understand. At least so far. I may
change it if it ends up being confusing.

------
angry_octet
While it has mostly gone out of favour, it really isn't due to some cosmic
plan. Intel just won in the microprocessor architecture race, and they
happened to be LE due to their evolution.

I still have PowerPC and MIPS machines that are BE, and which are occasionally
handy for seeing if pointer math is being done wrong.

------
chmike
Here is another point of view on the endianess issue:
[https://commandcenter.blogspot.fr/2012/04/byte-order-
fallacy...](https://commandcenter.blogspot.fr/2012/04/byte-order-fallacy.html)

------
deepnotderp
I think a couple of networking chips still use it.

~~~
AlexandrB
I remember reading somewhere that BE was chosen for IP partly for routing
efficiency. As you go from MSB to LSB in an IP address you narrow in on an
increasingly specific network segment. That's probably not true anymore as
smaller and smaller blocks of IPv4 addresses have been handed out (e.g.
instead of a full class C).

~~~
Const-me
> That's probably not true anymore

Right, and also because modern RAM is essentially a block device. You just
can’t access individual bytes anymore. E.g. on a typical modern PC with a
dual-channel DDR, the block size is 128 bits = 16 bytes.

------
takeda
I read and so far all arguments for little endian I see are because the CPUs
adopted it so it is more efficient to keep data the same way.

What are the other benefits of little endian, because in terms of readability
big endian makes most sense.

I could understand that having things in reverse could be somehow more
efficient, but why do it reverse in groups of 8 bits instead having all bits
reversed?

------
valarauca1
Big Endian is still supported natively on POWER8/9

Intel added movBE (mov big endian) support in BM2 extension.

Networks are still Big Endian.

SPARC chips (and now RISC-V) are BIG Endian by default.

MIPS just became an independent company a few days ago and they have native
Big Endian support.

Edit 1: I was wrong about Power8/9 LE/BE

~~~
wmf
_If you use Linux on POWER you 're forced to use Big Endian mode as the faster
Little Endian mode (according to marketting) is reserved for IBM's operating
systems._

That's not correct. Ubuntu is LE only and it looks like RHEL is also switching
from BE to LE.

"The base RISC-V ISA has a little-endian memory system, but non-standard
variants can provide a big-endian..."

SPARC and MIPS are dead.

~~~
Aloha
MIPS last I looked was quite alive in the embedded space - you're right in
that it is effectively dead everywhere else.

------
dnautics
little endian is only confusing for humans because we very inconsistently use
RTL vs LTR semantics.

------
amelius
Do some bignum libraries perhaps still use big endian, for whatever reason?

------
Bitcoin_McPonzi
Why not both? There was at least one chip that let you select your endian-
ness. (The MIPS R4000).

See Wikipedia on Bi-endian machines:
[https://en.wikipedia.org/wiki/Endianness#Bi-
endian_hardware](https://en.wikipedia.org/wiki/Endianness#Bi-endian_hardware)

------
floogtheunfound
If you work with some industrial machines via serial protocols it is very much
alive.

------
CoC_CRUSHER
Does it matter? on x86 you can bswap if you need to convert.

------
hapless
Yes.

