What you should do instead is write all your code so it is little-endian only, as the only relevant big-endian architecture is s390x, and if someone wants to run your code on s390x, they can afford a support contract.
The vast majority of modern network protocols use little endian byte ordering. Most Linux filesystems use little endian for their on-disk binary representations.
There is absolutely no good reason for networking protocols to be defined to use big endian. It's an antiquated arbitrary idea: just do what makes sense.
Use these functions to avoid ifdef noise: https://man7.org/linux/man-pages/man3/endian.3.html
You should actually use format-swapping loads/stores (i.e deserialization/serialization).
This is because your computer can not compute on values of non-native endianness. As such, the value is logically converted back and forth on every operation. Of course, a competent optimizer can elide these conversions, but such actions fundamentally lack machine sympathy.
The better model is viewing the endianness as a serialization format and converting at the boundaries of your compute engine. This ensures you only need to care about endianness at the boundary and that you have no accidental mixing of formats in your internals; everything has been parsed.
In fact, I'd be surprised if you made a big endian arch and then ran a browser on it if some large number of websites would fail because they used typedarrays and aren't endian aware.
The solution is not to ask every programmer in the universe to write endian aware code. The solution is to standardize on little endian
Network protocols and file formats still need a defined byte order, and the first time your code talks to hardware or reads old data, little-endian assumptions leak all over the place. Ignoring portability buys you a pile of vendor-specific hacks later, because your team will meet those 'irrelevant' platforms in appliances, embedded boxes, or somebody else's DB import path long before a sales rep waves a support contract at you.
Most existing CPUs, have instructions to load and store memory data of various sizes into registers, while reversing the byte order.
So programs that work with big-endian data typically differ from those working with little-endian data just by replacing the load and store instructions.
Therefore you should have types like int16, int32, int64, int16_be, int32_be, int64_be, for little-endian integers and big-endian integers and the compiler should generate the appropriate code.
At least in the languages with user-defined data types and overloadable operators and functions, like C++, you can define these yourself, when the language does not provide them, instead of using ugly workarounds like htonl and the like, which can be very inefficient if the compiler is not clever enough to optimize them away.
That's where big endian is now. All the BE architectures are dying or dead. No big endian system will ever be popular again. It's time for big endian to be consigned to the dustbin of history.
And, especially what most people call big-endian, which is a bastardized mixed-endian mess of most significant byte is zero, while least significant bit is likewise zero.
Cries in 68k nostalgia
However if designing a new network protocol, choosing big endian is insanity. Use little endian, skip the macros, and just add
#ifndef LITTLE_ENDIAN
#error
Adding other architectures to your build system also tends to reveal nasty bugs in general, e.g. you were unknowingly triggering UB on all architectures but on the one you commonly use it causes silent data corruption whereas one with a different memory layout results in a much more conspicuous segfault.
FWIW I doing hobby-stuff for Amiga's (68k big-endian) but that's just that, hobby stuff.
Or you can just be a nice person and make your code endian-agnostic. ;-)
(1) for JPG for embedded TIFF metadata which can have both.
[2] https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.ht...
The only question that matters: Do your customers / users want to run it on big-endian hardware? And for 99% of programmers, the answer is no, because their customers have never knowingly been in the same room as a big-endian CPU.
If the data stream encodes values with byte order B, then the algorithm to decode the value on computer with byte order C should be about B, not about the relationship between B and C.
The point (for me) is not that your code runs on a s390, it is that you abstract your personal local implementation details from the data interchange formats. And unfortunately almost all of the processors are little, and many of the popular and unavoidable externalization are big...
[0] https://commandcenter.blogspot.com/2012/04/byte-order-fallac... [1] https://github.com/apple/darwin-xnu/blob/main/EXTERNAL_HEADE...
Their x86 changeover moved the CPU's to little-endian and Aarch64 continues solidifies that tradition.
Same with Java, there's probably a strong influence from SPARC's and with PPC, 68k and SPARC being relevant back in the 90s it wasn't a bold choice.
But all of this is more or less legacy at this point, I have little reason to believe that the types of code I write will ever end up on a s390 or any other big-endian platform unless something truly revolutionizes the computing landscape since x86, aarch64, risc-v and so on run little now.
Most CPUs (including x86-64) have variants of the load and store instructions that reverse the byte order (e.g. MOVBE in x86-64). The remaining CPUs have byte reversal instructions for registers, so a reversed byte order load or store can be simulated by a sequence of 2 instructions.
So the little-endian types and the big-endian data types must be handled identically by a compiler, except that the load and store instructions use different encodings.
The structures used in a data-exchange format must be declared with the correct types and that should take care of everything.
Any decent programming language must provide means for the user to define such data types, when they are not provided by the base language.
The traditional UNIX conversion functions are the wrong way to handle endianness differences. An optimizing compiler must be able to recognize them as special cases in order to be able to optimize them away from the machine code.
A program that is written using only data types with known endianness can be compiled for either little-endian targets or big-endian targets and it will work identically.
All the problems that have ever existed in handling endianness have been caused by programming languages where the endianness of the base data types was left undefined, for fear that recompiling a program for a target of different endianness could result in a slower program.
This fear is obsolete today.
The adjacent POWER architecture is also still relevant - but as you say, they too can afford a support contract.
I contend it's almost never important and almost nobody writing user software should bother with this. Certainly, people who didn't already know they needed big-endian should not start caring now because they read an article online. There are countless rare machines that your code doesn't run on--what's so special about big endian? The world is little endian now. Big endian chips aren't coming back. You are spending your own time on an effort that will never pay off. If big endian is really needed, IBM will pay you to write the s390x port and they will provide the machine.
One difference is that when your endian-oblivious code runs on a BE system, it can be subtly wrong in a way that's hard to diagnose, which is a whole lot worse than not working at all.
u32::from_le_bytes
u32::from_ne_bytes the n stands for native
You're also the most likely person to try to run your code on an 18 bit machine.
Granted, I still work on a fair number of big endian systems even though my daily drivers (ppc64le, Apple silicon) are little.
> [fixes] specific to VMS (a.k.a. OpenVMS),
> For conformity with DECSYSTEM-20 Kermit ...
> running on a real Sun3, compiled with a non-ANSII compiler (Sun cc 1.22)
> this is fatal in HP-UX 10 with the bundled compiler
> OpenWatcom 1.9 compiler
> OS/2 builds
> making sure that all functions are declared in both ANSI format and K&R format (so C-Kermit can built on both new and old computers)
Oooooh! A clang complaint: 'Clang also complains about perfectly legal compound IF statements and/or complex IF conditions, and wants to have parens and/or brackets galore added for clarity. These statements were written by programmers who understood the rules of precedence of arithmetic and logical operators, and the code has been working correctly for decades.'
Many of the tests I did back in the 1990s seem pointless now. Do you have checks for non-IEEE 754 math?
It's also increasingly hard to test. Particularly when you have large expensive testsuites which run incredibly slowly on this simulated machines.
https://gist.github.com/siraben/cb0eb96b820a50e11218f0152f2e...
Nice article! But pity it does not elaborate on how...
But Hans's post uses user-mode emulation with qemu-mips, which avoids having to set up a whole big-endian system in QEMU. It is a very interesting approach I was unaware of. I'm pretty sure qemu-mips was available back in 2010, but I'm not sure if the gcc-mips-linux-gnu cross-compiler was readily available back then. I suspect my PPC-based solution might have been the only convenient way to solve this problem at the time.
Thanks for sharing it here. It was nice to go down memory lane and also learn a new way to solve the same problem.
On Linux it's really as simple as installing QEMU binfmt and doing:
GOARCH=s390x go testpresented at Embedded Linux Conf
Eh, is it? There aren't any big endian systems left that matter for anyone that isn't doing super niche stuff. Unless you are writing a really foundation library that you want to work everywhere (like libc, zlib, libpng etc.) you can safely just assume everything is little endian. I usually just put a static_assert that the system is little endian for C++.