Hacker News new | past | comments | ask | show | jobs | submit login
ABI vs. API (2004) (debian.org)
268 points by hiq on July 4, 2016 | hide | past | web | favorite | 61 comments

Best and short explanation I've ever read.

It's a pity that Colin Watson left Debian's Technical Committee (https://lists.debian.org/debian-ctte/2014/11/msg00052.html), but his blog is still worth reading: http://www.chiark.greenend.org.uk/~cjwatson/blog/

Great resource about C++ ABI compatibility (what changes ABI, what doesn't, and how to change code without changing ABI):


Having maintained a C++ API and ABI based on implementation inheritance with virtual functions, I have deep respect for both kinds of compatibility. Someone even used placement new to put objects with vtable pointers in shared memory, expecting destructors to work in a different process!

Check out the "fragile base class problem" some time for more things than can go wrong.

Something that want made explicit: If you support closed source, shipping binaries, then the ABI can't change, or you break all compiled binaries users have installed. If rebuilding the world from source is an option, changing the ABI is an option.

(Introducing a new ABI for new subsystems is still OK - hence renaming libc when it changes)

(Say what you want about Microsoft, but COM solved that problem pretty well in the '90s...)

> If rebuilding the world from source is an option, changing the ABI is an option.

It's an option ... but not a very good one.

It tends to create an unstable environment that requires a centralized organization just to keep it working, precluding a healthy diverse software ecosystem.

This is a major reason why it's very difficult to distribute binary packages when operating externally to the distribution packagers, and why commercial software distribution on Linux is a massive pain in the ass.

The ultimate guide on writing shared libraries, ABI compatibility, etc, is this document written by Ulrich Drepper: https://www.akkadia.org/drepper/dsohowto.pdf

Hmm. I've always understood 'ABI' to mean something like 'the conventions for turning an API into machine code'. So the fact that floating-point arguments go on the stack is part of the ABI, whereas the fact that some function takes two floating-point arguments is part of the API. This seems to be the meaning used in the System V Application Binary Interface [0].

[0] https://www.x86-64.org/documentation/abi.pdf

30 years ago, this article could have mentioned the OSI model (https://en.m.wikipedia.org/wiki/OSI_model)

Although desribed in terms of networks, it is flexible/abstract/content-free (1) enough to describe communication between software components on a single machine.

An API describes what programs/libraries say to each other, an ABI is a lower layer that describes how they should do that. I would place them on levels 4 and 3. Level 2 is the hardware in the CPU, level 1 are electrons moving through tiny wires, but one can change that to something different (light flowing through fibers, a human keeping track of state on paper, etc) without affecting ABI nor API.

Looking at this this way, it is clear that there are levels above an API. For example, one could place the semantics of the API (you can only close a file that is open, free memory you allocated, etc) at level 5.

(1) I think it is a mix of these, but feel free to pick an adjective you like.

That's why developing with shared libraries is such a PITA. Got to preserve those precious ABIs.

What exactly do you mean by PITA? In most cases in my experience, you have two very simple choices - 1. don't modify the existing API / public structures as you change things and the ABI will stay safe; 2. just change things and bump up major number when necessary.

The third way of carefully doing only additive modifications to the library to preserve ABI even over upgrades is hard - that's true. But I don't think it's needed that often once most of the features are in. And again - it applies only if you've got some really popular software that gains anything by doing it this way, rather than just bumping up the version.

This. Working with SO versions is much easier in my experience than working with Ruby gems where every minor upgrade could (and sometimes would) break everything.

So with shared library dependencies, you can be very broad (e.g. "any libm.so.5 please") and get security updates etc. automatically without needing to recompile, while in "modern" programming languages you will be locked onto old and rancid libraries because the app developer doesn't bother updating the locks.

And then distro package managers go about and complicate things by having overly rigid dependencies and only allowing one version of each package name to be installed.

And the recently introduced container based replacements are and exercise in shooting twee twee birds with AAA...

Sure, which is why we have `libopenssl0.9.8`, `libopenssl1.0.0`, etc. In Debian at least, the policy is that package names for libraries should include something approximating the SONAME to allow for exactly this.

Eventually you drop the old one, because you can no longer provide security support or for some other reason. But if there are compelling reasons otherwise, distributions can and do provide multiple ABIs for the same library.

And that just exasperates the problem.

At the very least shoulnd't they be providing and supporting multiple major versions on the same system? Major versions are, by definition, for breaking/incompatible changes, so having them being mutually exclusive creates impossible situations doesn't it?

Well yes, but using the package name to distinguish them makes it harder for third parties to do packages. This is the complaint that has lead to the development of container based formats in the first place.

Thing is that with sonames the actual files are distinguished anyways, so mangling the package names to get around collisions are just an artifact of an overly rigid package manager.

I think you are talking about compatibility in general (which is great to have!) but not about ABI compatibility. Nothing apart from cultural conventions prevents maintainer of a ruby gem from strictly adhering to semver or maintainer of a C library from breaking everything in a minor version release.

Agreed. Ruby just hasn't developed the culture of compatibility that's mostly adhered to e.g. in Qt/KDE land where I lived for a long time.

Related: I'd like it very much if compatibility was enforced automatically. For example, if I publish version 1.2 of my library to $repository, then $repository checks if 1.2 passes the 1.2 tests (of course), and also if it passes the 1.1 and 1.0 tests. If any of these tests fail, then release should be downright refused unless you upload it as 2.0.

And of course, once you have that, you can think of any check you like. In C++, you can pretty reliably detect code changes that change your ABI in backwards-compatible or -incompatible ways, so you could allow backwards-incompatible changes only in major and backwards-compatible changes only in minor releases.

What I mean is that in addition to rules of the language I program in (say C or C++) there is another bunch of seemingly arbitrary rules that I must constantly be aware of to ensure nothing breaks. Rules like "don't reorder fields of structs" or "you can add new methods to a class as long as they are not virtual. and sometimes you can add virtual methods too as long as they end up at the end of vtable".

How do I ensure that I conform to these rules apart from being disciplined about them? If everything compiles, am I good? No! If everything links, am I good? No! Depending on linker options and the nature of incompatibility the program using incompatible ABIs can blow up at runtime or just silently corrupt data.

And don't get me started on the venerable "pimpl idiom". A page of boilerplate just to ensure the most basic thing.

Hope that clarifies my short sentiment a bit :) I agree that once you grasp the rules following them is not that hard but it is just another bit of incidental complexity that we agreed to maintain.

I think most of the issues you listed are quite easy to check automatically. For example http://ispras.linuxbase.org/index.php/ABI_compliance_checker or anything in the "similar software" section.

Of course it won't save you from business logic bugs (changing the meaning, but not size/location of a value), but "how do I ensure that I conform" is 90+% possible to verify automatically.

Thanks, didn't know about it! Still, that's a 3-rd party tool and I still have to write my pimpls. I stand by my PITA sentiment :)

It is an additional thing to be concerned about, but the alternative (static linking everywhere) is so much worse! At least, for most (but not all) situations.

It's sort of like democracy (or capitalism?): the worst system in the world, except for every other alternative. :)

Having said that, many of your pain points are specific to C++ rather than system-level ABIs. C++ compilers have to abuse most OS-level system ABIs (example: symbol name mangling) because of linkage-related concepts in C++ that don't exist at the lower-level (and simpler, yet very different) system ABI provided by the OS.

In a way, you're really complaining (rightfully so, IMO) about one of C++'s core design principles: that the programmer should never pay a performance cost (compared to C) for language features that aren't used. For example, methods are non-virtual by default. Support for virtual methods are required for polymorphism, one of the defining features of OOP. However, calling a virtual method is always going to have at least a teensy bit of overhead vs. calling a non-virtual method. As a result, in C++ all methods are non-virtual unless defined otherwise.

Higher-level (and admittedly <= C++ in performance) OO languages tend to make everything virtual by default, and may not even have a mechanism for making a non-virtual method. These languages give away a small amount of performance in exchange for a reduction in cognitive load on the programmer. Reducing cognitive load is also a reduction in potential bugs, so there may be pragmatic reasons for using a higher-level language other than making the developer's life easier.

Most (but not all) other OO languages take away the option of direct memory management, and instead have some variant of garbage collection to keep memory usage somewhat constrained. C++ can't do this by default because then it would take a performance hit compared to C. So again, you get stuck with one (or two or many) options for higher-level memory management, but by default you're doing manual memory management a la C.

I can see (heh) C++ being very appropriate for certain types of projects and certain types of developers, but IMO its popularity is mysteriously much greater then its usual level of appropriateness. I don't dislike the language (OK, maybe a little), but it's unfortunate that it is considered the default alternative to C as often as it is.

C++ ABI compat rules really hurt when they prevent you from evolving a library in a sensible way. Say that you really need to add a method to a virtual class. Unless you planned ahead by adding placeholder methods, you're SOL. What a lousy design constraint!

This problem is solvable at the language/runtime level by resolving vtable offsets at runtime, but that doesn't help the existing C++ userbase.

It hurts so much that the boost people decided they wouldn't even bother trying.

This is a major flaw of C++'s ABI design, unfortunately, and one reason why we see so few supported ABI-stable C++ libraries shipped with major platforms.

Most people just have to incorporate their C++ dependencies directly instead, as there is no supported ABI.

C/C++ shared libraries are very easy to work with and distribute. Unfortunately this is becoming a lost skill in the world of javascript. People inherently thing 'old = bad' and are doom to reinvent the same thing over and over.

In my opinion, with every change in ABI, the build system should transparently update all binaries. The only thing the developer should notice is that the build takes longer.

Debian has automation to do exactly that: when a library changes ABI, but has the same source API, such that a rebuild will suffice to migrate to the new version, they can trigger a rebuild of every package that build-depends on the library -dev package. (That automation works for other kinds of rebuild-only migrations, too, such as rebuilding packages with a new version of a build tool that fixes some bug in the resulting built packages.)

That means the build system has access to all source code of all binaries compiled against it. That is doable within projects (make dependencies), and within centralized open-source distributions such as Debian (through archive rebuild triggers).

But it is impossible in general.

Wouldn't there be idioms in the assembly which represent compatibility with the old ABI? Couldn't you translate those directly in the assembly without having to re-run the compilation?

ABI describes things like how the stack works, but at the assembly level the only stack is the one you implement, which is generally compatible with an ABI version.

Not easily. What you are describing is annotating every entry point in a library with symbol tags, and annotating every function call with the same symbols. It basically requires you to have another JIT compilation phase or runtime library to resolve function calls. We do have examples of that (e.g. .net assemblies), but it requires a lot of infrastructure to support it.

And there's still more than one stack implementation choice at the assembly level. For example, what is the correct order for passing parameters on the stack? What is the memory alignment for off-size stack parameters? Is the caller or the callee responsible for reclaiming the stack space at function completion? How are function return values passed back?

For that, shouldn't the build system know how to build each library (i.e. it should have sources and makefiles)?

Can someone explain how the changing of a dependency abi breaks things both at compile and run time?

A good recent real life example of this is when the POWER architecture moved to Little Endian. The API's stayed the same (think GLib C API), but LE introduced a new ABI where registers (ie r12) which had a specific purpose in BE changed in LE. This all has to do with linking/compiling.

At runtime if you're reading/writing to memory you had to ensure that you read and write using the same "stride" ie reading and writing back in 32bits is fine, but if you read in 32bits do some transforms in a register and write it out in 16bits you'd have to make sure the internal register BE representation and the written out to memory LE representation made sense.

The GCC and other compiler guys can give you lots of war stories about SIMD (VMX/VSX on POWER)

Anyways, just something I thought off the top of my head :)

Thanks. I've been reading up on OS Dev and static and dynamic libraries and have been trying to wrap my head around why an ABI change could blow things up.

very tenuously related, but I forgot that Ian Murdock died over Xmas. crazy. any updates on what happened there?

One of the worst design decisions of Unix to fundamentally separate the source from the binaries. And Debian and those of their ilk had to make it worse with that -dev package nonsense.

Unless I misunderstand you, I thought the exact opposite was true: The separation of source from binaries is what made UNIX a success. When C was designed, it allowed one to take the source from one machine to another, disregarding the underlying machine architecture.

What made UNIX a success was AT&T being prevented to charge money for it, thus making available for free versus the alternatives.

I assume (presume perhaps) they're a windows user and the -dev mention is more for the case where debug symbols and packages are separated from non debug binaries.

I only know that windows does something different more akin to debug symbols in each binary by default with chicanery to not have it impact things in the general case. But I'm not a windows programmer this is about as much as I know about windows in that regard.

Never been a Windows user or programmer, but like you I've heard they do quite a lot of clever stuff under the hood. Probably a lot of VMS heritage, another system I've never had a chance to use but heard good things about. But every version I've tried since Windows 3.1 has had an abysmal user experience, so I've never once considered switching.

And that leaves no current viable alternatives to the Unix-like systems. Sad, really.

It didn't make Unix a success, it made Unix a punch-card-and-tape batch processing simulator in the same way the Mac was a paper pushing simulator.

What made Unix a success is a great mystery to me. Presumably it was the ability to run a time-sharing system on low-cost hardware, but then people started taking the tradeoffs it had to make as gospel, which is why we have this ridiculous situation of still wrangling processes on computers with 64 bit bloody address spaces.)

I am not sure I understand your comment. UNIX was commercial and didn't ship with source code at all unless you paid for a very expensive license.

Of Unix?

btw: so are you advocating a full GPL-like distribution of the sources alongsides the binaries for all programs?

> so are you advocating a full GPL-like distribution of the sources alongsides the binaries for all programs?

Not the parent, but I definitely advocate for that, so long as it maintains user freedom.

how different is ABI from XPCOM? Wasn't xpcom had the same goals to provide binary compatibility?

ABI is not a specific technology in the same way 'API' is not. So the ABI doesn't have any specific goals, it's just something that emerges when working with dynamic linking.

XPCOM is one particular ABI.

A decent explanation, but one part strikes me as odd:

> Changing a library's API in a backwards-incompatible way is in general bad form because it means that developers of programs using the library have to change their source code to port the programs to the new library. However, it does happen sometimes in the case of major libraries, such as some of those in GNOME and KDE.

So only the big guys are allowed to break the APIs? Really?

Nothing in that comment said you can't do whatever you want - it said, quite specifically, it's bad form to do it, and then explained exactly why that was the case.

Plenty of libraries break API/ABI all of the time. Notoriously FFMPEG was/is terrible about it. But depending on libraries that are hugely unstable in this way is terrible, so in most cases, people go out of their ways to avoid doing so (most frequently using another library, or in harder-to-avoid cases creating an interposer library to smooth out the instabilities of the underlying library).

OSS developers probably know more than anyone in the world that the most expensive thing to do in software engineering is maintaining a piece of software as time moves forward. Anything that makes that job more difficult needs to have an enormous upside to make it worth while, and in almost every case, using an unstable library is just not worth the extra maintenance costs attached.

All true, but for some reason he sees the major libraries as a different beast and I just don't see why, which is why I wrote it strikes me as odd.

I think you are interpreting it as "Major libraries are allowed to do it sometimes."

I think what he meant to say is "Even major libraries do it sometimes." It is already understood that minor libraries do it.

I read it as: major libraries are reluctanly tolerated to break their API/ABI now and then. They are different beasts, whether you like it or not.

A colorary to the above is: libraries that break their API/AIB too early or too ofter never get a chance to earn the 'major' label in the first place.

You could also argue that a major library that abuses their privilege and breaks backwards compatibility beyond their "karma ballance" will be eventually forked or otherwise replaced by a more stable competitor. I am not sure if this idea holds water in the real world, but I really hope it does.

A major system like that can't shift very often because if it changes even slightly it brings down a proverbial house of cards built on top of it.

Little inconsequential libraries can move fast and break things, though there's a peculiar paradox here. As the number of dependencies grows, the tolerance for change decreases, yet in terms of perception, little libraries that aren't major dependencies that change frequently look untrustworthy and are liable to never be trusted because of their flux.

That is, if you want to be successful, be predictable and consistent even if that means being leaving some potential unrealized.

I read it as "it's bad form, although it has been known to happen."

He is not saying that the "big guys" are allowed to break APIs. He is remarking that in the case of major libraries, they break APIs. Thats simply an observation, nothing else.

Ah this is possible, but then this observation seems limited to me as smaller libraries break APIs too.

The API break of Gnome 2 to Gnome 3 and KDE 3 to KDE 4 etc. are just more commonly known than those of some small library. So it is just a good example.

That's because you can simply add functionality and increase the minor version of the library. If you change the first digit you have removed or modified an interface. The same can be said about an API. Most API developers don't delete an API call or change the POST params on working code, they simply add a new function.

I think you should indicate that this is a off-topic discussion from 2004 debian user mailing list somewhere in the link title.

"Off topic" to what? The title chosen here matches the contents it links to.

Probably [OT] in the original title means "off topic" but I don't know why we should care about this here.

Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact