
OpenSSL 1.1 API migration path, or the lack thereof - brynet
https://www.mail-archive.com/tech@openbsd.org/msg36437.html
======
awalton
It's sad they didn't learn from other libraries making similar transitions
from public to private structures - introduce accessors, introduce a build
macro to mangle fields so code building against these fields generates
warnings, and then when code has been sufficiently ported, pull the rug,
remove the accessor macros.

This approach makes it quite apparent where missing accessor functions exist,
where loss of functionality might come from, and it's easy to do over an
intermediate period of time while not being too obnoxious to work with during
the period.

But good software engineering practices and OpenSSL seem to be mutually
incompatible so I guess I shouldn't be all that surprised. Leave it to them to
do the most obnoxious thing every time.

~~~
xemdetia
I think a comment like this makes light of the timeframe that ABI/API
compatibility has been for the 1.0.0 series. 0.9.8's ABI/API has been floating
around since 2005 and 1.0.0's been floating around since 2010- yet major
things have been added such as TLS 1.1 and TLS 1.2.

The ABI/API of the older versions had eventually become inadequate to describe
what modern TLS/crypto applications need to do, and people had just
consciously worked around it to dig into structs that they should have never
been accessing anyway. Most problems I have fielded regarding this almost
always turn out to be things that you never should have been accessing to
begin with, but because of necessity people did.

If anything this highlights a more general problem with linking strategy
against core libraries. There is no ability to add accessors without having a
high risk of breaking the universe, and this makes the OpenSSL problem worse
because packagers downstream like Debian and RHEL are making a bet against
doing nothing and breaking nothing or doing anything and breaking everything.

~~~
awalton
> There is no ability to add accessors without having a high risk of breaking
> the universe

Hogwash. Adding a function doesn't affect existing programs _at all._ The
tricky part is getting people to use the function instead of the field to get
the same information. That's the point of introducing the macro around field
access: either forcing people to define things in their build to keep things
compiling (acknowledging their impending doom and refusing to do anything) or
fix their code. If you write the macros in a clever enough way, they'd have to
touch every access site to get things building again anyways, so they might as
well use accessor functions when they go to port.

Then define a hard date in which things are going to break (near enough in the
future to call people to action, far enough to actually complete a port if
they're trying), and then break as close to that date as you can to both build
trust and confidence in the community. (This of course includes feedback; if
everyone is saying the port is taking twice as long as planned, adapt your
break date.)

This is not actually difficult, it just requires software maintainers that
care, and a healthy environment of engineers willing to do the work to clean
up their software. And it also requires software landfills like Debian and
RHEL repos systematically purge software that refuses to update or has zero
active upstream maintainers, as it's likely filled with badware, or at the
very lease insecureware, anyways.

Where OpenSSL breaks down is the healthiness to take these steps; although
recovering thanks in a large part on the spotlight placed upon them and the
few paltry dollars offered up by various tech companies to maintain the
commons, they still haven't caught up to simple practices like these. (I'll
partially blame this on fragmentation like LibreSSL and BoringSSL as enabled
by software licensing and companies like Apple (and to a lesser extent Amazon,
s2n) deciding to abandon public implementations and write their own based on
these public ones rather than returning effort to community solutions, but
that's a topic for another day when I feel like donning asbestos armor and
beating up trolls with cluebats.)

------
aaronmdjones
I help maintain an IRC server that can use OpenSSL (or GNUTLS, or MbedTLS) and
I added OpenSSL 1.1 compatibility _last year_ (that is, 2015, while it was
still under heavy development).

It will compile against (and work with) OpenSSL versions 1.0.0, 1.0.1, 1.0.2,
1.1.0, LibreSSL version 2.0, LibreSSL version 2.2, and LibreSSL version 2.4,
with only a small header file and some ifdef soup [1]. I imagine adding
support for LibreSSL 2.5 would be trivial, if anything needs changing at all.

The C pre-processor exists. Learn it. Use it. Love it.

[1] [https://github.com/charybdis-
ircd/charybdis/blob/release/3.5...](https://github.com/charybdis-
ircd/charybdis/blob/release/3.5/libratbox/src/openssl_ratbox.h)

~~~
jjnoakes
The contents of the email contain an argument against this approach.

~~~
aaronmdjones
I don't see it.

The email states that writing your own wrappers or accessors is a bad idea
(and I agree), but that's not what's going on here -- I'm using the
preprocessor to decide which code path to follow based on the version we're
building against.

It's as if it was only written for 1.1.0 (when building against 1.1.0), and
only written for 1.0.2 (when building against 1.0.2), you get the idea.

The only maintenance burden then is figuring out how to do something in both
versions, and adding future preprocessor branches if the API changes again.
This is always the case when deciding whether a new library feature is
available, anyway.

------
tiles
> But here's the thing - there are at least two options that OpenSSL had (and
> still have) that would have avoided this situation and allowed for software
> projects to make a smooth transition from the old API to the new API[.]

Are these two options specified anywhere in the email?

~~~
tedunangst
The accessors can be backported to older releases, or the types can remain
transparent in 1.1.0 and removed sometime later.

------
tedunangst
A bit of context here is that LibreSSL can add the 1.1 API accessors without
much trouble and still retain compat with the 1.0 API. But that won't directly
benefit users who use OpenSSL and for any one of many reasons would like to
develop against both library versions.

------
fortytw2
> There are also hidden gotchas for situations where different libraries are
> linked against different OpenSSL versions and then share OpenSSL data
> structures between them[7] - many of these problems will be difficult to
> detect since they only fail at runtime.

Why are libraries written in this way? Seems fairly insane to expose parts of
another library as a part of your public interface. This feels like the
shining example of a poor practice coming back to bite, hard.

~~~
cperciva
_Why are libraries written in this way? Seems fairly insane to expose parts of
another library as a part of your public interface._

It's not necessarily intentional. One of the more "interesting" bugs we've
seen in FreeBSD was when a program linked to libraries A and B, and library A
linked to libcrypto (openssl) while library B linked to libmd (a lightweight
hash function library).

Each part worked fine independently, but when it was all put together, the
dynamic linker said "we need to find an object called SHA256_Init? Oh look,
it's in this library we already loaded!" with predictably hilarious results.
(We now use preprocessor macros to define _libmd_SHA256_Init etc. to avoid
such linker confusion.)

What you think your public interfaces are and what the linker thinks your
public interfaces are don't necessarily align.

~~~
rkeene2
Symbol versioning can solve this at the linker layer without modifying the
source. One possibility is to use --default-symver when compiling OpenSSL and
libmd, then when they are linked the appropriately the soname is used as a
version tag for any unversioned symbols.

An example: [http://www.rkeene.org/viewer/tmp/linking-
troubles/log.html.h...](http://www.rkeene.org/viewer/tmp/linking-
troubles/log.html.htm)

If you are a library maintainer and export a stable ABI, it might also be good
to provide a version or every symbol if for no other reason than to avoid
issues with the flat namespace.

------
rurban
Contrary to the op's subject I can easily see the benefit of the API changes
in the new major version, and I see good documentation of the API migration
path:
[https://wiki.openssl.org/index.php/1.1_API_Changes](https://wiki.openssl.org/index.php/1.1_API_Changes)

It will need some time to port all these projects over to 1.1 but everyone can
easily see the benefits. Supporting both will be a nightmare. Just drop 1.0
support and require 1.1

~~~
gbrown_
> Just drop 1.0 support and require 1.1

The point is applications won't be able to easily do this. As the post points
out it's likely many distributions will adopt the 1.0 release series due to
the length of support. So if you only use to 1.1 API you may find the burden
of installing another OpenSSL release is placed on your users as their base
system contains a version of OpenSSL with an incompatible API.

To be clear I think the API change has good intention but it would be been
nicer to have marked parts depreciated and introduced new interfaces more
gradually. OpenSSH is a good example of this in terms of advertising ahead of
time breaking changes.

------
daurnimator
As someone who recently went through probably one of the larger bits of work
porting from openssl 1.0 to 1.1; I can say I _mostly_ appreciate the new API
(main problem is a few missing setters/getters, see e.g.
[https://github.com/openssl/openssl/issues/1620](https://github.com/openssl/openssl/issues/1620),
[https://github.com/openssl/openssl/issues/1885](https://github.com/openssl/openssl/issues/1885)).

Outside of bindings to OpenSSL for each language, the migration path is pretty
simple. This was a cleanup that had to happen sooner or later: projects using
OpenSSL were never going to have an option to _not_ change.

~~~
tedunangst
The migration itself is fairly simple, but once you make it you're shut out of
linking against 1.0, which is where the majority of users still are. It's an
unnecessary python3 type scenario. People are going to drag out the transition
for as long as 1.0 remains viable. So only like three more years.

~~~
daurnimator
You add small compatibility macros. Which most openssl code already had. We
still have to support openssl back to 0.9.8, so this isn't out of the
ordinary.

See
[https://github.com/wahern/luaossl/blob/master/src/openssl.c#...](https://github.com/wahern/luaossl/blob/master/src/openssl.c#L1243)
for all the compatible wrappers we have

------
mangix
Is this incentive for people to migrate to LibreSSL? I mean LibreSSL doesn't
break the API like this.

------
MBlume
Could a third party release a shim that everyone agrees to use, sort of like
Python six? [http://pythonhosted.org/six/](http://pythonhosted.org/six/)

------
disposablezero
Just ran into this with the puma ruby gem's native extension. OpenSSL 1.1
should've been called OpenSSL 2.0.

