Hacker News new | past | comments | ask | show | jobs | submit login
What changed in OpenSSL after heartbleed (arxiv.org)
158 points by gnu 26 days ago | hide | past | favorite | 64 comments

I'd be more interested in a comparison in the strategies used to harden the codebase in the forks like BoringSSL and LibreSSL, and how well those strategies have panned out.

There has historically been some crowing from the LibreSSL crowd about how their work avoided CVE's later discovered in OpenSSL: https://undeadly.org/cgi?action=article&sid=20150319145126

I've been doing differential fuzzing of many major cryptographic libraries. It currently tests symmetric crypto, some elliptic curve crypto, bignum arithmetic, message digests and MACs, KDFs, but not TLS, X509, ASN1 etc (yet). The list of bugs it has found so far can be viewed here [1].

The bug count per library can not be used as an absolute metric and not all bugs are security vulnerabilities (though many can be under specific circumstances).

BoringSSL has fewer bugs than LibreSSL, which has fewer bugs than OpenSSL. One of the reasons for this could be that the bug count is proportional to the complexity/SLOC: BoringSSL is smaller in terms of functionality (# of message digests, ciphers, ...) than LibreSSL, which is smaller than OpenSSL. OpenSSL sometimes commits new buggy code whereas BoringSSL does not, but OpenSSL has a higher commit frequency than BoringSSL, and BoringSSL might be aiming for a production-safe master branch where OpenSSL might not.

So it's not easy to derive a reliable metric from this, but with that said, you can't really go wrong with BoringSSL if it offers what you need.

[1] https://github.com/guidovranken/cryptofuzz#bugs-found-by-cry...

> but not TLS, X509, ASN1 etc (yet)

then this may interest you:


^^ ASN1 is really the bees-knees for fuzzing telecoms protocols in UMTS/LTE/5G etc and doesn't get enough love in other domains. It's a high learning curve but once you get beyond the "standardese" language in the docs it's opening doors to opportunities in so many industries.

> BoringSSL has fewer bugs than LibreSSL, which has fewer bugs than OpenSSL. One of the reasons for this could be that the bug count is proportional to the complexity/SLOC

the openSSL codebase is notorious but I think it's also because it has been in existing for so long. if I look around today I see 2 camps: cryptographers and software engineers. letting sw-engineers do crypto is usually a bad idea but it's often worse when cryptographers start coding. it's almost like a variation of the old joke of "the 2 most dangerous things in Tech are a sw-engineer with a soldering iron and a hw-engineer with a sw-patch" ... apart from complexity leading to bugs I'd also say there is another downside which is stronger in openssl: people end up using it wrongly which makes it a proverbial foot-gun for implenters.

> [1] https://github.com/guidovranken/cryptofuzz

very cool thanks!!

> not all bugs are security vulnerabilities

I think to make such a claim you need to specify attack model constraints. With an unconstrained model all bugs in a library are in fact security vulnerabilities.

This does depend on a very narrow reading of what constitutes a bug, for example I would not consider it a bug per se if a function does something that you wouldn't expect (and which perhaps the developers didn't intend) but does strictly match the documentation.

The problem is that the application software developer may depend upon documented features of your library in ways you didn't anticipate and if those documents are wrong (because of a bug) then you can't be sure it hasn't got security implications somewhere.

Code quality and hygiene mean absolutely nothing if you have a large number of academic types who use OpenSSL as the dumping ground for their pet research projects, that are enabled by default, of course.

Also, OpenSSL supports all kinds of ancient esoteric platforms that are essentially unused, yet were kept in the code base for sentimental reasons.

The real metric they should be looking at is the number of features/platforms/LOC removed from the project. Less code = less surface areas for exploits.

I think the multiplatform support is an interesting criticism but misunderstood.

I happened to be reading some OpenSSL code around the time of heartbleed and statically linking some of it in a project. I found a lot of the portability ifdefs were poorly done even for the standards of the 90s. It wasn't portability itself that was the issue, it was actual quality and in many cases the wrong abstractions in place.

One example that sticks out in my memory... There was a logger that had a Win32 ifdef. If you ran on Win32 it fed log messages directly to MessageBox(). Unless it detected the current process was an NT service, in that case it used another logging mechanism without asking. It wasn't actually "windows portability" or "windows support". The whole thing wasn't appropriate for a library. It could have had a mechanism to give the log lines to the application as a C string and be done with it. Instead, it was mixing of library layer stuff and application layer stuff, or just ordinary library bloat.

The other huge example I recall was compliance with older configurations that were pre-C99. C99 is much more universal now vs 20 years ago, although there are a few things Microsoft still doesn't support. But again, a bunch of these things seemed to be handled with ifdefs at the call site, rather than put in a proper compatibility layer that only an older configuration gets.

It does seem that in the years since then, cruft has been removed not only in forks but also in the upstream project.

An aside my feeling about Microsoft and their lack of C99 support is the community was inexplicably too nice to them. As well as compilers defaulting to C89 instead of the latest.

Also as someone that has a code base that targets a dozen different pieces of hardware ifdef madness is a strong indication that your program architexture is broken. Sometimes it's unavoidable but you should think of ifdef's as another name for FIXME.

Microsoft has gotten a bit better at it after many years of zero progress. Vs2010 introduced stdint and stdbool. Vs2015 finally allowed mixed declarations and code. I think the most lacking piece lately is the named struct member initializers.

Almost all the progress made was necessary to support modern C++ standards, although it is nice they enabled it in the C compiler. They haven't added any C only features in a very long time.

During a code review we found recently written Go code using SHA1, which has been functionally obsolete since 2005. It's a bit baffling why Google even included it in Go or at the very least didn't make it throw up a bunch of warnings to only use it for very special edge cases related to backward compatibility.

The developer didn't know what algorithm to pick so he just went with one at random, assuming it was ok since it was in the library. How many other security vulnerabilities are out there due to similar circumstances? It's a bit troubling.

Git still uses SHA1, as does a bunch of other not-really-security-critical things (mostly as a much stronger checksum against corruption rather than malice.)

If the purpose of the hash in that code was security-critical and compromised by malicious collisions, it would definitely be a problem. Otherwise it shouldn't be --- and jumping at things without understanding the nuance is precisely one of the problems with the "security industry" today.

Exactly. At a former employer, we had the VA once symbol dump a library we were using and try to knock us since it included a symbol for a sha1 function. The hash was used for a non-security-critical component of web-sockets as a simple collision resistant session identifier. Good luck explaining this to the same people that tried to knock us for using a function that is deprecated in the windows libc implementation for not length-checking its parameters but has always worked correctly on Linux and macOS, citing the windows developer portal as documentation for why our mobile Android and iOS apps were cause for worry.

Unused? Ha ha, good one. Our VAX will never die.

    $ set def [.openssl]
    $ run openssl
    WARNING: can't open config file: SSLROOT: 
    OpenSSL> version
    OpenSSL 1.0.0r 19 Mar 2015
    OpenSSL> quit

So that makes, what, 2 of you? For widely used libraries and programs "no one" should be approximated to "less than 0.01% of users".

The platform-specific stuff is not where most of the bugs have been found, so your assertion is misguided. Portability is one of the greatest strengths "keeping crypto freely accessible for all" and the fact that e.g. with a suitable C89 compiler I can create a binary that will run on an original 8088 IBM PC and perform SHA256 or AES should not be underestimated.

To a first approximation, crypto is pure maths. The rest can be taken care of by the standard C library.

Not only that, but the testing on oddball platforms can be a good way to reveal latent bugs.

"dumping ground for their pet research projects, that are enabled by default, of course"

How does that work? How does anyone even approve it if it isn't going to be used?

Admittedly I'm out of the loop as far as contributing to such projects, maybe letting that stuff in is the norm?

> How does that work? How does anyone even approve it if it isn't going to be used?

How do you know it won't be used if it isn't put in in the first place?

What kind of things are we talking about that someone would add to OpenSSL outside of its core functions?

Extensions to SSL, such as the SSL heartbeat extension (RFC 6520)... the one where the Heartbleed bug was found in. Other cipher suites would be another example.

The criticism here is that OpenSSL wasn't particularly choosy in which features of SSL (or other crypto in general) that it supported; it supported all of them, even if they were of more questionable utility.

The perils of being the reference implementation.

I'm glad there have been changes to the project. Heartbleed was certainly bad, but I personally never understood getting behind LibreSSL. Seeing one bad vulnerability from an established project and immediately jumping ship to a brand new one with less eyes and reputation seemed hasty to me.

As the LibreSSL devs have said, heartbleed was not the cause of the fork. They forked the project because they believed the OpenSSL project repeatedly made bad decisions.

From https://www.openbsd.org/papers/eurobsdcon2014-libressl.html

> Heartbleed can't even be considered the worst OpenSSL vuln. Previous bugs have resulted in remote code execution. Anybody remember the Slapper worm? That worm exploited an OpenSSL bug (which was apparently titled the SSLv2 client master key buffer overflow) which revealed not only encrypted data or your private key, but also gave up a remote shell on the server, and then it propogated itself. Yeah, I'd say that's worse. But no headlines.

> I mention this just to reinforce that LibreSSL is not the result of "the worst bug ever". I may call you dirty names, but I'm not going to fork your project on the basis of a missing bounds check.

> why fork

> Instead, libressl is here because of a tragic comedy of other errors. Let's start with the obvious. Why were heartbeats, a feature only useful for the DTLS protocol over UDP, built into the TLS protocol that runs over TCP? And why was this entirely useless feature enabled by default? Then there's some nonsense with the buffer allocator and freelists and exploit mitigation countermeasures, and we keep on digging and we keep on not liking what we're seeing. Bob's talk has all the gory details.

You can see "Bob's talk" here: https://www.openbsd.org/papers/bsdcan14-libressl/ though I think there is a YouTube video somewhere.

Also I would like to point out that Google also forked OpenSSL. So it seems that the LibreSSL folks are not the only ones who thought a fork was necessary.

Damn OpenSSL sounds a lot worse than I thought after reading those slides. The custom malloc, the function that allow you to jump anywhere in OpenSSL, the 17 layer deep IFDef, the dubious entropy that openSSL try to generate if the OS doesn't provide it, the bugs that sit in the issue tracker for years.

A lot of that uglyness seems to come from the fact that OpenSSL wants to support all environments (even DOS). I wonder why distributions haven't switched since LibreSSL was made to be API/ABI compatible with openSSL and target a POSIX OS. This would be much more justified than the ffmepg / libav thing imo.

>LibreSSL was made to be API/ABI compatible with openSSL and target a POSIX OS

LibreSSL is neither API compatible with newer OpenSSL versions, nor is it ABI compatible. In fact, they break ABI every six months. Furthermore LibreSSL upstream only targets OpenBSD, with the portable version existing as an afterthought.

The only linux distribution using LibreSSL is Void Linux (Alpine switched to OpenSSL some time ago). Even Void is considering switching to OpenSSL: https://github.com/void-linux/void-packages/issues/20935 .

Thank you ! That was interesting read. The main problems with LibreSSL are software compatibility since most software only build against OpenSSL and performance because the portable version doesn't include optimisations for other platforms than x86_64

The slides came out in 2014 so the API / ABI thing was probably true then but not anymore.

Maybe things would have been different if LibreSSL was backed by a major Linux distribution and OpenBSD. Even then Unix/Linux is not the only target of a lot of software and I doubt a lot of developer would have put the time to support both.

[Edit] I just saw in an other comment that LibreSSL is used in MacOSX and windows for openSSH. Maybe developers will consider it if it becomes available on major platforms

FYI Void recently enabled hardware acceleration for ARM and PowerPC architectures in LibreSSL.

Apple uses LibreSSL as I understand.

Yes, but it’s only for use by system libraries. The header files aren’t shipped, and applications should use their own copy rather than trying to use the system’s.

0. At least a couple did, but they were pretty niche-y. (Void was one, I think?)

1. It's a ton of packaging work even if the API/ABI were compatible (Calling it compatible is a bit of a stretch IMO).

2. One of the things LibreSSL removes is the FIPS validated stuff. Distributions that harbor ambitions of being used in large US corporate and government installations want that.

3. By the time the portable LibreSSL build system came out, there were already significant improvements afoot within the OpenSSL project.

I'm sure there are other reasons, but those are the big ones I'm aware of.

> 2. One of the things LibreSSL removes is the FIPS validated stuff. Distributions that harbor ambitions of being used in large US corporate and government installations want that.

Sounds like it could have happened if someone went to bat for it. Red Hat deciding to include it (even if they didn't replace OpenSSL with it immediately) and pushing to get it certified and the portability stuff more stable would have done this.

> 3. By the time the portable LibreSSL build system came out, there were already significant improvements afoot within the OpenSSL project.

That's probably the real reason. Although, given the stuff mentioned in that bug/talk and how much seems to have been based on extreme portability, unless OpenSSL decided to just give up on some aspects of that (I doubt it), then some of the problems (code complexity, not to mention ROP helpers) probably survive (not that I know).

> A lot of that uglyness seems to come from the fact that OpenSSL wants to support all environments (even DOS).

Someone correct me if I'm wrong, but I seem to remember either BoringSSL or LibreSSL (or both?) saying that their fork removed support for DOS because not only did almost nobody use it, but it didn't even work anyways.

Yes. One of the devs hacking on it chronicled his experiences on tumblr of all things if IIRC correctly.

> though I think there is a YouTube video somewhere

"LibreSSL with Bob Beck" at https://www.youtube.com/watch?v=GnBbhXBDmwU

Big-endian Support for AMD64 - the explanation from the person in the crowd is something of a horror story that I think a lot of have witnessed in various forms.


I seem to recall him giving a live feed of his findings while spelunking into the codebase.

I think I actually got grey hairs from that...

Then I realised that OpenSSL is quite entrenched, I'm still running it to this day, in production.

It wasn't about one bad vulnerability. Heartbleed was just the last straw.

OpenSSL implemented its own memory management system instead of using malloc. It would allocate one pool of memory and then manipulate into that. This meant that static analysis, runtime analysis, fuzzers were incapable of finding memory bugs. Because all pointers into that pool were "valid". LibreSSL stripped out OpenSSL's memory system and replaced it with malloc() and free() as provided by libc. Suddenly, static analysis, valgrind, and fuzzers found something like 3 dozen memory errors.

That is bad on so many levels.

OpenSSL supports big endian x86_64. OpenSSL supports EBDIC paths, certs, encodings, etc. OpenSSL supports DOS and Windows for Workgroups 3.11. It had a config -D NO_OLD_ASN1 and NO_ASN1_OLD. These defs performed different things. Having more eyeballs on a project didn't help OpenSSL, because all of those eyeballs glazed over immediately and/or had forks stabbed into them.

In something like the first month, the LibreSSL project deleted something like 90,000 LOC and 60,000 lines of comments/whitespace. Not changed: deleted. Removed useless cruft like mentioned above, and dangerous cruft like SSL 3.0 support. The latest OpenSSL tarball is 9.4MB, the latest LibreSSL tarball is 3.6MB. It has fewer eyeballs, sure, but it's an order of magnitude easier to audit.

OpenBSD also changed it's malloc implementation from sbrk to mmap, the net effect being that free returned memory to the OS far more often. It broke a lot of interesting stuff.

Custom allocators are (or were) common enough that Valgrind includes an API for them to use to tell the analysis engine where the valid memory blocks are:


I believe the state of the art in generic malloc implementations has improved since this sort of thing was commonplace, though.

> jumping ship to a brand new one with less eyes and reputation

That doesn't sound like an accurate description of LibreSSL, what with it being a part of OpenBSD.

This is a fair point, but it is being used as standard on other operating systems. I remember people switching to Void Linux en masse after they made it their standard mostly as a result of Heartbleed.

> but it is being used as standard on other operating systems

...so even more people are looking at it? not sure what problem you think is happening here.

Note that Windows ships with LibreSSL since a while too. (present in System32/LibreSSL on remotely modern Windows 10 installs, OpenSSH relies on it among other things)

MacOS Catalina 10.15.6

  /usr/bin/openssl version
  LibreSSL 2.8.3

Eyes doesn't make a project, but the development process

Read this (from older to newer posts) https://opensslrampage.org/ it explains a lot of the bad stuff on OpenSSL

Example: "“Do you really want to build OpenSSL for 16-bit Windows? Well, we don’t.”"


This made me think of BoringSSL and LibreSSL again.

Looking up on Wikipedia it seems that LibreSSL is focused on OpenBSD and removed lots of legacy code. BoringSSL (Google) got renamed to Tink but I couldn't not find much more.

It's sad to see that duplication of effort but it's also the force of open source

OpenSSL recently changed licenses to Apache 2.0, which is a unacceptable in some of the BSDs, which made the LibreSSL fork more valuable from a licensing perspective for them.

Tink isn't really a rename, but an API wrapper to prevent misuse that happens to use BoringSSL internally.

BoringSSL is still alive, Tink is something completely separate.

Google Tink is an encryption library providing a high level API for common cryptography problems. Such as authenticated encryption with associated data (AEAD). In a similar vein to libsodium, it's designed to not let you shoot yourself.

OpenSSL and it's forks expose the primitives directly, but as is the usual advice, don't roll your own crypto and say "We're using RSA 2048" because you copied an example from stack overflow without padding.

The paper points to data showing that OpenSSL is still the dominant SSL implementation on the net, so it's the one that matters.

If Windows is the dominant desktop computing platform, is it the only one the matters?

> It's sad to see that duplication of effort but it's also the force of open source

How's that a FOSS thing? Do you think there would be fewer TLS implementations running around if they were proprietary?

IIUC the point was that the ability to fork projects (and then potentially merge back) is one of the strengths of OSS, even if it means some duplicated effort.

The duplicated effort is needed. The OpenSSLs mono-cultur was one of the reason Heartbleed, and other OpenSSL security issues, where such huge problems and affected so many.

there is also https://www.wolfssl.com/

https://bearssl.org/ is kinda cool too.

For random reasons I can't read the full article but I wonder if they discuss the impact of LibreSSL on OpenSSL itself. Would anyone who moved to LibreSSL actually look back to OpenSSL today in 2020? Honest question as I'm not a crypto professional myself.

It's just a PDF link. The paper mentions the existence of LibreSSL but that is pretty much it; the focus of the paper is on the extensive efforts to improve OpenSSL.

OpenSSL recently passed a change in their vuln announcement policy to give a major firm, which everyone here knows I think, 7 days advance notice of any zero-day that they were made aware of.

This was the engineer who helped set up the new policy: https://awe.com

To be honest, maybe it's a good idea. It depends on how much support Huawei is willing to give OpenSSL.

In case anyone is curious:

> The OMC voted this week to update our security policy [1] to include the option of us giving prenotification to companies with which we have a commercial relationship. (Edited to clarify: the vote was to allow notification to our Premium Support customers and this does not include lower support levels, sponsors, or GitHub sponsors.)

* https://www.openssl.org/blog/blog/2020/05/12/security-prenot...

* https://www.openssl.org/support/contracts.html#premium

So, being realistic here, that means the Chinese government is given 7 days advance notice?

What's the current market share of OpenSSL vs LibreSSL vs alternatives?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact