
Akamai release source to their custom secure_malloc for OpenSSL - nikcub
http://thread.gmane.org/gmane.comp.encryption.openssl.user/51243?resub=1
======
nikcub
For background, this came up as Akamai mentioned in their Heartbleed
announcement that they didn't think they had any sensitive information[0]
disclosed by the bug as they had wrapped OpenSSL's malloc to create two heaps:
a secure heap for sensitive information like keys and an ordinary heap:

[https://blogs.akamai.com/2014/04/heartbleed-
update.html](https://blogs.akamai.com/2014/04/heartbleed-update.html)

It prompted a lot of interest in their solution, which prompted them to
release the patch.

edit: [0] by "sensitive information" we mean keys, if I understand the
implementation correctly, you could still overrun into web server allocation
and get back HTTP headers.

~~~
gphilip
From reading that blog post it is not clear to me if their wrapping code
existed prior to the _discovery_ of the heartbleed bug: The post summary says:

 _Akamai patched the announced Heartbleed vulnerability prior to its public
announcement. We, like all users of OpenSSL, could have exposed passwords or
session cookies transiting our network from August 2012 through 4 April 2014.
Our custom memory allocator protected against nearly every circumstance by
which Heartbleed could have leaked SSL keys._

So: did the custom memory allocator exist already in August 2012? From reading
the post this looks to be the case. Could it be that someone at Akamai took a
look at the heartbeat (or other OpenSSL) code, decided that it could lead to
memory leaks, and wrote their own memory allocator wrapper code to guard
against this?

Edit: SSL -> OpenSSL

~~~
alexkus
From the original post:

> This patch is a variant of what we've been using to help protect customer
> keys for a decade.

So it protects keys against heartbleed, but not other HTTP related data (urls,
cookies, headers, etc).

------
nly
Why can't long-term private keys be protected, and used solely by, the kernel?

Linux CryptoAPI seems to have an asymmetric key interface[0], RSA signatures
are implemented[1], and there's even a x509/ASN1 parser[1]. Shouldn't this be
the default on Linux rather than furthering the NIH syndrome with crude
library-local allocators? In the very least, shouldn't there be a secure
malloc in glibc? If not, why doesn't OpenSSL use an existing secure malloc
library[2]?

[0]
[https://www.kernel.org/doc/Documentation/crypto/asymmetric-k...](https://www.kernel.org/doc/Documentation/crypto/asymmetric-
keys.txt)

[1]
[https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....](https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/crypto/asymmetric_keys?id=refs/tags/v3.14)

[2]
[http://www.jabberwocky.com/software/secmalloc/](http://www.jabberwocky.com/software/secmalloc/)

~~~
thefreeman
There have been a lot of posts about the custom malloc. From what i
understand, parts of OpenSSL actually depend on the buggy implementation
(specifically, the fact that you can the last thing free'd can be malloc'd
again).

------
jgrahamc
I would be interested to know if this actually protects against heartbleed in
the real world. I can see how it would keep the private key protected, but not
sure if it protects intermediate values used during calculation.

Given that it's likely that the private key can be derived from those parts,
it would be good to know if this custom allocator is 'enough' (i.e. are the
intermediate values stored in memory allocated using the same allocator)?

This week has been full of surprises. Has anyone tested this?

~~~
mrud
I didn't but akamai itself did [1]:

    
    
        To test this belief, our engineers have rebuilt 
        every Akamai configuration since August 2012, 
        when we upgraded to openssl-1.0.1. This included 
        every kernel, hardware, and edge server software  
        combination; and then a careful inspection of the ways 
        in which memory was allocated, to see if any 
        non-long-term memory allocations might border 
        on our secure heap. Most of the configurations 
        were proven safe; but we found one configuration 
        that was not - there was an available memory block in
        range of the secure key store.
    
        This less safe configuration was active on our network
        for nine days in March 2013
    

[1] [https://blogs.akamai.com/2014/04/heartbleed-
update.html](https://blogs.akamai.com/2014/04/heartbleed-update.html)

~~~
josephagoss
That is very impressive. I'm amazed they essentially keep a revision history
of their system configuration.

~~~
mrud
Yes, I'm also very impressed by this. It shows that they have their OPs in
total control and know what they are doing.

~~~
brians
Thanks!

As someone who grew up in the Akamai ops environment... How else could you get
anything done? I've taken this tool for granted since it was built in 2000. I
expect every planetary scale computing company to do this. Isn't it what
Heroku and similar git-push-to-deploy systems are supposed to get you?

------
bello
Disclaimer: I'm not too familiar with the code.

"This arena is mmap'd, with guard pages before and after so pointer over- and
under-runs won't wander into it."

Doesn't that mean that this will only protect against overreads of a certain
max length (such that the int16 length in heardbleed)? Seems like that
wouldn't help with a length defined as a bigger int. I wonder if there's any
better ways of doing this.

~~~
ramidarigaz
The beginning and end of the allocated section of memory (the guard pages they
refer to) are marked as PROT_NONE with mprotect, meaning that any access to
them will cause a segfault. It's possible that a misbehaving process could
jump straight into the unprotected memory, but it would have to not read from
the guard pages at all. Buffer overruns don't have that problem (since they
access memory sequentially), and would cause the program to crash before any
sensitive data could be read (assuming the overrun starts outside the
protected area).

~~~
bello
That makes sense, thank you!

------
herf
Guard pages address lots of issues with really small overhead, nice. But if
you know where to look they're not foolproof.

Wondering what the cost of putting each primitive (e.g., RSA private signing +
key) in a separate (child) process would be? Mailservers (qmail and postfix)
seem to do designs like this.

You could also imagine a page type that would only be readable to certain code
segments (would take CPU support to do it.)

~~~
whoopdedo
> You could also imagine a page type that would only be readable to certain
> code segments (would take CPU support to do it.)

Isn't this what TPM is supposed to provide? It received an unfavorable
reputation because of its association with DRM, but it addresses the same
problem.

------
brianpgordon
A CDN writing a custom malloc sounds kind of crazy, but I guess it paid off.

~~~
judk
An open source SSL library writing a custom malloc sounds kind of crazy, but I
guess, oh, it was crazy.

------
lobster_johnson
Here's a good blog article which looks at their patch:
[http://blog.nullspace.io/akamai-ssl-
patch.html](http://blog.nullspace.io/akamai-ssl-patch.html).

------
DevX101
If Akamai realized there was a major security vulnerability more than a year
ago why didn't they share this information with the OpenSSL team?

~~~
matthavener
I don't think they knew about heartbleed. This is stopgap to protect against
leaking private keys via memory bugs.

~~~
ams6110
I think it's a fair question. If they had what they believed to be an improved
(more secure) OpenSSL why not contribute the patch back to the community?
After all they are standing on the shoulders of giants here, it seems a bit
selfish to take an open-source project, improve it, and then not share that
back.

Yes I know that many open-source licenses do not obligate one to do this, but
it still seems like the right thing to do to me.

~~~
mzs
I can't say what happened in this case but after you submit a patch to openssl
and wait 6mo, a year, two, or even close to four, and simply don't hear
anything back or if you do that they are doing something their own way
instead, you just sort of lose the will and might get to simply be pragmatic
and do what you need for your own job and customers after a while.

~~~
richardwhiuk
Probably this. Submitting patches back to open source is expensive - you have
to dedicate engineers to tidying up and submitting patches, for no benefit
(other than being closer to upstream, which is of marginal use, especially if
there are clear forks) whereas they could be developing new functionality. In
the real world, the priority if often to do new development instead.

------
okasaki
If OpenSSL had been GPL licenced, this would have been public a long time ago.

~~~
zobzu
Actually as long as they don't distribute the code, even under GPL they don't
have to share it.

~~~
jpalomaki
GNU Affero license changes that: [https://www.gnu.org/licenses/why-affero-
gpl.html](https://www.gnu.org/licenses/why-affero-gpl.html)

"It [Affero GPL License] has one added requirement: if you run the program on
a server and let other users communicate with it there, your server must also
allow them to download the source code corresponding to the program that it's
running."

~~~
Dylan16807
I severely question the enforceability of the AGPL. If they're not
distributing they don't need to accept the license in the first place.

~~~
slashdotaccount
AGPL is based on modification rather than public performance or use. You don't
have permission to modify copyrighted things by default so if you do modify,
then you either comply with the AGPL or violate copyright law.

~~~
brianpgordon
You can modify copyrighted things, you just can't in general copy your
modified version.

~~~
msbarnett
Because a number of court cases have sided with the interpretation that
"copying bits into RAM for execution" is itself governed by copyright law, in
practice you wouldn't be able to use your modified software without breaking
copyright. This is why software licenses are "needed" \-- lacking some kind of
limited license to make copies, execution of software isn't permitted.

