

Heartbleed Update v3 - mycodebreaks
https://blogs.akamai.com/2014/04/heartbleed-update-v3.html

======
Artemis2
Good politics: they had custom code for parts of OpenSSL, they published it
when it seemed needed, open-source community told them what was wrong, they
fixed it and apologized. No bullshit.

~~~
glimcat
Also: the way this played out makes an _awesome_ story for the next time
Akamai experiences internal pushback against participating in open source.
"Remember that one time it saved our bacon" is a political goldmine.

~~~
sobkas
They are only in problem now, because they have released the patch. If no one
outside the company saw that patch, they could spot the problem and fix it
without anyone noticing and creating a PR disaster. They probably wouldn't,
but they could. That is one argument someone could use against "participating
in open source".

It's a bit to late for saving bacon, patch should be publicly reviewed before
it went into production. So no bacon saved, but there is a hope for a smoke
detector.

~~~
ghshephard
If they had not released the patch, then it's entirely possible that black
hats would have been able to make off with the keys from their customers - and
_that_ would have been a real disaster.

------
beambot
Willem's letter to Akamai addressed three areas where their patch was likely
insufficient:

(1) First question is then, are 'p', 'q' and 'd' the only sensitive
parameters? NO. (The Chinese Remainder Theorem parameters are included in
OpenSSL private keys by default and they are enough to recover the private
key).

(2) Second question: Does OpenSSL ever copy the private key, or parts, after
the key has been read? YES.

(3) Third question: Does OpenSSL allocate temporary variables for
intermediates when performing operations with a private key. Can they be used
to recover the private key? YES and YES.

(Willem's post submitted to HN:
[https://news.ycombinator.com/item?id=7582990](https://news.ycombinator.com/item?id=7582990)
)

This update from Akamai only addresses (1), AFAICT.

~~~
csoandy
We are, in fact, evaluating those other claims, and testing them in our labs.
We are also evaluating claims made by other researchers, both publicly and
privately; and we hope to end up with both a better library out of this, as
well as a better understanding of the hazards we do and don't provide safeties
against.

Validating his first claim was sufficient to undermine our belief in key
safety; the others are therefore only relevant for protection against future
attacks. In that light, it was important to advise our customers and the
community at large.

~~~
sitkack
I think it is important to remember when validating flaws, that we show that
attacks are possible not they are necessarily demonstrated. Just because one
cannot construct an exploit doesn't mean that something is secure.

------
Yver
The part I still don't get because I haven't seen it discussed so far is that
their original patch mentionned they had been using a variant of the patch
"for a decade" but I never got why they didn't seek to get it merged upstream
a decade ago. Or why they released "a variant" of their patch instead of the
real thing to begin with.

------
chubot
What's interesting is that I just watched a video that strongly relates to
this problem last night (from DJB at CCC):

[https://www.youtube.com/watch?v=IuSnY_O8DqQ](https://www.youtube.com/watch?v=IuSnY_O8DqQ)

[http://facthacks.cr.yp.to/](http://facthacks.cr.yp.to/)

It is only part of the video, but they talk about searching Google and finding
partial private keys, e.g. ---- BEGIN PRIVATE KEY ---.

Even with text missing, it is trivial to recover the entire key, because of
redundancy that Akamai neglected to take into account. I never really thought
about how those ASCII strings are encoded, but it's actually the 6 or so
values encoded. If some are missing, you can recover the others.

In retrospect, why shouldn't the private keyh have RFC 822-like key value
format so this is obvious? I don't see why it has to be encoded at all; the
format is unnecessarily obscure.

The rest of this talk is also great. Most of it is accessible to non-
cryptographers (like me), as long as you understand basic arithmetic and have
an undergrad-level overview of public key crypto (which they give). They
provide very readable and runnable Sage scripts.

I really like the batch GCD attacks. Very elegant and effective.

~~~
jonhohle
Great talk, thanks for the link. I've used DJB's software for years, but this
was the first talk I've seen.

------
dsl
Right before this all came out our Akamai rep came on site and swore up and
down they were not vulnerable. They need to quickly communicate internally as
well that a vulnerability existed and send account managers back out to work
on high priority cert rollovers.

~~~
anonymouse01
Our rep said the same thing even when we insisted they rotate our certs on day
1.

Of course we're still waiting. I don't know who's decision it was to not
reissue all certs on day 1, but this is an epically bad decision bordering on
gross negligence.

Heads should roll.

------
wdr1
Interesting to note that if Akamai hadn't open sourced their modifications,
they never would have realized their certs actually had been compromised.

------
hrrsn
Good response from Akamai. No beating around the bush there.

~~~
daeken
No, this was not a good response. It was well written and took responsibility
for bad code, but it failed to address the elephant in the room: why were keys
not _immediately_ rotated?

I've said this a number of times lately, but it bears repeating that the
golden rule of security is: If a compromise occurs, assume that everything is
compromised. Even if their approach was tested to be perfect, they shouldn't
have made the assumption that that actually reflected reality.

They were extraordinarily irresponsible. There's simply no other way to slice
it.

~~~
blahedo
No, that goes too far the other way. If I have legitimate reason to believe
that an exploit does not affect me (especially if it's because I acknowledged
its possibility and programmed defensively against it), why would I
automatically act as if I were breached too?

Your argument is totally correct if I am simply unsure---if I don't know
whether an exploit affects me but I suspect it might, I assume the worst and
act accordingly. But in this case Akamai had good reason to believe they were
unaffected. (And, when someone demonstrated otherwise, they responded
responsibly.)

~~~
daeken
> If I have legitimate reason to believe that an exploit does not affect me
> (especially if it's because I acknowledged its possibility and programmed
> defensively against it), why would I automatically act as if I were breached
> too?

If you have a memory read out of a process that contains sensitive data, you
should _always_ assume that the sensitive data was breached, even if you
"programmed defensively against it". Period. No exceptions.

They had good reason to believe they were unaffected, which is why they
shouldn't have hit the "oh shit, the world is ending" button. But the decision
to not roll their keys in the face of a breach was a massively irresponsible
one. We have how many years of major bugs to look at? And in how many cases
are the likes of Akamai (that is, organizations with theoretical mitigations
in place) actually safe when they think they are?

Always, always assume your protections are faulty in a way that a motivated
attacker will figure out.

------
sjtgraham
I'm so grateful we have people like Willem Pinckaers uncovering these things.

------
joeybaker
This is the perfect example to point people to when they ask why the company
shouldn't just maintain an internal fork or write its own version of an open
source project.

Actually contributing the code you write is helps _everyone_.

------
sobkas
I would like to know how many security related patches they are keeping for
themselves? And I think not only Akamai did this. I hope that this situation
will prove that security needs a bit more patch reviews and a bit less of
secret sauce.

~~~
sweettea
I suspect their patches are oriented toward paranoia. Their particular patch
(keeping what they believed to be all critical values in a special region of
memory used only for critical things) had no obvious necessity -- it didn't
obviously provide more security -- but was just paranoia. Paranoia isn't
always necessary, or even desirable; hence, the remainder of the patches have
no obvious external utility, so they presumably haven't released them. <p>
They may also have performance improvements, conceivably, and those have a
more obvious rationale for keeping secret.

~~~
kevinr
As Andy tweeted, we built that particular patch to keep unencrypted secrets
off disk -- it actually was intended to provide tangible security benefit, it
wasn't just paranoia:
[https://twitter.com/csoandy/status/455307255895060480](https://twitter.com/csoandy/status/455307255895060480)

~~~
Serow225
If you folks have used this technique for years as I believe it has been
stated, any particular reason why it wasn't contributed back to OpenSSL
earlier?

~~~
kevinr
Licensing issues, primarily. See
[https://news.ycombinator.com/item?id=7581509](https://news.ycombinator.com/item?id=7581509)

~~~
Serow225
Thanks for the info! Sorry that I didn't reply back earlier, I missed seeing
that you had replied :)

------
Sami_Lehtinen
Why people are only talking about SSL keys? Because Heartbleed allowed to
fetch any data, that could have also included SSH keys and or any other
authentication credentials / information the server got, for itself or to
other services. As example if your site got Facebook API keys it's possible
that those have been leaking too.

~~~
jbg_
Unless your web server makes SSH connections from some kind of in-process
application, why would this bug allow SSH keys to leak?

~~~
jevinskie
Is memory wiped with zeros by malloc/free in libc or in the kernel? If not,
you're at risk.

~~~
bzbarsky
Yes, it is. Leaking memory contents across processes would be a pretty serious
bug in the kernel. So when your process asks the kernel for memory (via sbrk
or mmap) the memory it gets is zeroed out.

------
zoowar
I don't know akamai's relationship with open source projects, but it sounds
like they had local changes and tried to karma whore them as, "trust us, our
customers were protected, and we contribute to open source." I hope I'm wrong,
or they deserve the public shaming they are receiving. Ultimately, we need
more engagement from commercial organizations benefiting from open source
projects.

------
forrestthewoods
This does not inspire confidence in akamai for me. These days people on the
internet seem to get super excited when various companies write post mortems
on how they screwed up. Yay, they screwed up and were honest about it!!! I
find those to be wildly overrated. I'd rather there no be a screw-up to begin
with. And I don't particularly care if you come clean or not.

I know bugs are written and bugs happen. It's an unfortunate fact of life. And
anyone who posts their code will likely have holes poked in it by the
community. But the akamai thing really sets of warning flags. The fact they
thought it would work and it doesn't, at all, does not sit right.

