
Heartbleed disclosure timeline: who knew what and when - dctrwatson
http://www.smh.com.au/it-pro/security-it/heartbleed-disclosure-timeline-who-knew-what-and-when-20140414-zqurk.html
======
mindstab
I still like how these timelines are all about the big tech companies and not
about governments, services and banks etc. These are arguably where some of
the biggest risk targets are actually at (see today's post about Canada
government's Revenue Services (tax agency) loosing people's info to
heartbleed).

We probably need a stronger web security system people can be on. Also some of
the blame falls to the big companies, banks, telcos etc themselves, I mean who
wants to report security flaws to Telcos in a world where they then turn
around and instead of giving you a $15k bounty, send you to jail like Weeve.
And ok that's not the best comparison because he went well beyond discovering
and did exploit as well. But none the less, some of the big corporate world
need to clue in and get more friendly with tech rather than hostile, because
right now they don't have a very good reputation and so then tech leaves them
out in the cold. Did anyone think about notifying telcos and banks much? or
just other big tech companies.

~~~
orky56
A report suggested that the NSA already knew about the Heartbleed bug 2 years
ago and took advantage ([http://www.bloomberg.com/news/2014-04-11/nsa-said-to-
have-us...](http://www.bloomberg.com/news/2014-04-11/nsa-said-to-have-used-
heartbleed-bug-exposing-consumers.html))

This suggests that first the NSA discovered it, then Neel Mehta, and finally
Codenomicon. As other commenters have rightly suggested, the government, or at
least an agency there of, uses information asymmetry as a weapon of
intelligence on its users (aka citizens). In the case of private corporations,
they have a more contractual duty to their users to ensure that these bugs are
patched so that those with malevolent intentions can't harm their users.

In my opinion, a bug that has been discovered but not shared is just as bad as
a weaponized biological agent. Each rely on secrecy to exploit a group that is
unaware.

~~~
thefreeman
That article contains exactly 0 facts. It is complete FUD. The only evidence
they have that the NSA knew about it is an unquoted statement that "two people
familiar with the matter said".

I am not saying it's impossible the NSA knew. But people throwing around that
article as evidence is absolutely laughable.

------
unreal37
Has anyone asked how two security researches supposedly found the same exact
bug (that's been in the code for 2+ years) within days of each other? How
likely is that?

~~~
tptacek
It happens routinely. Allow me to illustrate: on a pentest a couple years ago,
Vitaly McLain from our Chicago team found that if you accidentally slipped a
NUL into a header that nginx reverse-proxied, nginx would short-copy (it used
strncpy) and reveal server memory. We reported to our client, and in the time
it took for them to OK our upstream report --- a matter of hours --- someone
else reported the same bug.

What's funny about that is that the nginx bug is functionally identical to
Heartbleed (modulo that it only worked on nginx).

~~~
azakai
> It happens routinely.

I think that still leaves open the question of how likely it is, or perhaps
the question is better phrased as whether it is due to chance or not.

It is extremely unlikely to be due to chance in this case. It was out for
years, but discovered independently within days of each other.

One possibility is that it was discovered independently many times, but we
only find out about it after someone actually goes public. At that point,
those finding it earlier either never wanted to mention publicly that they
found it (or they already would have), or even if they want to, they would be
admitting to finding it earlier but not disclosing, so they'd look bad.

That leaves others finding it slightly after the group that goes public - soon
enough that it wasn't public yet, so they have evidence that they actually did
find it independently. That would lead to exactly the result we see here, in
fact. But this does imply that there were, very likely, multiple other groups
that found this earlier but never went public.

The only other option is that it wasn't random, but some event led to both
discoveries. Perhaps a hint was around, a new method of finding
vulnerabilities, or anything at all that could nudge people's minds in that
direction. That option is much less worrying, but very hard to prove.

~~~
tptacek
I think if you reread my comment closely, you'll see that it describes a
scenario that is _spookily_ similar to Heartbleed: same impact, same latent
bug (presumably in the code for many years, maybe even longer than TLS
heartbeats), and co-discovery not in a matter of days but _hours_.

Sorry, there's probably nothing interesting happening here, except for
coincidence.

~~~
azakai
Yes, it is spookily similar, I agree 100% (I did read it carefully ;)

What I disagree with is that one event happening means that another event very
similar to it is likely in a statistical sense.

Of course, it is an argument that supports that to _some_ extent. But it is
fairly weak support, when on the other hand statistics strongly imply the
opposite.

~~~
jballanc
> What I disagree with is that one event happening means that another event
> very similar to it is likely in a statistical sense.

Why? It makes perfect sense to me that, when one type of vulnerability is
discovered, many more of same type will be discovered very soon thereafter.
You have to consider that vulnerability discoveries don't happen in a vacuum.
There's a near-infinite number of attack routes that one could investigate,
but _which_ one you're looking at now is a product of the environment you
operate in.

For example, let's say you're investigating a web server. Then, some security
researcher demonstrates a flaw in an image codec where even using "safe"
memory copy functions in C leads to a vulnerability if tainted values are
passed in for the size parameters. You think, "Hmm...I'm not decoding images,
but web servers do copy memory. I should check to see if any memory copy
operations are using tainted values." Bam! You discover Heartbleed...but do
you honestly think you'd be the only researcher working on web servers that
saw the image codec demo and made that connection? Unlikely.

~~~
azakai
Certainly, yes - that would mean that this is not a coincidence.

I'm not arguing this is a coincidence. Just that if it was totally random, it
would be very unlikely. So the plausible possibilities are (1) what you
suggested, some common cause, or (2) that the discovery happened randomly
multiple times but was only disclosed once.

------
jdubs
"CloudFlare later boasts on its blog about how they were able to protect their
clients before many others." I wonder if some companies will boast to
potential new customers regarding their relationships with vendors that will
offer them advanced patches on critical issues such as heartbleed. Kudos to
their opportunistic marketing team, but I hope this trend does not continue.

~~~
tptacek
While I don't care so much who got notified first (any given embargo timeline
is going to frustrate large numbers of HN people; if that's a problem for you,
start finding bugs or post a bounty), I find several things not to like in
Cloudflare's marketing. Near as I can tell, Cloudflare is the origin of the
notion that TLS private keys weren't going to be in the heap near packet data,
a supposition they jumped to for no reason I can discern.

They didn't find the bug. They benefited from a heads-up on the bug. Then they
promoted a mistaken assumption about the bug, along with a gamified challenge
site that was in fact a poor vehicle for investigating nginx+OpenSSL (how
about, for instance, any decent debugger instead?).

Meanwhile, ops teams at big companies were in heated debates with security
about whether keys needed to roll.

There are good people working at Cloudflare and I'm not part of any outrage
battalion. I'm just not a fan of how they handled this particular incident.

~~~
sinak
Neel Mehta tweeted that private keys wouldn't be in the heap near packet data
on April 8th:

[https://twitter.com/neelmehta/status/453625474879471616](https://twitter.com/neelmehta/status/453625474879471616)

~~~
tptacek
I think I'm fine giving the guy who found the bug a pass for his 140 character
suggestion.

I'm not looking to collect pelts. I just think there was a better way to
address the question of private key exposure than "The Cloudflare Challenge".
In the future, maybe we can address serious questions with engineering instead
of marketing stunts; how about the "let's all work to instrument OpenSSL"
challenge?

~~~
daeken
My issue with the Cloudflare Challenge can be summed up in: no matter what the
results are, it will give people a diminished impression of the bug's actual
impact. I can't fathom any way in which the Cloudflare Challenge was
beneficial to the security of their customers (or anyone else, for that
matter), which should be the goal of bounties; not to simply be a PR move.

~~~
tptacek
My problem is that if you're trying to figure out how key material is
distributed throughout heap memory, asking people to answer that question
about an unknown private key through heartbleed "peeks" is about the most
obtuse possible way to find out.

------
skandl
> Facebook say after the disclosure: "We added protections for Facebook’s
> implementation of OpenSSL before this issue was publicly disclosed, and
> we're continuing to monitor the situation closely."

I wonder if Facebook found and fixed the bug locally beforehand? Makes me
think of all the OSS libraries we use, without bothering to do much more in
the way of static-analysis, etc besides simply making sure they compile.

------
protomyth
How did Facebook get prior word but not Instagram?

I worry that Amazon wasn't given prior notice according to the timeline.

~~~
euank
Amazon Web Services was explicitly listed in the group of companies at the
bottom that did not get notification.

~~~
Xorlev
It was a day after disclosure our ELBs were patched.

------
marcins
How does it happen that after being in the wild for two years the bug was
independently found by two different security researchers at roughly the same
time? I'm not trying to suggest a conspiracy or anything, just genuinely
curious how that works!

~~~
jrockway
Perhaps they read the same article or attended the same talk, and something in
the article or talk triggered a similar thought process.

~~~
supersystem
The Apple "goto fail", which affected TLS, and the GnuTLS bugs were widely
published just a couple of weeks earlier.

------
orky56
Clearly the need has presented itself for a tool/infrastructure that can
safely transmit a security flaw to the necessary parties without there being
some arbitrary pecking order as this situation has revealed. With torrents,
those who exploit something can quickly share usernames/passwords with the
world with little repercussion. The same effectiveness, speed, and risk should
also be associated with those who are trying to fix things.

------
rshm
I wonder instead of notifying to select few parties with an embargo, if it
would have been a better handled by releasing an encrypted sources with
documentation containing the url to high availability server containing keys
that serves only after pre-defined point in time. And documentation on
integrity verification, accessment of the source changes and implications on
other softwares using openssl.

~~~
tptacek
How would that not just be an overcomplicated way to do exactly what the
reporters of this bug did anyways?

~~~
ggggg5
I think the idea is that everyone would get the info at the exact same
instant. It also allows everyone to be "at their computer" ready to implement
the fix.

It would mitigate the possibility of it leaking and getting exploited by
someone.

~~~
kelnos
That's foolish, and doesn't take into account how software updates are
actually rolled out in the real world.

Many vendors will not just simply compile a new version of a library from
upstream source and just throw it on their machines. They depend on a tested
release from their distribution maintainer, or something along those lines.

Also many vendors aren't prepared to do a simple upgrade: some may have
customization they need to forward-port and test. Or perhaps they'd prefer to
backport the fix to their older version.

So basically, your "everyone gets info at once" means that blackhats can get
the information and exploit it almost immediately, while the good guys
scramble to -- much more slowly -- patch their systems.

------
ithinkso
I wonder how one should proceed (if not working for any of this big tech
companies) when one discover as critical bug as heartbleed?

~~~
asperous
It's only fair to publicly disclose immediately. You can't possibly alert
every trustworthy company on earth.

Now, if you want a bug bounty, you have to file a report and wait a certain
amount of time before you are allowed to disclose.

~~~
baby
You don't want to disclose it before releasing a patch.

------
ohwp
I doubt if the NSA did know about this bug. Because if they did they would
have put the economy at a huge risk.

~~~
AlexLord
How has all the other outed operations not put the economy at huge risk?

~~~
ohwp
Because most companies don't care enough. A lot of companies are still hosting
in the US.

But hackers overtaking banking for example might be a real risk.

------
johnvschmitt
There are many hackers who do not reveal their exploits.

This is an overly optimistic account of who knew what when.

And, how can we protect? Patch? Yes, that's necessary, but not sufficient. We
have so many other protocols we rely on that are fragile & intercepted. Self-
censorship sucks, and can't possible protect everything either. What to do?

~~~
lkd
This is not claiming to be a complete list of who knew what when. It is only a
list of confirmed cases.

~~~
johnvschmitt
Really? If there are caveats or disclaimers, they are very well hidden.

For most people reading this actual article, I think they will come away with
the impression that it's a complete account.

~~~
mcphilip
Was the article changed since you read it? The first three paragraphs make it
clear that this is an attempted reconstruction of events and
clarifications/corrections/ are requested.

>Ever since the "Heartbleed" flaw in encryption protocol OpenSSL was made
public on April 7 in the US there have been various questions about who knew
what and when.

>Fairfax Media has spoken to various people and groups involved and has
compiled the below timeline.

>If you have further information or corrections - especially information about
what occurred prior to March 21 at Google - please email the author:
bgrubb@fairfaxmedia.com.au. Click here for his PGP key.

~~~
johnvschmitt
Sorry, that just still looks like glossing it over.

I mean, when the NSA & other actors do NOT submit their data to this guy, they
can say, it's complete now?

It just stinks to me like it's complacency. Just change your passwords &
patch, & then don't worry, share everything again, it's private. In 2014, I
don't think we're safe & private anymore at all. I can take the downvotes. I
don't like it either.

