
Cloudflare Reverse Proxies Are Dumping Uninitialized Memory - tptacek
https://bugs.chromium.org/p/project-zero/issues/detail?id=1139
======
tptacek
Oh, my god.

Read the whole event log.

If you were behind Cloudflare and it was proxying sensitive data (the contents
of HTTP POSTs, &c), they've potentially been spraying it into caches all
across the Internet; it was so bad that Tavis found it by accident just
looking through Google search results.

The crazy thing here is that the Project Zero people were joking last night
about a disclosure that was going to keep everyone at work late today. And,
this morning, Google announced the SHA-1 collision, which everyone (including
the insiders who leaked that the SHA-1 collision was coming) thought was the
big announcement.

Nope. A SHA-1 collision, it turns out, is the _minor_ security news of the
day.

This is approximately as bad as it ever gets. A significant number of
companies probably need to compose customer notifications; it's, at this
point, very difficult to rule out unauthorized disclosure of anything that
traversed Cloudflare.

~~~
espadrine
It is far from over, too! Google Cache still has loads of sensitive
information, a link away!

Look at this, click on the downward arrow, "Cached":
[https://www.google.com/search?q="CF-Host-Origin-
IP:"+"author...](https://www.google.com/search?q="CF-Host-Origin-
IP:"+"authorization:")

(And then, in Google Cache, "view source", search for "authorization".)

(Various combinations of HTTP headers to search for yield more results.)

~~~
toyg
Lol, Google just purged that search.

EDIT: but there's still plenty of fish:
[http://webcache.googleusercontent.com/search?q=cache:lw4K9G2...](http://webcache.googleusercontent.com/search?q=cache:lw4K9G2F1WgJ:lightnetwork.ph/ofw-
family-day-december-1/&num=1&hl=en&gl=uk&strip=0&vwsrc=1)

This will take weeks to clean, and that's just for Google.

EDIT2: found other oauth tokens, lots of fitbit calls... And this just by
searching for typical CF internal headers on Google and Bing. There is no way
to know what else is out there. What a mess.

~~~
camus2
Ouch, you really see everything :

> authorization: OAuth oauth_consumer_key ...

what a shit show. I'm sorry but at that point there must be consequences for
incompetence. Some might argue "But nobody can't do anything" ...

I'm sorry, CF has the money to to ditch C entirely and rewrite everything from
the ground up with a safer language, I don't care what it is, Go,Rust
whatever.

At that point people using C directly are playing with fire. C isn't a
language for highly distributed applications, it will only distribute memory
leaks ... With all the wealth there is in the whole Silicon Valley, trillions
of dollars, there is absolutely 0 effort to come up with an acceptable
solution? all these startups can't come together and say: "Ok,we're going to
design or choose a real safe language and stick to that"? where does all that
money goes then? Because this bug is going to cost A LOT OF MONEY to A LOT OF
PEOPLE.

~~~
dunham
These guys were probably saved by using OAuth - there is a consumer secret
(which the "_key" is just an identifier for) and an access token secret, both
of which are not sent over the wire. Just a signature based on them. (The
timestamp and nonce prevent replay attacks.)

OAuth2 "simplified" things and just sends the secret over the wire, trusting
SSL to keep things safe.

~~~
EGreg
Does this have anything to do with CloudFlare's ambitious attempt to be the
first service to proxy your https traffic to your users?

Perhaps the largest MITM ever eh?

------
_wmd
Step 1) MITM the entire Internet, undermining its SSL infrastructure, build a
business around it

Step 2) leak cleartext from said MITM'd connections to the entire Internet

I recently noted that in some ways Cloudflare are probably the only entity to
have ever managed to cause more damage to popular cryptography since the 2008
Debian OpenSSL bug (thanks to their "flexible" ""SSL"" """feature"""), but now
I'm certain of it.

"Trust us" doesn't fly any more, this simply isn't good enough. Sorry, you
lost my vote. Not even once

edit: why the revulsion? This bug would have been caught with valgrind, and by
the sounds of it, using nothing more complex than feeding their httpd a random
sampling of live inputs for an hour or two

~~~
bigiain
Step 0) Obtain black funding from NSA budget to start and "VC invest" in a
global CDN company...

(Now I'm trawling Crunchbase to see if I can work out which investors are NSA
front companies, then I'm gonna look to see what _else_ them and their
partners have invested in...)

~~~
nine_k
Covertly get into a company that terminates ssl for half the internet, and...
spill your precious secrets everywhere, instead of siphoning them off
silently?

~~~
jessaustin
Plausible deniability? "How could we have known the flaw was exploited by NSA
and FBI? We didn't know about the flaw at all!" When, actually, it was
_designed_ by NSA, before they created CF as an attack vector. Eventually the
vuln is discovered as was inevitable, but because the caches were
theoretically "public" no one notices all the drone strikes and parallel
constructions correlated with CF use.

I don't actually believe that, but it isn't an unreasonable theory.

------
jkells
My first thought was relief, thank god I'm not using Cloudflare.

Where would you even start to address this? Everything you've been serving is
potentially compromised, API keys, sessions, personal information, user
passwords, the works.

You've got no idea what has been leaked. Should you reset all your user
passwords, cycle all or your keys, notify all your customers that there data
may have been stolen?

My second thought after relief was the realization that even as a consumer I'm
affected by this, my password manager has > 100 entries what percentage of
them are using CloudFlare? Should I change all my passwords?

What an epic mess. This is the problem with centralization, the system is
broken.

~~~
joepie91_
> My second thought after relief was the realization that even as a consumer
> I'm affected by this, my password manager has > 100 entries what percentage
> of them are using CloudFlare? Should I change all my passwords?

Yes. Right now. Don't wait for the vendor to notify you.

> What an epic mess. This is the problem with centralization, the system is
> broken.

Yep.

~~~
bubblethink
How do you check if a website uses cloudflare ? Any scripts that do that ?

~~~
manigandham
Response headers will contain a "cf-ray" header or "server: cloudflare-nginx"

~~~
cancancan
Both should be there, as well as 'Set-Cookie: __cfduid=...'

    
    
      $ curl -I okcupid.com
      Set-Cookie: __cfduid=...
      Server: cloudflare-nginx
      CF-RAY: 335f033b77742b76-AMS
    

EDIT: Better yet, make that 'curl -IL domain.com' to follow redirects because
it may not show in the first response.

------
fagnerbrack
TL;DR for the lazy ones:

> The examples we're finding are so bad, I cancelled some weekend plans to go
> into the office on Sunday to help build some tools to cleanup. I've informed
> cloudflare what I'm working on. I'm finding private messages from major
> dating sites, full messages from a well-known chat service, online password
> manager data, frames from adult video sites, hotel bookings. We're talking
> full https requests, client IP addresses, full responses, cookies,
> passwords, keys, data, everything.

This is huge.

I mean, seriously, this is REALLY HUGE.

~~~
sparkling
I don't get it. How is this info leaked? From the blog posts, it seems that
"only" the HTTP Headers are being leaked and somehow being crawled by Google?
But since when does Google store HTTP request info? Can someone explain?

~~~
kalmi10
Headers (among other sensitive stuff) were being leaked inside document
bodies.

~~~
sparkling
So just to clarify: some bug makes Cloudflare leak the HTTP Headers into the
HTML being served and those HTML pages containing sensitive Info got cached by
Google (and others)?

~~~
chrisper
Yes. Think of it this way.

You have a function that strips all colons from your input. For some reason -
in certain cases - your code misbehaves and when you are replacing the colons
with an empty character you accidentally replace that colon with other data
you have in the memory. So now all the colons in your input have been replaced
with data that you shouldn't have touched. So now whoever sent you an input,
gets back that input + more data they shouldn't be able to see.

And Google in this case caches those output strings.

~~~
homero
But how is Google getting headers from the users of the sites, it should be
from their crawler

~~~
chrisper
If I (user A) access upwork.com (I just saw this on the list of affected
websites, so it's not meant to be an ad), I am sending them my headers. Let's
say my headers and other data are saved in M1 (memory register 1).

Then Google accesses the website as the crawler (user B), and their header and
data is saved in M2. However, Google triggered a bug and now has access to M1
as well. So now Google sees their own headers + my data + other garbage.

------
user5994461
> The greatest period of impact was from February 13 and February 18 with
> around 1 in every 3,300,000 HTTP requests through Cloudflare potentially
> resulting in memory leakage (that’s about 0.00003% of requests).

1) From the metrics I recalled when I interviewed there, and assuming the
given probability is correct, that means a potential of 100k-200k paged with
private data leaked every day.

2) What's the probably that a page is served to a cache engine? Not a clue.
Let's assume 1/1000.

3) That puts a bound around a hundred leaked pages saved per day into caches.

4) Do the cache only provide the latest version of a page? I think most do but
not all. Let's ignore that aspect.

5) What's the probably that a page contains private user information like auth
tokens? Maybe 1/10?

6) So, that's 10 pages saved per day into the internet search caches.

7) That's on par with their announcement: "With the help of Google, Yahoo,
Bing and others, we found 770 unique URIs that had been cached and which
contained leaked memory. Those 770 unique URIs covered 161 unique domains."
Well, not that we know for how long this was running.

8) Now, I don't want to downplay the issue, but leaking an dozen tokens per
day is not that much of a disaster. Sure it's bad, but it's not remotely close
to the leak of the millennia and it's certainly not internet scale leak.

9) For the record, CloudFlare serves over one BILLION human beings. Given the
tone and the drama I expected way more data from this leak. This is a huge
disappointment.

Happy Ending: You were probably not affected.

~~~
mattbee
This assumes that the Bad Guys hadn't noticed the bug before Tavis, and hadn't
started intensively mining Cloudflare for data.

~~~
user5994461
Intensive mining indeed, if it's true that it requires 3.3M requests to get a
page leak.

With a fixed 100Mbps connection and assuming 2kB per HTTP request-response,
you can hope to get one leak every 11 minutes and 6.6GB of traffic, which is a
constant 5k requests/s.

Maybe if Google reassigns all its SHAterred ressources to doing that...

... and then I realize that we were talking about cloudflare and my mining bot
a capcha.

\---

edit: correction. The bug was affecting only some pages with some content
filtering options enabled, and was more prominent under some specific
circumstances.

Hence why it only happens 1/3.3M in average. An attacker could allegedly leak
data much more reliably if he was able to identify the patterns that are more
likely to trigger leaks.

~~~
gelatocar
Couldn't an attacker construct a page that triggers the memory leak and just
keep accessing that page to get different pieces of memory?

~~~
brians
Yes. Sign up for service, configure a page with crafted invalid HTML at your
origin, activate all three buggy features, and spam it with requests.

If you can find such a page already, just jump to the last step and avoid
signing your work.

------
spydum
People are going to lambast CF for downplaying the impact, and there could be
merit in that.

However, I really want to say I am absolutely impressed with both Project Zero
AND Cloudflare on so many fronts, from clarity of communication, to
collaboration, and rapid response. So many other organizations would have
absolutely tanked when presented with this problem. Huge kudos for CF guys
understanding the severity and aligning resources to make the fixes.

In terms of P0 and Tavis though, holy crap. Where the heck would we be without
these guys? Truly inspiring !

~~~
Kalium
CF's infosec team is very, very good at their jobs.

~~~
jrcii
Obviously not, right?

~~~
Kalium
They're human too. Look at the response times!

~~~
oldsj
Yea but... seems like a quick run of valgrind would have caught this

~~~
ars
Not necessarily.

If they just keep reusing a buffer and forget to clear it in between requests
there is nothing automated that would find it.

Bounds checking languages would not help either - they would only work if they
delete and reallocate the buffer on each request, since that's slow it's
unlikely anyone would do that.

They probably wouldn't even clear the buffer, instead they rely on keeping
track of the length of data in it, so any errors in there would be a problem.

~~~
robryk
If they used asan/msan and its support for manually marking regions of memory
as invalid/uninitialized, that could have caught such cases too.

------
dantiberian
From Twitter:

"@taviso their post-mortem indicates this would've been exploitable only 4
days prior to your initial contact. Is that info invalid?" \-
[https://twitter.com/pmoust/status/834916647873961984](https://twitter.com/pmoust/status/834916647873961984)

"@pmoust Yes, they worded it confusingly. It was exploitable for months, we
have the cached data." \-
[https://twitter.com/taviso/status/834918182640996353](https://twitter.com/taviso/status/834918182640996353)

~~~
jgrahamc
From my blog on this:

    
    
        The three features implicated were rolled out as follows. 
        The earliest date memory could have leaked is 2016-09-22.
    
        2016-09-22 Automatic HTTP Rewrites enabled 
        2017-01-30 Server-Side Excludes migrated to new parser 
        2017-02-13 Email Obfuscation partially migrated to new parser 
        2017-02-18 Google reports problem to Cloudflare and leak is stopped

~~~
benwilber0
Well fuck. I have no idea what (if any, or all) of my authenticated web
sessions have been going through CloudFlare in the last 6 months. How do I
even start to protect myself from this?

~~~
sarcasmic
1\. rotate passwords, tokens, auth stuff on any and all service you use that
may have used CloudFlare in this time period (as of time of writing this list
has not been enumerated)

2\. hope that no personally-identifiable info or damaging plaintext that can
be tied back to you has been exposed, but you will probably never know for
sure

3\. join class action lawsuits if you so desire and receive the chump change
that is your share once they inevitably get settled

4\. ponder what it truly means to willingly (or unknowingly) give information
to or through a "trusted third-party" who may employ other "trusted third-
parties"

5\. languish in unsatisfactory answers and outcomes, return to step 2.

~~~
nikisweeting
I've compiled a list of 7 million+ domains that use Cloudflare here:
[https://github.com/pirate/sites-using-
cloudflare](https://github.com/pirate/sites-using-cloudflare)

Including the subset of the Alexa 10,000 that use Cloudflare in the README.

~~~
ndemoor
Here is also a non-exhaustive list of websites using cloudflare:
[https://index.woorank.com/en/reviews?technology=cloudflare](https://index.woorank.com/en/reviews?technology=cloudflare)

------
Xorlev
> One of the advantages of being a service is that bugs can go from reported
> to fixed in minutes to hours instead of months. The industry standard time
> allowed to deploy a fix for a bug like this is usually three months; we were
> completely finished globally in under 7 hours with an initial mitigation in
> 47 minutes.

Great, that makes me feel so much better! I'm sorry, don't try to put a cherry
on the top when you've just leaked PII and encrypted communications.

Additionally, most vendors in the industry aren't deployed in front of quite
as much traffic as CloudFlare is. It's a miracle that ProjectZero managed to
find the issue.

------
CapacitorSet
>Cloudflare pointed out their bug bounty program, but I noticed it has a top-
tier reward of a t-shirt.

Considering the amount and sensitivity of the data they handle, I'm not sure a
t-shirt is an appropriate top-tier reward.

~~~
minxomat
Not only that, but the "reward" in the program is laughable and frankly
insulting to any serious researcher considering the scope of CF. Bug bounty
platforms are already becoming the fiverr of ITSEC (that's not a good thing),
CF just made an extra effort do diminish the value for researchers.

Management: "Why do we offer $5k for a small bug again? Look at CF, they don't
offer any money!"

~~~
eli
If serious researchers are looking to get paid, I think bug bounties are the
wrong approach entirely

~~~
mabbo
It's about payoff * probability.

Let's say I (an idiot, but knowledgeable enough) stumble upon a serious
vulnerability in Google.

Option 1: I could try to sell that on a darknet market for a decent amount of
money. State actors, hacker groups, lots of people want to pay for such things
to exploit. But, I might not get paid very much, I might get screwed over, I
might go to jail, who the heck knows, I'm playing with a bit of fire here.
Could make a good pay day though.

Option 2: Google offers a bug bounty that is known to pay well. It probably
offers guidance on how much my exploit is worth. They'll almost certainly pay.
And hey, no one gets exploited, which most people feel is a good thing.

Value = payout * probability. If bug bounties pay well, option 2 has a higher
value most of the time. But if a company offers t-shirts, or is known for
screwing over the discoverer, the perceived value falls quickly.

That's why companies who take security seriously pay good bounties, loudly and
publicly.

~~~
flukus
> I might go to jail

Is selling exploits illegal? If so is selling them to google also illegal?

~~~
SteveNuts
You're not so much selling them to google, you're disclosing them.

It's more of a contractual agreement between you and Google, or whatever
company you're reporting the vulnerability to.

As long as you follow the rules for their bug bounty, you'll be fine.

------
kyledrake
Friendly reminder that Cloudflare willingly hosts the top DDoS-for-hire attack
sites, and refuses to take them down when they are reported.

Run WHOIS on them, it's almost 100% behind Cloudflare:
[https://www.google.com/#q=ddos+booter](https://www.google.com/#q=ddos+booter)

I would be less concerned about the fact that Cloudflare is spraying private
data all over the internet if people weren't being coerced into it by a
racket.

We won't have a decentralized web anymore if this keeps going. The entire
internet will sit behind a few big CDNs and spray private data through bugs
and FISA court wire taps. God help us all if this happens.

~~~
chipperyman573
>Friendly reminder that Cloudflare willingly hosts the top DDoS-for-hire
attack sites, and refuses to take them down when they are reported.

Why should CF be required to police the internet? CF doesn't even host them,
they just protect their sites from DDoS and DNS.

~~~
kyledrake
Cloudflare has spent a lot of time gaslighting people into believing this, but
it physically, scientifically, OSI model-y isn't true. Cloudflare _hosts web
sites_. When Cloudflare CDN edges that content, that content exists on _their_
servers. Just because the canonical store is on another machine doesn't mean
they don't host the site. If I mirror a site from some other server, and
you're loading that site from my server, I'm the one hosting that site. That's
how HTTP works.

The argument that they don't know what's hosted on their network has also been
demonstrated by evidence as nonsense. The reason the Pirate Bay got blackholed
by Cogent last week was because Cloudflare was grouping all of the BitTorrent
sites on their network onto a single IP address, and a Spanish court order
related to a different site ended up BGP blackholing over two dozen torrent-
related sites as collateral damage.

[http://seclists.org/nanog/2016/Jul/400](http://seclists.org/nanog/2016/Jul/400)
[https://mailman.nanog.org/pipermail/nanog/2017-February/thre...](https://mailman.nanog.org/pipermail/nanog/2017-February/thread.html#90144)

Cloudflare is completely capable of enforcing this, yet they don't do anything
about it. It benefits them financially to not do anything, because they get
business from these DDoS attackers trashing other networks on the internet,
making it so you can only have sites stay up if they are hosted by
Cloudflare's broken, bleeding servers.

This is fundamentally an extortion racket. Frankly, it should be a crime. This
is _exactly_ the kind of problem laws exist for.

~~~
Blahah
It's not the responsibility of anyone except the police to police those sites.
Cloudflare aren't providing those attack sites with an attack vector, they are
just serving their webpages. The post office isn't responsible for policing
blackmail letters sent through the mail.

~~~
kyledrake
The theory that Cloudflare only enforces against sites they receive court
orders for is yet another argument that is not backed by evidence. They
actively take down phishing attacks, without warrants or court orders.
Presumably because if they didn't, Google would shitlist them in pagerank.
They behave responsibly and morally when it benefits them financially, and
tell everyone they need court orders when it doesn't, even if that decision
hurts the web.

It is everyone's responsibility to be responsible members of the internet
community. Just because they've found a temporary legal loophole does not give
them a moral blank check to be complicit in the murder of the Internet's
ability to function.

~~~
Dylan16807
The morality of hosting the sites of jerks is not nearly as objective as you
claim. I could make an argument that they behave morally by treating everyone
equally, but they make an exception and perform immorally with phishing sites
because google would punish them.

But the real answer is a lot simpler. The DDoS sites are not doing the DDoS
through cloudflare. The phishing sites are doing the phishing through
cloudflare.

And exposing some DDoS sites to DDoS is not going to fix the root problem.
People will still sell DDoS services, and people will still put insecure
devices online to become part of botnets.

------
AYBABTME
This comes around to me as something that just shouldn't have happened.
CloudFlare are pretty big on Go, as far as I can tell (and I guess Lua for
scripting nginx). Why was this parsing package written in a non memory-safe
language? Parsing is one of those "obvious" things easy to mess up; the
likelihood of a custom, hand written parser being buggy is pretty high. If
it's somehow understood that your library is likely to have bugs, why do it in
C/C++, where bugs often lead to bleeding memory? In a shop that's already
fluent in Go, where they have the institutional knowledge to do it safely?
Sure performance is not going to be the same, but with some care it'll get
pretty close.

Sorry I hate to just be a coach commentator. Obviously hindsight is 20/20\.
Still I think there's a lesson here.

~~~
ars
This could easily happen in Go as well. All that would be needed is to reuse
the buffer in between requests, and rely on the buffer length instead of
clearing it.

To make it safer you would need to deallocate and reallocate the buffer for
each request, but that might be slow. Doing that would fix it for Go, or for
C, it would be the same either way.

So I'm not convinced that using Go would have helped here.

~~~
zzzcpan
"This could easily happen in Go as well."

Not really true. Go operates on slices that panic on out-of-bounds accesses.
So, for this to happen in Go you would have to reinvent slices and use a lot
of manual C-style code to operate on them, which literally nobody does in Go,
because it's too hard.

~~~
j_s
Recycling memory buffers, like CloudFlare does?

[https://blog.cloudflare.com/recycling-memory-buffers-in-
go/](https://blog.cloudflare.com/recycling-memory-buffers-in-go/)

------
nikisweeting
I've compiled a list of 7,385,121 domains served through cloudflare using
several scrapers. [https://github.com/pirate/sites-using-
cloudflare](https://github.com/pirate/sites-using-cloudflare)

The full list is available for download here (23mb)
[https://github.com/pirate/sites-using-
cloudflare/raw/master/...](https://github.com/pirate/sites-using-
cloudflare/raw/master/sorted_unique.zip)

I will be updating it as I find more domains.

~~~
jkells
More than 7 million domains... Letting that sink in...

I'm assuming this list is based on DNS records? I wonder what proportion of
those offloaded their SSL to Cloudflare.

~~~
nikisweeting
I had duplicates, it's actually only 4,287,625 (still a lot though).

Fixed the duplicates: [https://github.com/pirate/sites-using-
cloudflare/raw/master/...](https://github.com/pirate/sites-using-
cloudflare/raw/master/sorted_unique.zip)

------
mattbee
Cloudflare isn't just a security hole in the middle of the internet, they're a
protection racket.

If you wanted to pay to DDoS a site, search for "booter" and you'll get a list
of sites that will take another site off the internet for money with a flood
of traffic.

quezstresser.com webstresser.co topbooter.co instabooter.com booter.xyz
critical-boot.com top10booters.com betabooter.com databooter.com

etc. etc. - from the first 30 results I could find 2 booter sites that _weren
't_ hosted by Cloudflare.

But hey, pay Cloudflare and your site too can be safe from DDoS attacks...

~~~
sfeng
You are essentially arguing against freedom of speech. Cloudflare will protect
any site that doesn't host child porn. Yes that includes things which you
don't like, but it also includes all the things you do.

~~~
abeyer
> _Cloudflare will protect any site that doesn 't host child porn_

Doesn't that make it worse? They aren't saying they don't or won't police the
content they protect. They are obviously capable and willing to draw a line on
ethical or legal grounds, if they have done so in that case. They have just
chosen to draw that line on one side of porn but another side of DDoS
services.

Ultimately it is their decision to make, but I don't think it's unfair for
people to question possible conflicts of interest in how that decision is
made.

~~~
Trundle
Why are you combining legal and ethical? They're capable and willing to draw a
line on legal grounds. Seems pretty clear.

~~~
abeyer
Not combining, that's why I said _or_.

And I said that because I'm not sure why they've made that decision...it could
have been either or both. And sale of DDoS service is arguably illegal in at
least some places, so they obviously aren't rejecting _all_ illegal content.

------
dmitrygr
Cloudflare's announcement, as it is currently worded, deserves the
understatement-of-the-centry award.

~~~
danielweber
"Don't worry, the keys weren't compromised."

I know how to replace my TLS keys. I have no idea how to replace everything
else.

It's like people who think losing my credit card number is the worst thing.
No, it can be a hassle, but once I replace it I'm okay. It's everything else.

~~~
fulafel
The implied comparison to Heartbleed problem is that everyone's old encrypted
traffic was suddenly in the open, key change didn't help.

(except for the enlightened few who used PFS before Heartbleed)

------
rdl
Neither this thread nor the Cloudflare blog post include concise steps for
customers who were exposed.

There's an argument for changing secrets (user passwords, API keys, etc.) for
potentially affected sites, plus of course investigating logs for any
anomalous activity. It would be nice if there were a guide for affected users,
maybe a supplemental blog post.

(and yet again: thank you Google for Project Zero!)

~~~
Kalium
Right there with you. I'm currently scrambling for remediation ideas. "Change
everything" isn't tractable.

~~~
garblegarble
>I'm currently scrambling for remediation ideas. "Change everything" isn't
tractable.

It's not easy to deal with but it is the best remediation available to you,
given the exceptionally broad scope and months-long period where data was
apparently leaking (the cloudflare blog post lists 2016-09-22 as the first
date when leaks were possible)

~~~
jstanley
Change my name? Change my address? Change my date of birth? My mother's maiden
name? My passport number?

It's simply not possible to change all of the sensitive information that might
have been leaked.

~~~
rdl
I think I've settled on "change admin passwords, change any m2m auth
credentials which don't require user intervention (API keys in apps, etc.
should be rolled regularly anyway)"

Forcing individual end users to change their passwords is probably a net-
negative. I might prioritize it if I have OTHER security improvements to roll
out soon, though (2FA, upgrading auth infrastructure, other potential
compromise, etc.).

I don't think anything else is really viable.

Bitcoin addresses/keys which transited Cloudflare probably should be updated,
though, on the extremely off chance.

~~~
Kalium
Yeah. I'm changing all the passwords I can get my employer to go for and
upgrading auth infrastructure for the rest.

------
vermontdevil
I got an email from Cloudflare and here's an excerpt about the # of sites
affected by this.

Not sure what to make of it - the low number of domains affected.

====================================

 _In our review of these third party caches, we discovered data that had been
exposed from approximately 150 of Cloudflare 's customers across our Free,
Pro, Business, and Enterprise plans. We have reached out to these customers
directly to provide them with a copy of the data that was exposed, help them
understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered
exposed data in any third party caches. The bug has been patched so it is no
longer leaking data. However, we continue to work with these caches to review
their records and help them purge any exposed data we find. If we discover any
data leaked about your domains during this search, we will reach out to you
directly and provide you full details of what we have found._

~~~
problems
Yeah, I got this this morning too. It seems to be a pretty big downplay - it
should be closer to "change all your passwords, have all your customers change
all their passwords". They're busy shredding data from caches, but anyone
scraping cloudflare sites in recent days might have data around that they'll
never know about.

But I don't blame them entirely - it's unlikely this will have been used and
unlikely a given customer's data would be present, so it'd induce panic which
would probably never have resulted in an attack.

------
the_common_man
How does such a simple bug not get picked by auto tests, ci or end to end
tests? I am baffled. Since we are behind cloudflare, I am not sure what I
should tell my manager now. I lack the technical know how to parse that
extremely technical article. Are we supposed to just assume all our traffic
that passed via cloudflare is possibly compromised?

It's also a bit sad that travis has to contact cloudflare by twitter.
Seriousy?

Edit:
[https://twitter.com/taviso/status/832744397800214528](https://twitter.com/taviso/status/832744397800214528)
is the tweet in question

~~~
rdl
I don't think he _had_ to, but he got an answer in minutes. I don't think
that's the part to be worried about.

As for what you should do: it sounds like the impact is relatively low. I'd
personally change easily-changed secrets which go over the session, and
potentially externally facing customer passwords (yes in enterprise, maybe not
in consumer).

(I don't have any insider info on this breach, though, but I read both posts
and know how the system works.)

~~~
tyingq
Sounds bad to me...

 _" We've discovered (and purged) cached pages that contain private messages
from well-known services, PII from major sites that use cloudflare, and even
plaintext API requests from a popular password manager that were sent over
https (!!)."_

The trouble is you have no way to know if someone discovered this earlier, and
harvested info for a long time.

Or, how much harvested info from your site might be in a Google cache for
someone else's site.

~~~
rdl
Does 1Password really send anything meaningful in their API queries, or is it
encrypted separately and then just sent over HTTPS?

~~~
fjarlq
For what it's worth, I've posted this question in 1Password's support forum,
which is frequented by 1Password staff:
[https://discussions.agilebits.com/discussion/75711/cloudblee...](https://discussions.agilebits.com/discussion/75711/cloudbleed-
cloudflare-cdns-have-been-leaking-sensitive-1password-data)

~~~
zaatar
1Password has said via their blog that _nothing_ was compromised whatsoever:
[https://blog.agilebits.com/2017/02/23/three-layers-of-
encryp...](https://blog.agilebits.com/2017/02/23/three-layers-of-encryption-
keeps-you-safe-when-ssltls-fails/)

More details are promised in the coming days.

------
DannyBee
So, does the t-shirt say: "I found a zero-day bug in cloudflare and all i got
was this lousy

X-Uber-token:

X-Uber-latitude:

... "

~~~
leni536
"GO EASY ON THE BUGS, leave some for the next person."

------
DangerousPie
Has anybody else actually received an email from Cloudflare about this? I'm a
paying customer, but haven't heard anything from them yet. I hope they don't
expect they can leave it at a random blog post that will go by unnoticed?

~~~
tonyztan
Another paying customer here. No email communication from CloudFlare. Found
out about this on HN.

~~~
ndesaulniers
You and OP should contact them, as customers.

------
thurston
Author of Ragel here.

An experienced Ragel programmer would know that when you start setting the EOF
pointer you are enabling code paths that never executed before. Like,
potentially buggy ones. Eek!

~~~
mcintyre1994
I'm not sure if you've had a chance to look at the Cloudflare blog post yet
([https://blog.cloudflare.com/incident-report-on-memory-
leak-c...](https://blog.cloudflare.com/incident-report-on-memory-leak-caused-
by-cloudflare-parser-bug/)), but while they take full responsibility they do
point out (under root cause of the bug) that a generated equality check could
be >= instead which would avoid the bug. Obviously GIGO applies and it's their
bug, but it might be worth seeing if there's anything you can do on Ragel
side?

~~~
thurston
Well doing that would mean ragel would incorrectly read one character, rather
than run off forever. Personally I'd rather have the latter. Much easier to
catch with memory checkers. Eventually you try to read some thing you're not
allowed to read, or blow something else up, instead of just read the first
byte of the int following the buffer, or whatever.

There would have to be an additional bounds check when issuing a goto in an
error action, but doing that is contrary to the simple execution model that
ragel users have come to rely on.

Gotta ask the question, where was the testing when they altered 7 year old
code without the involvement of the original developer?

------
chm
Some important parts:

    
    
        The examples we're finding are so bad, I cancelled some
        weekend plans to go into the office on Sunday to help
        build some tools to cleanup. I've informed cloudflare
        what I'm working on. I'm finding private messages from
        major dating sites, full messages from a well-known
        chat service, online password manager data, frames from
        adult video sites, hotel bookings. We're talking full
        https requests, client IP addresses, full responses,
        cookies, passwords, keys, data, everything.
    
        Cloudflare pointed out their bug bounty program, but I
        noticed it has a top-tier reward of a t-shirt.
    
        Cloudflare did finally send me a draft. It contains an  
        excellent postmortem, but severely downplays the risk
        to customers.

~~~
rrdharan
Connecting some dots, I'm wondering if the "well-known chat service" is Slack:

[http://www.computing.co.uk/ctg/news/2462266/whatsapp-
reddit-...](http://www.computing.co.uk/ctg/news/2462266/whatsapp-reddit-and-
slack-knocked-offline-because-of-cloudflare-problems)

~~~
Deimorz
I'm fairly sure that it's Discord.

~~~
PuffinBlue
Yes, I found some leaked data referencing Discord still in Google's cache so
I'd say it's them.

~~~
b1naryth1ef
Mind emailing me some details? az@discordapp

~~~
PuffinBlue
I didn't keep details, sorry. It was late (UK time) and I was attempting to
get my own response out the door.

I saw three domains directly myself with compromised details:

android-cdn-api.fitbit.com

iphone-cdn-client.fitbit.com

api-v2launch.trakt.tv

I saw data relating to Discord whilst on various cached pages when I was
looking at the above domains.

The pages are no longer available in Google's cache so I can't link to them.

Some cached pages had data from multiple sites all together, it was a mess.

~~~
b1naryth1ef
No worries, thanks for the response anyway!

------
jgrahamc
Full details from Cloudflare: [https://blog.cloudflare.com/incident-report-on-
memory-leak-c...](https://blog.cloudflare.com/incident-report-on-memory-leak-
caused-by-cloudflare-parser-bug/)

~~~
HappyTypist
Why is your company severely downplaying it?

Honestly, this is the biggest security incident in a long time, and proper
mitigation would probably warrant:

\- forcefully terminating all cookies on CloudFlare sites, cloudflare already
injects JS onto the page anyway

\- MITMing all CloudFlare sites with a warning for users to change their
passwords

~~~
xuki
> MITMing all CloudFlare sites with a warning for users to change their
> passwords

REALLY?

------
bartkappenburg
From a cloudflare employee:

"We were working to disclose the bug as quickly as possible, but wanted to
clean up search engine caches before it became public because we felt we had a
duty of care to ensure that this private information was removed from public
view. We were comfortable that we had time as Google Project Zero initially
gave us a 90 day disclosure window (as can still be seen in their incident
tracker), however after a couple of days, they informed us that they felt that
7 days was more appropriate. Google Project Zero ended up disclosing this
information after only 6 days."

~~~
ricardobeat
Straight from the issue tracker:

    
    
        They then told me Wednesday, but in a later reply started saying Thursday
        [...] If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks.
    
        https://security.googleblog.com/2013/05/disclosure-timeline-for-vulnerabilities.html

------
joepie91_
This is probably a good moment to recall the article I published a while ago
about how CloudFlare is actively putting the web at risk:
[http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-
hav...](http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-have-a-
problem/)

This is precisely why. The only thing that surprises me about this, is that it
was an accidental disclosure rather than a breach. Other than that, this was
_completely to be expected_.

EDIT: Also, this can't be repeated enough: EVERYBODY IS AFFECTED. Change your
passwords, everywhere, right now. Don't wait for vendors to notify you.

Anything could have irrevocably leaked, and you have no way of knowing for
sure, so assume the worst.

~~~
sparkling
Related: [http://crimeflare.com/](http://crimeflare.com/)

~~~
WillyOnWheels
Just looking at that site... what's so bad about Wikipedia? There's a lot to
criticize about Wikipedia, but I've never heard of them violating someone's
privacy.

------
Tiksi
> Many of the logged urls contained query strings from https requests that I
> don't think they intended to share.

I guess this confirms a few things.

\- The complete query strings are logged,

\- They don't appear to be too concerned with who accesses the logs internally
or have a process that limits the access, and

\- They're willing to send those logs out to a random person.

~~~
jgrahamc
This has nothing to do with logging.

~~~
Tiksi
The quoted part that specifically mentions logged urls containing query
strings has nothing to do with logging?

~~~
jgrahamc
That's Google logging stuff

~~~
geofft
That's not how Google tells it, if I'm reading this right:

 _Cloudflare explained that they pushed a change to production that logged
malformed pages that were requested, and then sent me the list of URLs to
double check._

 _Many of the logged urls contained query strings from https requests that I
don 't think they intended to share._

(I'm reading that as "intended to share with Google".)

~~~
jgrahamc
Ah. I see what you mean. Apologies, kind of tired.

~~~
Kalium
Understandably, I think. I can't imagine you've had much sleep this week.

~~~
jgrahamc
Little

~~~
shock
I'm sorry you (or anybody else) have to go through this.

~~~
jgrahamc
Thanks.

~~~
shock
FWIW, I admire you for how you're handling this, taking the grunt of the storm
all by yourself and not throwing anyone under the bus. Tip of the hat to you
and may better times lie ahead!

~~~
jgrahamc
No one else's mistake other than ours.

------
steven_pack
If only there were a systems programming language, offering c-like performance
with memory guarantees and well suited to high throughput network servers that
would catch this class of bugs at compile-time [1] [2]

[1] [https://www.rust-lang.org/en-US/](https://www.rust-lang.org/en-US/) [2]
Self declared rust fanboy

------
mabbo
Signs you are about to have a bad time: Tavis Ormandy publicly tweets that he
urgently needs someone from your security team to contact him, and no, the
public disclosure form won't do.

------
ComputerGuru
Some day, the world will wake up to the fact that we've taken the beauty of a
decentralized internet and willingly traded it in for a single-point-of-
failure design.

I will refrain from any criticism of Cloudflare and what I think about this
because they're going through hell as it is. But everyone else is fair game.
The higher a level of service you centralize, the more you stand to lose.

------
xenadu02
Another day, another C memory safety bug that completely breaks all security
everywhere.

We're definitely doomed to repeat the same mistakes over and over.

~~~
sneak
Probably only for 30-50 more years, honestly.

~~~
mshenfield
Most honest comment in this entire thread.

------
tr32q423
The root cause is apparently coming from auto-generated code that causes
buffer overrun:

    
    
        /* generated code */
        if ( ++p == pe )
            goto _test_eof;
    
    

_With the help of Google, Yahoo, Bing and others, we found 770 unique URIs
that had been cached and which contained leaked memory. Those 770 unique URIs
covered 161 unique domains._

The examples in the report shows Uber, okcupid , etc. It would be good to know
the full list, to know what password might have been compromised.

[https://blog.cloudflare.com/incident-report-on-memory-
leak-c...](https://blog.cloudflare.com/incident-report-on-memory-leak-caused-
by-cloudflare-parser-bug/)

~~~
nikisweeting
We're working on getting a full list up here:
[https://github.com/pirate/sites-using-
cloudflare](https://github.com/pirate/sites-using-cloudflare)

I'm currently just searching the Alexa top 10,000 doing DNS scraping, but I'll
updating it with reverse resolves from cloudflare.com/ips/ next.

------
alkonaut
Just stop using pointer arithmetic and manually managed buffers for anything
security/safety related already.

Had this proxy been written in nearly _any_ other language it wouldn't have
had this vulnerability, like so many similar vulnerabilities.

Using ML or Rust or Java or whatever doesn't magically make all
vulnerabilities disappear but it sure makes those that are intrinsic to C
disappear. And that's not just a few.

There is just no excuse.

------
dkarapetyan
Every piece of dependency in your stack is a vulnerability vector. I feel like
this is the only sane assumption to make these days. Yesterday I was thinking
of doing some stuff with cloudflare and today I'm reading this report.

The modern web requires a paranoid attitude.

~~~
nine_k
Only the paranoid survive, Andy Groves famously said.

------
tannhaeuser
Holy sh*t. Is this the end of Cloudflare with the trust being absolutely
destroyed and lawsuits coming in? Can't say I'm sad for them. Cloudflare sells
you DDOS protection, and hosts (eg. masks the IP of) the very DDOSers to
protect against themselves, which I find bordering on the criminal.

Hosters like Hetzner, OVH have for a year now offered DDOS protection (I'm
guessing it's heuristic rate limiting, but they won't tell details b/c that
would make it trivial to workaround it, so they say). Could someone
characterize their offering and tell me if it's any good?

To those spinning a story against C programming here: it is entirely possible
(trivial, even) to isolate address spaces between requests, and has been for
like 25 years (CGI programming) and more. When you absolutely must use a long
running, single-address space service container, OpenBSD's httpd shows how to
do it right (goes to great lengths to randomize/re-initialize memory etc.). I
agree, though, that using straight C isn't a good choice for the latter.

~~~
tannhaeuser
From [https://arstechnica.com/security/2017/02/serious-
cloudflare-...](https://arstechnica.com/security/2017/02/serious-cloudflare-
bug-exposed-a-potpourri-of-secret-customer-data/):

    
    
        A while later, we figured out how to reproduce 
        the problem. It looked like that if an html page
        hosted behind cloudflare had a specific
        combination of unbalanced tags,
        [...]
        The leakage was the result of a bug in an HTML
        parser chain Cloudflare uses to modify Web pages
        as they pass through the service's edge servers.
    

Ahem, at the risk of sounding pedantic, but this wouldn't have happened when
using a proper HTML/SGML parser ([1]).

[1]:
[http://sgmljs.net/blog/blog1701.html](http://sgmljs.net/blog/blog1701.html)

------
pmahoney
I haven't found a clear answer to this:

CloudFlare has multiple SSL configurations:

> Flexible SSL: There is an encrypted connection between your website visitors
> and Cloudflare, but not from Cloudflare to your server.

> Full SSL: Encrypts the connection between your website visitors and
> Cloudflare, and from Cloudflare to your server

(I'll add Full SSL mode still involves CloudFlare terminating SSL (decrypting)
before re-encrypting to communicate to your server)

If I am running in Full SSL mode, is (or was) my data vulnerable to being
leaked?

~~~
maknz
Full SSL requests still terminate at CloudFlare, and would still be
vulnerable. It's just that CloudFlare's connection to your origin is also
encrypted.

~~~
pmahoney
Thanks. Wish they had explicitly stated that all SSL modes were affected
(unless I missed it...)

------
hehheh
I'm a little drunk so please forgive me if I'm way off base here or if I'm
ultimately describing a service that already exists.

Unless I'm mistaken, CloudFlare's services necessarily require they act as a
MITM. Would it be possible or practical change the DDoS protection service
such that it uses an agent on the customer's end (the CF customer) that relays
relevant data to CF, instead of having CF MITM all data?

As it is now, we have:

    
    
      End user <-> CF MITM to inspect packet data <-> CF Customer site
    

where CF uses the data discovered through MITM (and other metadata such as IP)
to determine if the end user is a bad actor.

What if we, instead, had something like:

    
    
      End user <-> CF TCP proxy <-> CF Customer site
                       ^                    |
                       |                    v
                 CF decision agent <-- CF metadata ingest
    

The CF captive portal would not work with this but they could still shut down
regular ol boring TCP DDoSes.

~~~
manigandham
You wouldn't be able to have any CDN caching, only transit of encrypted
traffic. Which is fine, but all the major clouds have load balancers that
already do this and have varying levels of included and paid DDoS protection.

------
ffjffsfr
Does anyone know answer to this question someone is asking there at the end?
Is it related?

> could you tell us why a lot of people had to re-authenticate their Google
> accounts on their devices all of the sudden? It may not have been related,
> but Google definitely did something that had us all re-authenticate.

I too had to reauthenticate and was very worried because it was first time I
had to do this, I thought something bad happened with my account and it was
very suspicious.

------
jlgaddis
Anyone wrote a script yet that checks the top 1M (or so) web sites to find out
which use Cloudflare? It would help with knowing what secrets I need to change
(as an end user -- I'm not a Cloudflare customer, thank $deity).

~~~
nikisweeting
Yup, running it now. Results are being posted as fast as I can here:

[https://github.com/pirate/sites-using-
cloudflare](https://github.com/pirate/sites-using-cloudflare)

~~~
leoh
Looks cool, you have a lot of duplicates, though.

    
    
       $ cat sorted_unique_cf.txt | wc -l
        7385121
    
       $ cat sorted_unique_cf.txt | uniq |wc -l
        4287625

~~~
nikisweeting
Apologies, fixing that now! I ran uniq before but piped it into the wrong file
and ended up uploading the non unique version.

~~~
nikisweeting
Fixed: [https://github.com/pirate/sites-using-
cloudflare/raw/master/...](https://github.com/pirate/sites-using-
cloudflare/raw/master/sorted_unique.zip)

------
ThrustVectoring
Maybe I'm being a bit too paranoid, but shouldn't your services be set up in a
way that doesn't let Cloudflare touch that sort of sensitive data in the first
place? You can't distrust everything, of course, but "compromised reverse-
proxy acts as a MITM by logging and exfiltrating sensitive information" seems
like it ought to be in the threat model of service providers.

------
aioprisan
CloudFlare's disclosure severely downplays the impact that this can have on
their customers. We're going to close our account shortly.

------
sparkling
This might be the time to point out the CloudFlare watch blog:
[http://crimeflare.com/](http://crimeflare.com/)

~~~
sfeng
The thesis of that blog seems to be that Cloudflare should be censoring
content and deciding who gets to have websites on the internet.

~~~
mark_edward
So you think businesses have no responsibility to police themselves and their
users? If a shop was caught knowingly facilitating say welfare fraud they'd
get fined up the wazoo but for some reason being ~digital~ makes it OK?

------
Karupan
This is huge and CF is certainly downplaying the issue. To be clear, I think
the kind of tech that they deal with is extremely complex, which makes it ever
harder to test or uncover them easily. And they have been reasonably good with
disclosures (prior to this incident).

When I was evaluating CF for a small personal app, I really thought hard about
using a public reverse proxy and decided that it wasn't worth it for the scale
I was dealing with. No one can predict these security issues, but I sure am
glad I didn't go with them!

------
askvictor
Could this be the reason behind having to reauth my Google accounts in the
past couple of days? I.e. did Google invalidate all auth tokens in case they
leaked via a third party website via CF?

~~~
orian
It seems so.

------
xt00
Wow apparently they never fuzzed their input and looked at the output. A
malformed html input should be about the easiest possible thing to try...
yeouch...

------
omgtehlion
What bothers me is not the bug itself, but the fact that so much sites and
apps terminate SSL at cloudflare that NSA/FBI/other-3-letter-agency does not
need to come after any separate company, but just needs to tap cloudflare and
call it a day.

~~~
27182818284
Everyone has heard the expression that we need to diversify portfolios.
Perhaps it is true of CDNs

------
packetized
Salient question at this point: Did Cloudflare have any systems in place that
would allow themselves to identify queries that were abusing this defect?

------
manigandham
Side note: HackerNews uses CloudFlare.

------
homakov
Chrome marking Cloudflare HTTPS as "Secure" must be turned into something
different, like "Not So Secure" or whatever. Secure = end to end.

Cloudflare is MitM by design. Chrome and others must not tolerate it. This
vulnerability is just another reason to do it asap.

~~~
bubblethink
That's because HTTPS allows that. Whether it's cloudflare, or your own servers
and load balancers, it's all legal. So it would be unfair to single cloudflare
out. You could take some measures to identify their flexible-ssl traffic, and
that's a grey area, but their regular ssl is fine. If it weren't for them, you
would roll your own solution, which wouldn't be very different.

~~~
lifthrasiir
Ultimately I believe CF is sustaining its business by filling a gap in the
Internet, namely DDoS protection. Until somehow the gap is closed we will see
CF-like services continue to be popular even after this incident.

~~~
homakov
So there is no cheap in-house solution to DDoS but CF?

~~~
lifthrasiir
CF's success (especially in the free plan) suggests that this might be
actually true---I'm afraid I cannot prove or disprove the claim (that's why I
_believe_ so). My observation comes from drawing the parallel to djb's
Internet Mail 2000 [1], which tries to counter spams by changing mail storage
to the sender's responsibility.

[1] [https://cr.yp.to/im2000.html](https://cr.yp.to/im2000.html)

------
artursapek
Holy shit, this could be a company-ending event. For CloudFlare or any of its
clients.

~~~
chx
Their clients, sure, especially they are HIPAA regulated (let's pour one out
for the poor sods) but CF only if everyone abandons them and many won't. Gross
negligence does not even exist online AFAIK and so criminally you can't even
start because there's nothing to work with, perhaps negligence but that's a
slap on the wrist. A civil suit ... sure you can sue anyone in civil court for
whatever but you need to prove damages here and that'll be bloody hard.

------
actuator
I wrote a script which checks the domains you have visited from your chrome
history to see if they use Cloudflare by checking if the header `cf-ray` is
present in their response headers:
[https://gist.github.com/kamaljoshi/2cce5f6d35cd28de8f6dbb27d...](https://gist.github.com/kamaljoshi/2cce5f6d35cd28de8f6dbb27d586f064)

Found my bank's site on it. :(

------
kogir
I'm not 100% clear: Only three features were affected, and only sites with one
or more of those features enabled leaked data into their pages.

But was the leaked data similarly limited to only the sites with the features
enabled? Or could it have come from any request - even an entirely unrelated
site?

~~~
toyg
_> only sites with one or more of those features enabled leaked data_

No. From what he says, enabling that feature on a CF proxy basically triggered
the bug on any site that happened to go through that proxy, regardless of
whether it used the feature or not.

~~~
richardwhiuk
It only triggered the bug on sites that were using those features, but any
other CF site was vulnerable to getting dumped out.

~~~
toyg
yeah that's what i meant - content could be dumped from any site going
through, regardless of whether they used the broken features.

------
chousuke
I used the lastpass CLI tool and some UNIX tools to do a tentative check of
which of my domains might be affected. Something like the following should
work okay:

    
    
        lpass ls | egrep -o '[a-z]+\.[a-z]+' | sort > mydomains.sorted
        sort sorted_unique_cf.txt > cf_really_sorted
        comm -12 mydomains.sorted cf_really_sorted
    

It's not perfect (since it will only look at the lastpass item description,
not the actual URL, and will only match foo.tld type domains), but it still
found a number of domains for me

------
abalone
Cloudfare's bug bounty maximum reward[1]:

 _1\. Recognition on our Hall of Fame._

 _2\. A limited edition CloudFlare bug hunter t-shirt. CloudFlare employees
don 't even have this shirt. It's only for you all. Wear it with pride: you're
part of an exclusive group._

 _3\. 12 months of CloudFlare 's Pro or 1 month of Business service on us._

 _4\. Monetary compensation is not currently offered under this program._

Guessing they're gonna reconsider #4 at this point.

[1] [https://hackerone.com/cloudflare](https://hackerone.com/cloudflare)

~~~
kbody
Indeed, I've heard the issue of low signal-to-noise ratio, but it's pretty
irresponsible not to offer any real reward.

------
dreamcompiler
Only inherently unsafe languages like C make it possible for an amateur-hour
HTML parsing blunder to spew secrets all over the Internet. If you can't be
bothered to check your return codes, at least use a language that doesn't
multiply the damage from that mistake a million-fold.

------
techolic
Is there an International Day of Internet Security? I think we should make
today that day.

------
kmfrk
So how does one find or generate a list of companies using CloudFlare to
figure out how you're affected - kinda like HaveIBeenPwned.com?

~~~
nikisweeting
While theoretically many non-cloudflare sites can be affected, I'm compiling a
list of known cloudflare-using domains here: [https://github.com/pirate/sites-
using-cloudflare](https://github.com/pirate/sites-using-cloudflare)

It can serve as a starting point, just Ctrl+F for domains you use.

------
DanielDent
There are still a lot of results with leaked data in Google's Cache and they
are pretty easy to find..

Some possible queries: "CF-Int-Brand-ID", nginx-cache "Certisign Certificadora
Digital",

Once you find one, you can look through the results for unusual
strings/headers which you can use to find more results.

Many results have clearly been removed from Google's cache, but.. many also
have not.

------
bcl
Here's a simple little Rust app to check a list of domains for CF usage --
[https://github.com/bcl/uses-cf](https://github.com/bcl/uses-cf)

------
loeg
Anyone know which password manager uses Cloudflare? Just trying to figure out
if I'm affected.

~~~
davb
Thankfully it looks like it's not 1Password, who seem to use AWS CloudFront.

~~~
chris_7
I am confused - why is 1Password is using anything but iCloud or Dropbox?
Those are the only options I see (and "Folder" which is presumably just
local).

~~~
scott_karana
They have "for Teams/Families" options now that appear to use a proprietary,
server-side sync, instead of the safer guarantees of the traditional client.

~~~
floatboth
It's still very safe. They do not rely on TLS for protection, they actually
send already encrypted data over TLS.

------
hendzen
I think this bug is kind of an indictment of Ragel. It has some great ideas,
but since the generated code is so low level - and allows arbitrary blocks of
code to be executed in the guts of the parser, bugs like these can result in
this horrible memory issues - particularly since the generated code is often
used to parse untrusted user input.

~~~
jgrahamc
I don't blame Ragel.

~~~
eridius
Ragel shares part of the blame. Why did it use a strict equality check when it
could have trivially done a >=?

~~~
arestor
It's C. If you have an array, you may only compare to one element behind the
last. Everything else is undefined behavior. So a compiler may just "optimize"
your >= to ==.

~~~
eridius
No it won't. It's using pointers, not array indices. The compiler has no
possible way of knowing that `pe` is the one-past-the-end address.

~~~
arestor
It's still UB. The array could potentially be at the end of the address
space...

~~~
eridius
Well yes, it could, but that's not really an argument for saying that Ragel
using == is just as good as using >=.

------
jitbit
Webmasters and App-devs running on CloudFlare. You (at least) have to "force-
logout" your users that have "remember me" cookie set.

At least change the cookie name so the token stops working. For example, in
ASP.NET - change the "forms-auth" name in the web.config file. etc etc.

------
wereHamster
So.. when are we going to stop using unsafe languages which allows these kinds
of memory corruption or leaks? If this is not reason enough, what else needs
to happen before people realise that whatever language the cloudflare proxy is
written in is a really bad one?

------
afandian
In addition to comments here calling the words 'memory leak' disingenuous
because it's technically correct but underplays the problem, I'm now seeing
articles in non-technical publications referring to the incident as a "leak".

In the wider world the word "leak" doesn't mean memory access patterns, it
means deliberate sabotage.

The headline in "The Verge" is "Password and dating site messages leaked by
internet giant Cloudflare". That's technically correct too, but also gives
completely the wrong message.

Simpler, proactive messaging from Cloudfront might have helped here.

------
pjmlp
Time for the C. A. R. Hoare's weekly quote, taking time to reflect on what
happened since 1981 regarding computer security on system languages.

The first principle was security: The principle that every syntactically
incorrect program should be rejected by the compiler and that every
syntactically correct program should give a result or an error message that
was predictable and comprehensible in terms of the source language program
itself. Thus no core dumps should ever be necessary. It was logically
impossible for any source language program to cause the computer to run wild,
either at compile time or at run time. A consequence of this principle is that
every occurrence of every subscript of every subscripted variable was on every
occasion checked at run time against both the upper and the lower declared
bounds of the array. Many years later we asked our customers whether they
wished us to provide an option to switch off these checks in the interests of
efficiency on production runs. Unanimously, they urged us not to - they
already knew how frequently subscript errors occur on production runs where
failure to detect them could be disastrous. I note with fear and horror that
even in 1980, language designers and users have not learned this lesson. In
any respectable branch of engineering, failure to observe such elementary
precautions would have long been against the law.

\-- Turing Award lecture 1981

------
jjoe
Everyone: change your HN password asap!

~~~
StavrosK
Everyone: HN should implement U2F or TOTP so we don't need to only rely on
passwords :(

~~~
simcop2387
How do i get the key information to HN without passing through cloudflare so
that someone could sniff it?

~~~
jjoe
Someone just posted how:
[https://news.ycombinator.com/item?id=13719366](https://news.ycombinator.com/item?id=13719366)

------
mrep
What is the optimal balance between centralization and decentralization? Most
people in this thread are complaining about how using a big centralized
service (cloudfare) causes so much damage when security issues come up, and
yet I have seen many people advocate using a single password manager (like
1password) to which this exact type of huge security problem can happen (your
password manager is the single point of security failure which can comprimise
all of your accounts!!!).

What is the optimal solution???

~~~
floatboth
There's a difference between a MITM proxy in front of a huge portion of the
web and a password manager that's running locally on a personal machine.

Also there's the 2-factor stuff to protect you when you somehow lose your
manager's master password. What protects you when the proxy in front of you
misbehaves and exposes your shit?

------
Blackthorn
This is probably gonna get buried at this point, but one thing I'm surprised
about is this seems like yet another parser bug. Why are we still using hand-
written parsers? Even if you're Very Smart, you'll probably get it wrong. We
have parser generators for a lot of things. Even for mostly unparseable
garbage like wild-type HTML we have pretty good libraries for handling it.
Fresh hand-written parsers are just bombs waiting to explode.

~~~
niftich
Your comment doesn't apply for this particular case, because the submission
goes into great detail that the parser in question was written with Ragel, a
parser generator. The code written by them in Ragel contained a bug, which lay
uncaught and dormant for years, and manifested only when calling/wrapping code
was altered.

~~~
Blackthorn
It still seems like a gross mismatch of power though. Correct me if I'm wrong
but Ragel only can output parsers for regular languages, yes? You can't call
their Ragel code an HTML parser because Ragel can't output a parser powerful
enough to parse HTML.

~~~
nostrademons
HTML isn't a CFG. The HTML spec is setup as a state machine ( = regular
language) + a number of side data structures like the stack of open elements
and list of active formatting elements. This maps very easily to Ragel, where
your actions can easily have side-effects and reference internal state within
the language.

~~~
Blackthorn
> HTML isn't a CFG. The HTML spec is setup as a state machine ( = regular
> language) + a number of side data structures like the stack of open elements
> and list of active formatting elements.

That's...that's what a context-free grammar is.

(FWIW, wild-type html might not be context-free but require a higher powered
parser.)

~~~
nostrademons
Operations on the stack of open elements don't have to follow the same LIFO
discipline that a set of recursive productions (i.e. a CFG) would generate.
For example, parts of the HTML5 parsing algorithm (eg. the Adoption Agency
Algorithm) involve conditionally popping elements off the stack of open
elements while they appear in the list of open formatting elements (which is
itself a FIFO queue), and then pushing the popped elements _back_ onto the
list of open formatting elements. Other parts (eg. tables) involve re-
parenting elements into parents that are currently on the stack of open
elements.

(I've written a fairly well-used conforming HTML5 parser, so I do have some
domain knowledge in this area...)

~~~
Blackthorn
Fair enough...I'll cop to not knowing much about HTML5, my knowledge stopped
with HTML4. I went and referenced the parsing rules and they are quite the
mess. There's no official grammar for HTML5 that I can find and even if there
was, I don't think there's a solid parser generator out there that handles
anything more powerful than context-free grammars.

------
coindork
And it shall be called Cloudbleed.

~~~
Natanael_L
Still hoping for Downpour

------
daxfohl
Anyone know of a way to google for your passwords (assuming you have strong,
unique passwords) to see if they've been exposed anywhere, without exposing
them?

~~~
ceejayoz
Change them, _then_ Google them?

------
cypherpunks01
"We also undertook other search expeditions looking for potentially leaked
information on sites like Pastebin and did not find anything."

How comforting!

------
fulafel
Yet another strong argument for end-to-end security. Terminate in the middle,
and you risk things like this.

Hopefully people will learn something from today.

------
rickdmer
I created a Chrome extension that searchs your bookmarks for sites that use
Cloudflare: [https://chrome.google.com/webstore/detail/cloudbleed-
bookmar...](https://chrome.google.com/webstore/detail/cloudbleed-bookmark-
check/egoobjhmbpflgogbgbihhdeibdfnedii)

------
Globz
I was planning on moving my website over DigitalOcean and I now
[http://www.doesitusecloudflare.com/?url=www.digitalocean.com](http://www.doesitusecloudflare.com/?url=www.digitalocean.com)
is telling me that they are affected by cloudbleed, I guess I should wait it
out...

------
danvdragos
How was https traffic leaked? Cloudflare, in order to offer its services, acts
like a man in the middle and internally decrypts https traffic [0]

[0]: [https://scotthelme.co.uk/tls-conundrum-and-leaving-
cloudflar...](https://scotthelme.co.uk/tls-conundrum-and-leaving-cloudflare/)

------
stevebmark
Is there a government body that can enforce fines over this? Or is a class
action lawsuit the only way to seek damages?

------
mixedbit
There is a huge fleet of compromised machines out there that belong to
botnets. Soon we will see the botnets operators extracting content from these
compromised machines browser caches to hunt for data leaked in this incident.
Clearing search engines caches is just not enough, all secrets need to be
replaced.

------
borplk
It says their bug bounty program has a top-tier reward of a t-shirt? Wow ...
don't go bankrupt Cloudflare.

------
symlinkk
I've been going through Google's and Bing's caches for about 2 hours looking
for leaked credentials and I don't see much - many results don't have an
option to view a cached copy. I think Google and Bing are wiping any cache
entry that are affected by this vulnerability.

------
helper
I was able to get a few hits from a quick google search that are still in
google's webcache.

------
planetix
So time to reset password and logout of all mobile apps to get new
authorization tokens?

~~~
tonyztan
Seems like we need to reset/rotate everything. So yes, I believe so.

------
SandB0x
Can someone provide a lay-person's explanation of the issue and its
implications?

~~~
arestor
See
[https://news.ycombinator.com/item?id=13719437](https://news.ycombinator.com/item?id=13719437)

------
_pmf_
From the incident report at [https://blog.cloudflare.com/incident-report-on-
memory-leak-c...](https://blog.cloudflare.com/incident-report-on-memory-leak-
caused-by-cloudflare-parser-bug/) (not the article):

> About a year ago we decided that the Ragel-based parser had become too
> complex to maintain and we started to write a new parser, named cf-html, to
> replace it. This streaming parser works correctly with HTML5 and is much,
> much faster and easier to maintain.

I'd assume that at this point, customers would like to have a little more than
a vague promise.

------
bovermyer
Well, my day tomorrow is going to be busy. So's my evening tonight, I guess.

------
sparkling
I know what Cloudflare is but i don't quite understand the underlying issue.

Can someone explain in simpler terms what happened here and how it a) affects
sites using Cloudflare and b) Users accessing sites with Cloudflare?

~~~
arestor
Some features had a bug which lead to uninitialized memory (AKA previous
memory contents) in the output of a malformed HTML page was requested.

As one such server handles many sites, everything that the server handled
before that request may be compromised. This includes all HTTP-GET/POST data
(credentials, direct messages to other users, ...), Headers (API tokens,
Login-Cookies) and contents.

So, you have to assume that everything you did on a CF "protected" website in
the last months (especially between 2017-02-13 and 2017-02-17) is potentially
compromised.

~~~
zaatar
Where is a reliable list of CF-protected websites so I may identify which ones
I have interacted with?

~~~
tonyztan
An unofficial list is being compiled here: [https://github.com/pirate/sites-
using-cloudflare](https://github.com/pirate/sites-using-cloudflare)

------
acd
Cloudflare is also breaking anonymous surfing by throwing captchas at you.
Security wise they do DDoS ok but not WAF which Incapsula does a lot better.
When I mean better I mean protection against exploits.

------
aerovistae
I noticed StackOverflow is on the list of compromised sites. I sign into that
via my google account. Does this mean I need to change Google credentials?

~~~
niftich
Most of these 'Sign in with [Social Identity Provider]' implementations,
including Google [1] use OIDC ("OpenID Connect"), which in turn itself is
built on top of OAuth 2.0. From the OAuth 2.0 side, the site into which you
wish to gain access into -- in this case, StackOverflow -- only sees opaque
tokens that are usually short-lived.

However, OIDC then typically delivers some choice personal info -- no more
than you agreed to when first consented to the integration, but usually
account name and/or email, and maybe real name and some demographic data -- to
the requesting service so that they can both find you in their datastore, and
sync up these attributes. In the case of a service whose OAuth/OIDC callback
url's SSL is terminated with CloudFlare, which we (as of writing) don't yet
know if applies to StackOverflow, this info will touch CloudFlare servers and
could have been contents of memory that was exposed. However, your password
would not be, as in your case, the password was supplied to the Social
Identity Provider (Google) who didn't use CloudFlare to terminate _that_
connection, and the password never left Google, which was the precise usecase
and requirement that the OAuth/OIDC specs were authored to support.

[1]
[https://developers.google.com/identity/protocols/OpenIDConne...](https://developers.google.com/identity/protocols/OpenIDConnect)

------
hacknat
I'm surprised to learn that people with real security concerns are using
Cloudflare. I put it in front of my blog, but I would never use it in front of
something that has sensitive data. I just don't get how companies like Zendesk
could be so stupid. I barely blame Cloudflare. If you think terminating SSL
with a CDN is a good idea you get what you deserve.

------
piker
Given that the plaintext is cached (or feared to be), is googling/binging
one's passwords a bad way to check for pwnage?

~~~
artursapek
Sounds like a bad idea. Maybe search for a small part of the password?

------
tete
Wow, I only recently had a discussion about "What if this happens?". Great
timing to make a point. Unique "told you so" opportunity, but I actually am
sad that this happened. Millions of people wasting time on password changes
and related things again. :(

And now off to resetting a lots of password and checking where OTPs are
possible.

------
matthewowen
So, they know which sites leaked data in responses. It sounds like they can
also say categorically that some sites won't have been affected (if they don't
share any infrastructure with the sites that could have leaked data).

Will Cloudflare be explicitly notifying customers about whether data from
their site could have been leaked by this bug?

------
mordant
Apparently, the only way to change one's Uber password is to use the 'Forgot
password' path on their login page.

So, I clicked on that - and I get a 500 error from NGINX.

My guess is that a lot of services are going to be overwhelmed by the sheer
volume of password reset requests, thus preventing users from resetting their
passwords.

------
a3n
Password managers are mentioned.

I looked on the lastpass blog (s/www/blog/), nothing about this. Is it just
too early?

~~~
sleepychu
Lastpass does the crypto on the client side, your encrypted password database
could have been leaked but if your master password is sufficiently strong then
it will be hard to break.

That said if you reset all your current lastpass passwords with newly
generated ones after changing your master password you'll protect yourself
from any attack.

------
csomar
Oh boy, this is bad as fuck. Major bitcoin exchanges were affected and these
are exchanges where if you can login, you might be able to withdraw the cash
irreversibly for ever.

I'm trying to figure out how bad this is; and a part from the exchanges I'm
using which other sensitive sites are concerned.

------
Rican7
Yeaaaaa, this isn't good.

This is what CloudBleed looks like, in the wild:
[https://gfycat.com/ElatedJoyousDanishswedishfarmdog](https://gfycat.com/ElatedJoyousDanishswedishfarmdog)

A random HTTP request's data and other data injected into an HTTP response
from Cloudflare.

Sick.

~~~
jeromenerf
> This is what CloudBleed looks like

Ironically, gfycat seems down now.

------
secfirstmd
"and even plaintext API requests from a popular password [1Password] manager
that were sent over https"

Plaintext?

------
dorianm
Here is a list of domains where I found public leaked data:
[http://doma.io/2017/02/24/list-of-affected-cloudbleed-
domain...](http://doma.io/2017/02/24/list-of-affected-cloudbleed-domains.html)

------
hkjgkjy
One of the reasons I prefer paying with Bitcoin over credit card, is that when
using cryptocurrency I don't have to give them they key to my account -
instead they give me an inbox that I send the value into.

Guessing a lot of credit card details are ripe for picking in the data they
leaked.

~~~
ceejayoz
> Guessing a lot of credit card details are ripe for picking in the data they
> leaked.

Sure, but that's where credit cards _shine_ in comparison to Bitcoin. In the
US, you're protected by Federal law in that scenario. A brief pain in the ass
- reporting the fraud and getting a new card number - and you're out $0.

Meanwhile, a bit of malware can drain millions of dollars of Bitcoin with zero
recourse. It's gone. This isn't theoretical, it has happened.
[https://www.theguardian.com/technology/2016/aug/03/bitcoin-s...](https://www.theguardian.com/technology/2016/aug/03/bitcoin-
stolen-bitfinex-exchange-hong-kong)

------
faragon
Does anyone know if there is a way for mapping virtual addresses to areas with
zeroes and replacing it with the memset to 0 on write access, so software
could be still efficient without calling calloc() instead of malloc()? (i.e.
memset to 0 only for actually written zones)

------
mderazon
So did anyone find out why so many Google accounts got "action required"
alerts yesterday ?

------
XorNot
Incidents like this remind me that the password problem is only partially
solved by password managers: most of the internet (i.e. if you're not my bank)
needs a simple, easy to script protocol that allows me to automate the process
of rolling a lot of passwords.

------
fagnerbrack
Cloudflare blog post related to this incident:
[https://blog.cloudflare.com/incident-report-on-memory-
leak-c...](https://blog.cloudflare.com/incident-report-on-memory-leak-caused-
by-cloudflare-parser-bug/)

------
soheil
What's the rationale behind sending user PII through a CDN? Presumably that is
useful to that one user only so a CDN wouldn't be super useful in distributing
the load across its edges. Also doesn't CDN caching kinda defeat the purpose
of having SSL?

~~~
orthecreedence
Cloudflare terminates SSL and then forwards the request to your servers as one
of their services. This isn't about the CDN, but about them terminating SSL,
then leaking the plaintext data back through other requests.

~~~
soheil
What are the benefits of terminating SSL early at the CDN level? It seems to
me the risks associated with not having SSL still remain they're just shifted
to between the CDN and the backend. Is it much more than just giving lip
service to SSL and getting away with things like browser restrictions, etc.?

~~~
Laforet
DDoS protection and general ease of use. There are several options for the
extent of encryption between CF's edge and the origin server but the onus is
on the site owner to configure it properly.

Sure it defeats the ideals about TLS and the internet in general ala "every
connection should be point to point" but we've been ruining that with
firewalls and NATs for a long time and having some degree of TLS is still
better than nothing at all.

~~~
orthecreedence
> having some degree of TLS is still better than nothing at all

Apparently not.

------
SadWebDeveloper
Oh boy what a great week... first we have SHA-1 getting a fast-track to the
obsolete hashes and now cloudflare is f*cking everyone because they tried to
obfuscate emails from websites and fail to "test every edge" case... whats
next is the question.

------
joeyh
The bug was in cloudflare servers, not code run on customer's own web servers,
right?

------
tkachenko
Small service to check if your site is POSSIBLE affected to CloudFlare data
leaks
[https://cloudflareleaks.webtls.com/](https://cloudflareleaks.webtls.com/)

------
djhworld
Can someone explain to me why they were parsing HTML in the first place?
That's the bit I don't fully understand, but I've not got experience of what
Cloudflare does, I thought they were a CDN

------
no_protocol
Is there a list of sites potentially affected?

I'm assuming I need to change my passwords on a significant number of sites.
So far none of them have alerted me to a potential breach. Would love to have
a head start.

~~~
nikisweeting
Compiling a list here: [https://github.com/pirate/sites-using-
cloudflare](https://github.com/pirate/sites-using-cloudflare)

It will be updated as we find more.

------
stevenhubertron
Well now I have a great response to the sales guy that bugs me everyday.

------
yclept
for easy firewalling and i'm sure a fun internet experience
[https://www.cloudflare.com/ips-v4](https://www.cloudflare.com/ips-v4)

------
stephenr
Who wants to bet this won't change a lot of developer's making statements like
"I use <Insert HTTPS offering CDN> so my site is secure"

------
snikeris
Anyone have any additional information about this bit from the comments:

> and even plaintext API requests from a popular password manager that were
> sent over https (!!).

------
frankmoodie
Question: what about the %99 of the internet users who have no idea what
SSL/HTTP/any other web tech is ? How are they even going to be notified?

~~~
fatal510
Nothing will come of this. Just another hype event. You will get your usual
change your password PR emails from a few companies.

------
meowface
Never been so relieved my company uses a different CDN...

~~~
snaky
Do they have a t-shirt awards in their bug bounty programs?

------
Soarnrobertson
This article is beginning to look like a whole bunch of people talking about a
leak and not saying that they would use that data for vicious things.

------
apple4ever
I have yet to receive an email about this. Very disappointed that I had to
find out via another source 12 hours after the blog post was up.

------
Soarnrobertson
Welp, time to move and get a different IP again :\

------
curuinor
Can we start a list of affected right now? I found:

OKCupid

Uber

people claiming 1Password, can't find

Reddit

Lyft

Yelp

Pingdom

Digital Ocean

Montecito Bank and Trust

~~~
curuinor
I found:

FitBit

Hacker News

Stack Overflow

Zendesk

Discord

FastMail (not really see below)

~~~
nmjenkins
We, FastMail, are not affected by this. We do not proxy TLS connections via
any third party. We use CloudFlare for DNS distribution only, which is not
part of this issue.

~~~
interfixus
The least surprising message of the day. Thank you.

My Fastmail-money is well spent.

------
benevol
Well, keep centralizing and this is what you get, sooner or later.

Also, mono-cultures have always been a very bad idea, not just in agriculture.

------
hkjgkjy
HaveIBeenPwnd must be having a great day today!

------
jtchang
This is scary stuff. Any key/password that you used on a cloudflare site
should be considered compromised.

That's a crapton of keys.

------
hatsunearu
>(It took every ounce of strength not to call this issue "cloudbleed")

and some chap did it anyways. yay, i guess.

------
clebio
So, two of the three hard problems in computer science (fencepost and cache
invalidation)?

------
willtim
If you must write your HTML parser in C/C++, then you should expect buffer
overruns.

------
jacquesm
Hm. Not so good. The main website that I log in to that uses CloudFlare is
this one.

------
codezero
Why do they need to add google analytics to random people's web pages?

------
HugoDaniel
So its not only the tor browser experience that sucks with cloudflare.

------
Soarnrobertson
So, would LastPass be involved in this at all? Do tey use CloudFlare?

~~~
bartread
Not according to the lists others have posted, but 1password are.

------
jcwayne
Makes me wonder if the Great Firewall has a caching layer.

~~~
jononor
It would surprise me immensely if they did not. Many mobile operators also
operate caches.

------
philip1209
Interesting. Cloudflare uses a lot of Go, which should hypothetically be
memory safe. Was this system in Go? If so, I would be interested in seeing
proof of concept code for a vulnerability like this.

~~~
daenney
Their old Ragel-based parser was affected. According to their post mortem both
the old parser and the new cf-html one are compiled as nginx modules so I'd
venture a guess that this is probably C/C++ code since afaik you can't extend
nginx through modules written in Go.

> It turned out that the underlying bug that caused the memory leak had been
> present in our Ragel-based parser for many years but no memory was leaked
> because of the way the internal NGINX buffers were used. Introducing cf-html
> subtly changed the buffering which enabled the leakage even though there
> were no problems in cf-html itself.

[https://blog.cloudflare.com/incident-report-on-memory-
leak-c...](https://blog.cloudflare.com/incident-report-on-memory-leak-caused-
by-cloudflare-parser-bug/)

There's a section in that blog post titled "Root cause of the bug" that goes
in further detail.

------
orasis
Our CNAME pointing to github pages was down on Cloudflare today with a 1014
error. I'm guessing they broke some other stuff while scrambling to fix this
privacy issue? Not a good day for them.

~~~
golf1052
Oh that happened to my site as well. Was fun trying to figure out what was
going on in the middle of the day.

Nothing on their status page about it though :|

------
ReedJessen
Reddit just told me my account was compromised

~~~
gooeyblob
Unrelated.

------
sambull
RIP Cloudflare 2017.. took you long enough

------
jhgjklj
Cloudflare please stop asking me if i am a robot and then ask to pick the
store board posts for ever. What kind of idiot coded that, asking me always.

------
throwaway7959
Can anyone ELI5 what's going on?

~~~
nikisweeting
Change all your passwords. Many sites use Cloudflare to serve secure content,
and it was recently discovered that Cloudflare has been leaking secure
content, including passwords, API tokens, etc. to unintended viewers.

------
johnhenry
"Cloudbleed".

------
ers35
Hacker News uses Cloudflare:
[http://bgp.he.net/dns/news.ycombinator.com#_ipinfo](http://bgp.he.net/dns/news.ycombinator.com#_ipinfo)

Add the following to your hosts file to bypass Cloudflare and access HN
directly:

    
    
      50.22.90.248 news.ycombinator.com

~~~
misframer
How did you find that IP?

~~~
ers35
I used Censys to search for the IPv4 addresses of servers serving matching TLS
certificates:
[https://censys.io/ipv4?q=443.https.tls.certificate.parsed.na...](https://censys.io/ipv4?q=443.https.tls.certificate.parsed.names%3A+*.ycombinator.com)

~~~
ndesaulniers
Couldn't someone DDoS'ing a site use this to get around Cloudflare
"protection?"

Uh, asking for a friend.

~~~
ers35
Yes. One can identify the IP address of the origin server behind a reverse
proxy if the server responds to direct requests in a way that identifies
itself. See: [https://cloudpiercer.org/](https://cloudpiercer.org/)

Two steps towards obscuring the origin server include requiring that the HTTP
Host header is set and only responding to Cloudflare IP ranges:
[https://www.cloudflare.com/ips/](https://www.cloudflare.com/ips/)

------
snek
gg to project0

------
ta2987
We need an official and comprehensive list of domains served by Cloudflare
throughout the affected period.

~~~
nikisweeting
I'm compiling an unofficial list, hopefully they'll release an official one
though:

[https://github.com/pirate/sites-using-
cloudflare](https://github.com/pirate/sites-using-cloudflare)

The issue is they have to get permission from their customers before releasing
affected domains.

------
nkkollaw
Could some kind soul do an ELI5?

I'm not lazy, it's just overwhelming trying to figure out what's actually
going on with all these comments...

------
brilliantcode
how far back does this affect websites on cloudflare? I removed mine a year
ago because I was using it for the SSL.

This will put the final lid on cloudflare anyhow. Sticking with AWS.

------
enraged_camel
Your comment got flagged and killed, which I thought was bullshit so I vouched
for it.

Because you're correct: if CF's info sec team is "very very good at their
jobs", how did this incident happen?

~~~
dang
Vouching for a good comment that happens to be dead is fine—that's what
vouching is for. Posting commentary about it is not fine. There are plenty of
non-bullshit reasons for comments to be dead, including that an account is
banned for having abused the site.

We created vouching because some banned accounts sometimes post ok comments;
indeed some banned accounts _only_ do that when they're banned, and
immediately start posting abusively the moment they're unbanned. Life is
complicated.

We detached this comment from
[https://news.ycombinator.com/item?id=13720437](https://news.ycombinator.com/item?id=13720437)
and marked it off-topic.

------
implr
This seems relevant
[https://i.imgur.com/l4kjNba.png](https://i.imgur.com/l4kjNba.png)

------
chiefalchemist
Let's be honest. There are holes. More than we care to admit. The truth, if
embrassed, could undermine the world's economy. It's just question of when.

~~~
chiefalchemist
Down voted? God bless the naive.

------
Sami_Lehtinen
I made similar site too, but with geolocation, tags, and fully threaded
replies and private messages. Like & Dislike - As well as machine learning
which will dig most interesting posts for you. As well as score near by posts
higher etc. But nobody cared. So I'll be shutting it down in 6 months. (Domain
expires)

------
cwisecarver
This sounds to me like an object lesson in "Why you shouldn't write your own
HTML parser."

Every time I see a dev trying to parse HTML with a custom solution or regex or
anything other than a proven OSS library designed to parse HTML I recoil
reflexively. Sure, maybe you don't need a parser to see if that strong tag is
properly closed but the alternative is ...

~~~
abeyer
You're right in 99+% of cases. But I suspect that the needs of cloudflare for
this use case aren't typical of what's expected of an html parser. I'm not
certain that there isn't an existing parser that would work for them, but I'm
equally not certain that there is.

~~~
cwisecarver
I can see the argument but 99+% of this audience isn't cloudflare. My comment
was more directed at those who aren't. Special use-cases are all over the
place. It's just making sure you're choosing because your use-case really is
special and that when you re-implement something that you're doing it because
it's different and better, not because you'd rather write something than
integrate.

