
The Critical Security Flaws that Resulted in Last Friday's Hack - neilwillgettoit
http://blog.cloudflare.com/the-four-critical-security-flaws-that-resulte
======
kijin
Why does Google allow the hacker and the account owner to keep resetting
passwords in rapid succession? The timeline indicates that two lengthy ping-
pong sessions took place during the incident. That kind of behavior should
immediately raise a red flag. How often do legitimate users reset passwords
alternately from two different locations 10 times in 15 minutes?

I'm surprised that Google doesn't detect two people fighting for control of
one account. They could have easily detected ping-pong sessions and and locked
both parties out of their accounts for a couple of hours. Or they could have
penalized the newly added recovery address by forcing an exponential delay
between resets using that address. This is not the first time I've heard of
somebody breaking into a Gmail account while the account owner is using that
very same account.

~~~
manmal
I was in the same situation about 1,5 years ago, when some security hole in
Gmail allowed my account to be hijacked. The account was suspended
automatically after the attacker had sent out 7 spam mails, and I could reset
the account again without the attacker returning ever again. But, the ping
ponging did go on for a while and Google did not bother.

------
j_s
Rule #1 when publicizing security incidents: always publish something else to
the blog within 1.5 hours so that the security incident isn't the top post.

Edit: semi-tongue-in-cheek per comments below; as a CloudFlare customer I went
to the blog when this first came up expecting to see something but bounced
when the first post was a discussion of SSL BEAST since that was the hotness
back in the fall of 2011.

I do believe it was not planned, but I also feel that vulnerability
disclosures should be pinned for a while somehow if possible. I think one way
this is done is having a separation between 'new feature' blog and 'ops' blog.

~~~
eastdakota
Haha. Wish we were that organized. We have a big announcement on Wednesday and
need the announcement about Polish ([http://blog.cloudflare.com/introducing-
polish-automatic-imag...](http://blog.cloudflare.com/introducing-polish-
automatic-image-optimizati)) and the feature we're announcing tomorrow
(Mirage) to come out before then.

I think being among the top stories on Hacker News will take care of people
seeing it. And, for the record, I voted the breach story up.

~~~
dguido
Hey, just wondering, did someone make that infographic by hand or is there
some software that does it for you? EDIT: Made by a graphic designer, answered
below.

And now I will shamelessly take this moment to request a few features related
to account security :-).

* Alerting: SMS or e-mail notification when an unrecognized device logs into my account or when records in my domains change.

* 2-factor Authentication: Prompt for a code delivered via SMS, e-mail, Google Authenticator, or DUO Security to login from an unrecognized device.

* Login Accounting: Let me see what IPs logged into my account, when, geoip info for each, and preferably what actions they took while logged in. Provide an API for this info so I can write an automated script to analyze it for suspicious events.

If you end up making any of these features, it would be cool to open-source a
library you used to do it. There are a bunch of large SaaS providers out there
that use features like these but they're all homegrown implementations afaik.

Btw, the Google Apps Admin Audit API exists but I have never seen anyone do
anything with it and it makes me sad. A few hours with [name a scripting
language] and you could probably have a pretty robust Google Apps monitoring
system, but no one seems to care: [https://developers.google.com/google-
apps/admin-audit/get_st...](https://developers.google.com/google-apps/admin-
audit/get_started)

------
sc00ter
Part of this attack bears a certain resemblance to the recent Bitcoinica
compromise, in that, from what I understand, they were also forwarding admin
emails to personal accounts, one of which was compromised leading to the
attacker gaining control of the bitcoinica virtual servers. Cloudflare were
fortunate - at least the changes made were reversible, whereas the bitconica
compromise resulted in the virtual servers being unrecoverably deleted after
breach and theft.

There are lessons to be learned from both incidents.

------
aSig
Does anyone else get the feeling that the attacker is going to be someone the
Cloudflare team knows? Firstly they would have had to have known Matthew's
phone number. Then, assuming the attacker always had the plan of disrupting
the target site, they would have had to have known that the password reset
mails were BCC'd to admins.

~~~
eli
Getting someones phone number seems pretty insignificant compared to using a
previously undisclosed google security flaw.

And it's probably safe to assume that once you control the admin email account
for a site, it's game over. You could request resets from other providers

------
sp332
O.o This reminds me so much of the hack sequence from the game Uplink. The
game was based on "hacking" but intentionally used hacking techniques from
Hollywood :) It was pretty fun. Anyway for the highest-level targets, you had
to get a voiceprint from the phone of an admin, crack the password on the box,
and break the encryption while bypassing monitors.

Just found out: Uplink is on Steam, and in the Ubuntu Software Center now.

------
Mizza
Incredible hack! Hats off to the hacker and to Cloudflare for the transparency
of the response.

What customer was the target?

~~~
neilwillgettoit
Another poster on r/netsec
([https://pay.reddit.com/r/netsec/comments/ui0k4/cloudflare_wa...](https://pay.reddit.com/r/netsec/comments/ui0k4/cloudflare_was_hacked_with_a_flaw_in_google_apps/))
mentioned that it was 4chan.org.

------
cypherpunks01
Wow - that's some epic transparency. Kudos to Matt and the Cloudflare team for
that.

------
ejfox
I like the infographic showing the series of events, definitely goes a long
way in terms of aiding transparency. I wonder if it could be a new trend?

~~~
dfc
I would love to know if the graphic was made by hand or in an automated
fashion.

~~~
eastdakota
Made by hand by Kevin our great graphics designer. Was up late very last night
putting it together. Filled in the last few details this AM.

~~~
dfc
Its awesome, hat tip to Kevin. You guys should make the smaller version a link
to the full version. I did not see the text link at first...

~~~
eastdakota
Posterous usually does that automatically. Not sure why it isn't here. Will
see if I can wrap the image in an <a> tag.

Update: Done.

------
metafour
Would most of the attack been rendered impossible if Matthew answered his
phone at 11:39 instead of letting it go to voicemail?

~~~
cobbal
I'm not sure of the details here, but it wouldn't be too hard to make sure the
reset call arrived in the middle of another call.

~~~
eastdakota
That makes a lot of sense and could have been what happened. It would also
make it more difficult for Google to do something like ignore responses that
come after 4+ rings.

~~~
greyboy
I'm more curious why a "secure" PIN is simply left, automated, as a message. A
more "secure" option, I would think, would be to require some sort of input
from the person who answered (say, "Press 1 for the PIN" where that number is
randomized, or something).

------
startupfounder
"AT&T was tricked into redirecting my voicemail to a fraudulent voicemail box"

Capitan Crunch called...

------
Bob_W
The hack seems very well planned - I wonder just how many smaller sites have
been hacked the exact same way as practice, and not picked up since they
didn't have direct access to Google's security people?

~~~
ceejayoz
I'd imagine you wouldn't blow a discovered flaw in Google's two-factor
authentication setup on a small site. You'd sit on it until a big enough
target came along, which CloudFlare certainly has become.

~~~
Bob_W
I don't think you'd be blowing it if you have a reasonable idea that the site
owner just isn't big enough to get Google to return their calls.

~~~
ceejayoz
Any usage of an exploit risks its discovery.

------
BryanB55
Pretty intense hack... I think Google gives you the option of setting up a
recovery phone to receive voice or SMS messages. It looks like SMS may be more
secure.

It also sounds like he didn't have 2 factor setup on his personal gmail
account. I wonder if that would of helped.

~~~
gwillen
I believe that once you have it set to use a phone number, it's always
possible for the attacker to choose to have it call instead of sending a text.
(I seem to recall it provides that option on the 2-factor login screen.)

------
gojomo
So, the "Five Whys" analysis came up one short, eh?

Just kidding, this is a great level of detail and much appreciated, to
understand CloudFlare's process and how to protect against or recognize these
tactics elsewhere.

------
techinsidr
[http://www.securityweek.com/exclusive-google-two-factor-
auth...](http://www.securityweek.com/exclusive-google-two-factor-
authentication-flaw-exposed-google-apps-customers)

------
sc00ter
Is flaw # 5 (or #1 depending on how you look at it) not having two-factor auth
on the personal account? Or does account recovery by-pass two-factor auth by
design?

------
lsh123
The funny part - cloudflare still hosts the DNS for the guys who claim
responsibility for this attack:

    
    
       Domain Name: UGNAZI.COM
       Registrar: ENOM, INC.
       Whois Server: whois.enom.com
       Referral URL: http://www.enom.com
       Name Server: LEE.NS.CLOUDFLARE.COM
       Name Server: RUTH.NS.CLOUDFLARE.COM
       Status: clientTransferProhibited
       Updated Date: 29-may-2012
       Creation Date: 22-jan-2012
       Expiration Date: 22-jan-2013

------
radicaldreamer
Hrm [http://exiledonline.com/isucker-big-brother-internet-
culture...](http://exiledonline.com/isucker-big-brother-internet-culture/)

------
slavak
Great job on the transparency.

And now I know about the Google Authenticator app. Fancy little thing, that;
glad to find out about it.

------
madao
The fact that they are relying on an external mail vendor, and had passwords
in their emails is a very sad practice.

~~~
ams6110
I'm guessing that they didn't have passwords, but were BCC-ing the password
reset emails/links to the gmail account.

But yeah... I don't like the idea of using Gmail for this either.

------
saturn
Sounds like if the hacker had just done it out of hours, perhaps when the
person in question was asleep, they would have had uncontested access to the
accounts and the hack might have been far more damaging.

------
jsprinkles
I'm amazed Cloudflare got a response from Google that quickly. I'm a paying
Apps customer and I don't see responses for 24 to 48 hours on security
incidents, not to mention that Google doesn't have a stellar reputation when
it comes to things like "support".

Goes to show, it's always who you know (or it's bullshit, which is less
likely). Or I don't have enough users.

~~~
jrockway
Paying Apps customers can call Google and talk to someone immediately.

