Why does Google allow the hacker and the account owner to keep resetting passwords in rapid succession? The timeline indicates that two lengthy ping-pong sessions took place during the incident. That kind of behavior should immediately raise a red flag. How often do legitimate users reset passwords alternately from two different locations 10 times in 15 minutes?
I'm surprised that Google doesn't detect two people fighting for control of one account. They could have easily detected ping-pong sessions and and locked both parties out of their accounts for a couple of hours. Or they could have penalized the newly added recovery address by forcing an exponential delay between resets using that address. This is not the first time I've heard of somebody breaking into a Gmail account while the account owner is using that very same account.
I was in the same situation about 1,5 years ago, when some security hole in Gmail allowed my account to be hijacked. The account was suspended automatically after the attacker had sent out 7 spam mails, and I could reset the account again without the attacker returning ever again. But, the ping ponging did go on for a while and Google did not bother.
I would just be worried about how they lock us out, what if the block both of our IP's and then the hacker manages to change his IP and can now get in and has an advantage over the legitimate account holder.
Good idea I think! Lock it after 2 account changes for a few hours at the very least, then the original owner will have it back, who is most likely the legitimate owner.
Rule #1 when publicizing security incidents: always publish something else to the blog within 1.5 hours so that the security incident isn't the top post.
Edit: semi-tongue-in-cheek per comments below; as a CloudFlare customer I went to the blog when this first came up expecting to see something but bounced when the first post was a discussion of SSL BEAST since that was the hotness back in the fall of 2011.
I do believe it was not planned, but I also feel that vulnerability disclosures should be pinned for a while somehow if possible. I think one way this is done is having a separation between 'new feature' blog and 'ops' blog.
Haha. Wish we were that organized. We have a big announcement on Wednesday and need the announcement about Polish (http://blog.cloudflare.com/introducing-polish-automatic-imag...) and the feature we're announcing tomorrow (Mirage) to come out before then.
I think being among the top stories on Hacker News will take care of people seeing it. And, for the record, I voted the breach story up.
Hey, just wondering, did someone make that infographic by hand or is there some software that does it for you? EDIT: Made by a graphic designer, answered below.
And now I will shamelessly take this moment to request a few features related to account security :-).
* Alerting: SMS or e-mail notification when an unrecognized device logs into my account or when records in my domains change.
* 2-factor Authentication: Prompt for a code delivered via SMS, e-mail, Google Authenticator, or DUO Security to login from an unrecognized device.
* Login Accounting: Let me see what IPs logged into my account, when, geoip info for each, and preferably what actions they took while logged in. Provide an API for this info so I can write an automated script to analyze it for suspicious events.
If you end up making any of these features, it would be cool to open-source a library you used to do it. There are a bunch of large SaaS providers out there that use features like these but they're all homegrown implementations afaik.
Btw, the Google Apps Admin Audit API exists but I have never seen anyone do anything with it and it makes me sad. A few hours with [name a scripting language] and you could probably have a pretty robust Google Apps monitoring system, but no one seems to care:
https://developers.google.com/google-apps/admin-audit/get_st...
That seems kind of shady, I mean sure it sucks that it happens and it is kind of bad to have it as your top post but publishing just because you don't want it to be the latest post kinda downplays the publicity factor of being honest about what happened.
I'm sure j_s's comment was meant as tongue-in-cheek and not as an actual criticism. I have every confidence that eastdakota's response (http://news.ycombinator.com/item?id=4066982) indicating that it was just a coincidence is true.
Part of this attack bears a certain resemblance to the recent Bitcoinica compromise, in that, from what I understand, they were also forwarding admin emails to personal accounts, one of which was compromised leading to the attacker gaining control of the bitcoinica virtual servers. Cloudflare were fortunate - at least the changes made were reversible, whereas the bitconica compromise resulted in the virtual servers being unrecoverably deleted after breach and theft.
There are lessons to be learned from both incidents.
Does anyone else get the feeling that the attacker is going to be someone the Cloudflare team knows? Firstly they would have had to have known Matthew's phone number. Then, assuming the attacker always had the plan of disrupting the target site, they would have had to have known that the password reset mails were BCC'd to admins.
Getting someones phone number seems pretty insignificant compared to using a previously undisclosed google security flaw.
And it's probably safe to assume that once you control the admin email account for a site, it's game over. You could request resets from other providers
O.o This reminds me so much of the hack sequence from the game Uplink. The game was based on "hacking" but intentionally used hacking techniques from Hollywood :) It was pretty fun. Anyway for the highest-level targets, you had to get a voiceprint from the phone of an admin, crack the password on the box, and break the encryption while bypassing monitors.
Just found out: Uplink is on Steam, and in the Ubuntu Software Center now.
I too like it but it feels... I'm not sure how to describe it, but I felt like I was having to concentrate to read it properly and keep track of what point I was at. I'm not sure if that's a problem with me or the format, but trying to read it properly was less enjoyable than just scanning it.
That makes a lot of sense and could have been what happened. It would also make it more difficult for Google to do something like ignore responses that come after 4+ rings.
I'm more curious why a "secure" PIN is simply left, automated, as a message. A more "secure" option, I would think, would be to require some sort of input from the person who answered (say, "Press 1 for the PIN" where that number is randomized, or something).
The hack seems very well planned - I wonder just how many smaller sites have been hacked the exact same way as practice, and not picked up since they didn't have direct access to Google's security people?
I'd imagine you wouldn't blow a discovered flaw in Google's two-factor authentication setup on a small site. You'd sit on it until a big enough target came along, which CloudFlare certainly has become.
Pretty intense hack... I think Google gives you the option of setting up a recovery phone to receive voice or SMS messages. It looks like SMS may be more secure.
It also sounds like he didn't have 2 factor setup on his personal gmail account. I wonder if that would of helped.
I believe that once you have it set to use a phone number, it's always possible for the attacker to choose to have it call instead of sending a text. (I seem to recall it provides that option on the 2-factor login screen.)
So, the "Five Whys" analysis came up one short, eh?
Just kidding, this is a great level of detail and much appreciated, to understand CloudFlare's process and how to protect against or recognize these tactics elsewhere.
Is flaw # 5 (or #1 depending on how you look at it) not having two-factor auth on the personal account? Or does account recovery by-pass two-factor auth by design?
Sounds like if the hacker had just done it out of hours, perhaps when the person in question was asleep, they would have had uncontested access to the accounts and the hack might have been far more damaging.
I'm amazed Cloudflare got a response from Google that quickly. I'm a paying Apps customer and I don't see responses for 24 to 48 hours on security incidents, not to mention that Google doesn't have a stellar reputation when it comes to things like "support".
Goes to show, it's always who you know (or it's bullshit, which is less likely). Or I don't have enough users.
I'm surprised that Google doesn't detect two people fighting for control of one account. They could have easily detected ping-pong sessions and and locked both parties out of their accounts for a couple of hours. Or they could have penalized the newly added recovery address by forcing an exponential delay between resets using that address. This is not the first time I've heard of somebody breaking into a Gmail account while the account owner is using that very same account.