

Cloudflare Attack Post Mortem; Apparent Google Apps/Gmail Vulnerability - demonfly77
http://blog.cloudflare.com/post-mortem-todays-attack-apparent-google-app

======
ashconnor
The headline of this submission is misleading.

tl;dr; Hacker used social engineering to add a secondary recovery email
address to the victims Google account.

2-factor wasn't even enabled on the victims personal account but the hacker
didn't need a 2-factor code to reset the password on the victims company
account.

From the article:

Google reports that they discovered a "subtle flaw affecting not 2-step
verification itself, but the account recovery flow for some accounts. We've
now blocked that attack vector to prevent further abuse."

~~~
eli
How is that social engineering? It doesn't seem like human behavior played any
role in this at all. The hacker interacted only with automated systems.

This is a security bug that was exploited. It should not be possible to reset
a password without the 2-factor auth code, period. If that is possible, then
that the 2 factor auth is broken.

~~~
danielpal
When people say 2-factor auth code is broken, it implies there was an issue
with the 2-factor authentication code itself. That would mean things like RSA
and other Token based authentication might be at risk. So no 2-factor auth is
not broken. Authentication in Google is broken.

-disclaimer: I am the founder of Authy.com.

~~~
eli
Well of course the _concept_ of 2-factor auth is still sound. No inherent
flaws in the math or the theory. But this particular implementation is quite
broken.

------
sauteedbiscuits
"We have senior contacts at Google who we worked with in order to regain
control of the Google Apps accounts"

Boy, its nice they get that privilege. I love gmail, but god help me the day I
lose access to my account- I can only imagine the support whack-a-mole us
mortals would have to go through to get any resolution.

Anyone remember the "thomas monopoly" story from a while back where gmail
disabled the guy for weeks with no comment and the dude had to contact every
person in the world to get someone to look at it? (And then they reinstated
his account)

~~~
notatoad
Cloudflare isn't special. Any business (or individual) who pays for google
apps gets 24/7 email and phone support. I hate this perception that google has
terrible customer support. If you want support, stop being a free user.

~~~
kolev
I friend of mine got his Gmail deactivated (as you know, without any
explanation or means to dispute it). With it, hundreds of purchased Android
apps got lost as well. As there isn't much you can do, after a week, he
created a new account and had to repurchase most of the Android apps and
switch all his accounts to it. Some services require access to your old email,
so, he had to recreate some of those as well.

------
rdl
There are a lot of good reasons not to use gmail or google hosted apps
(performance, especially, but apparently also security). Two factor auth as
google implements it goes partway, but email really is the skeleton key to all
accounts. I'm not sure how meaningful Google Authenticator is as two factor
auth when you're actually using the iPhone or iPad to access mail much of the
time. Google Authenticator protects against untargeted mass attacks, but if
you are specifically a target, it's a pretty mediocre two factor solution. I
think it's fair to consider administrators of any major SaaS app to be
specific targets.

Unfortunately, there really isn't a great alternative. The next best thing
I've found is to just run your own mail server, which is pretty obnoxious if
you have any non-technical users to support. Kerio is the thing which most Mac
shops I know use (since people want directory services and the other stuff you
can't easily implement just from open source). Yet, there's better search on
server on gmail than from iOS to IMAP (i.e. Kerio), so that's a sacrifice.

I kind of hate Google for being the Craigslist of email -- a crappy product at
$50/yr which sets the maximum price people are willing to pay below the point
where anyone can offer a less crappy product. More like $250-$500/yr/mailbox
would be a supportable price for real email.

~~~
tommi
Please define "real email" and what justifies it to cost $250-$500/yr/mailbox
in your opinion.

~~~
Terretta
If you're questioning the grandparent's notion that Gmail is "crappy" compared
to expensive email products, I'd agree with you.

For me, the answer to what justifies a cost for email is deliverability. Of
course, having recipients receive your email promptly, and receiving theirs
promptly. But more importantly, receiving next to zero spam in my actual
inbox, and misclassifying absolutely zero ham in my spam folder.

The only solution I've found that accomplishes this at any price is GMail /
Google Apps.

Not only does it easily keep 12 - 18 year old email addresses that have been
on usenet spam free, it happily handled over 3000 spam per second per box
while letting through valid emails for sex.com. The owners could actually use
their addresses on Google Apps. They hadn't been able to on email systems
costing 4 and even 5 figures. Refreshing the spam label while seeing the inbox
stay empty, day after day and month after month, was astonishing.

~~~
rdl
Gmail does better than most alternatives for antispam. It's pretty good for
deliverability (although it's actually kind of bad for receiving mail from
semi-sketchy places; it doesn't even go into my spam box).

It's a slow UI (I often see 20-30 second page loading times on a fast network,
over SSL), the support on the free tier is horrible (i.e. non-existent), the
$50/user/yr support is below what I'd want for a company to rely on it (it's
ok as second or third tier support for your in-house people, I guess, but not
great frontline support for non-technical users when traveling).

The real killer, though, is lack of the enhanced features Exchange offers for
directory, scheduling, etc. I always hated people who depended on those kind
of things, but once I started using them, it made life annoying to not have
them. Google Calendar and Google Contacts are feature-incomplete and often
buggy (syncing to devices, other than maybe Android phones), especially when
you have multiple calendars, sharing, etc.

It's all fine for individual users, but for a group of 5-25+ people (startup,
pe fund, etc.), using google apps often sucks.

------
StavrosK
If I understand it correctly, the attacker managed to reset the password by
adding a secondary, fraudulent email account as a recovery option, and Google
just logged them in after the password was reset, not requiring a token from
the 2-factor system.

That's quite the attack.

------
runn1ng
Just an information - the unnamed customer was 4chan.org, and, according to
the group ... I think twitter, it was because they don't like My Little Pony
fans, that have a board over there.

Stupid internet drama indeed.

------
lomegor
This is really important news. It was mostly social engineering hacking, but
it seems to have been very clever. It's important to notice that it wasn't a
flaw in two factor authentication itself, but on the recovery options.

I really hate the recovery options. I don't know what they are there. If I
forgot my password, my recovery e-mail, I don't want to answer questions to be
sure it was me. Who knows how this exactly worked, but it sure seems to be a
problem with that logic.

~~~
joelhaasnoot
So it seems that there was a bug in the two factor authentication affecting
"some accounts" during the recovery procedure (see the update to the article).
Slightly worrying that the two factor authentication can be bypassed when
resetting a password.

~~~
boundlessdreamz
But even after resetting the password, two factor authentication is needed to
log in right? How was that bypassed?

~~~
monopede
The blog author didn't actually have two factor authentication enabled. The
headline is wrong.

~~~
Strom
You are incorrect, quote from the article:

 _all CloudFlare.com accounts use two-factor authentication. We are still
working with Google to understand how the hacker was able to reset the
password without providing a valid two-factor authentication token._

------
veverkap
Seems to me that there were two flaws evident here. One was Google's reset
flow, which they have since (hopefully) corrected.

The other issue, that seemed to be glossed over, was that they were BCCing
password reset emails to a compromisable account for support/debugging
purposes. I understand the desire to be aware of fradulent submissions early,
but we, as an industry, should be treating password reset emails (and tokens)
like passwords. Do not store them in plain text.

I appreciate CloudFlare's transparency on this, but feel they missed an
opportunity to address a flaw that probably exists in a large number of other
apps.

~~~
FuzzyDunlop
Copying the emails stood out to me as a ridiculously stupid thing to do. It
was probably a quick and dirty solution to save coding up a way to monitor
resets on the backend.

------
wpietri
The most interesting part to me is that somebody managed to get phone support
for GMail.

~~~
ericabiz
Google Apps for Business (a paid solution) offers phone support:
[http://www.google.com/apps/intl/en/business/upgrade.html#utm...](http://www.google.com/apps/intl/en/business/upgrade.html#utm_campaign=upgrade&utm_source=hc-
global-upgrade&utm_medium=et)

Since they were using their own domain name, that's likely what they were on.

------
demonfly77
as stated on Softpedia, UGNazi got access to all Cloudflare customers' IPs and
IDs, and they will sell these informations
[http://news.softpedia.com/news/CloudFlare-Reports-Breach-
UGN...](http://news.softpedia.com/news/CloudFlare-Reports-Breach-UGNazi-Takes-
Credit-273334.shtml)

------
druiid
So on one side Cloudflare is saying that they just compromised a single
account and on the other we have the hacker claiming that all data was
compromised... I guess at that point the question becomes not whom is telling
the truth, but what any Cloudflare users (myself included) should be doing
right about now?

~~~
notatoad
Presumably after the hacker compromised the CEOs email account he had access
to any account he wanted, as I interpret the blog post he was only able to use
that access to compromise a single account before being locked out.

Nobody is necessarily lying.

------
lsh123
The funny part - cloudflare still hosts DNS for UGNazi

    
    
       Domain Name: UGNAZI.COM
       Registrar: ENOM, INC.
       Whois Server: whois.enom.com
       Referral URL: http://www.enom.com
       Name Server: LEE.NS.CLOUDFLARE.COM
       Name Server: RUTH.NS.CLOUDFLARE.COM
       Status: clientTransferProhibited
       Updated Date: 29-may-2012
       Creation Date: 22-jan-2012
       Expiration Date: 22-jan-2013

------
citricsquid
Cloudflare supports hacking group by providing them services -> gets hacked by
them. Reminds me very much of what happens in every episode of 24, the
terrorists kill their accomplices after they're finished with them, it never
works out for anyone involved.

~~~
daeken
Which "hacking group" was supported by Cloudflare?

~~~
tedivm
Cloudflare doesn't necessarily "support" hacker groups- although their CEO did
a great talk at Defcon last year about keeping Lulsec up (which I consider
more of an activist group than a hacker group, personally). However, they do
very little to keep their network safe- they have no dedicated abuse team (or,
if they do, they're impossible to contact directly), are slow to response to
abuse requests (if they respond at all), and even when they do respond take
only the most basic steps to filter the issues (which takes literally minutes
for their account holders to work around, and they don't actually disable the
accounts).

Dozens, if not hundreds, of hacker and malware related sites use Cloudflare to
mask their origin and essentially use the other shared sites as hostages. If
you block one of the malicious sites then you also end up blocking legitimate
sites, and since Cloudflare goes out of their way to not take down sites (even
those hosting malicious programs- including the drive by exploits that install
them) it makes it very difficult to trace the malware back to it's source or
actually take it down and stop people from getting infected.

~~~
X-Istence
I'm hoping that with the coming of IPv6 many of these issues are going to go
away. At that point Cloudflare could give every customer their own /128 that
is static across the world (they route their /64 or /48 however they please
using anycast) and forward traffic as they have always done with caching.

Then it becomes trivial to block a single IP, since it is assigned to a single
customer you won't have any collateral damage!

------
demonfly77
this is the tweet where CosmoTheGod states that UGNazi hacked Cloudflare and
4chan <https://twitter.com/CosmoTheGod/status/208656791708508160>

------
quarterto
WHERE IS YOUR GOD NOW, JEFF? [1]

[1]: [http://www.codinghorror.com/blog/2012/04/make-your-email-
hac...](http://www.codinghorror.com/blog/2012/04/make-your-email-hacker-
proof.html)

~~~
stack0v3erfl0w
Have you read the article ? the 2 factor authentication wasn't compromised, it
was mostly a social engineering attack. Even if it was the 2 factor
authentication that was compromised, how would this be Jeff's fault ?

~~~
eastdakota
There was a flaw in the system that Google uses to allow the transfer of
control from an account. Two-factor authentication wasn't compromised itself,
but the attacker was able to bypass it to access it. That flaw, they tell us,
has since been patched on their end.

~~~
boundlessdreamz
Can you explain this? After the attacker reset the password of cloudfare.com
email address what exactly did they do? How exactly did they login?

