
Details Behind Today's Internet Hacks - dknecht
http://blog.cloudflare.com/details-behind-todays-internet-hacks
======
WestCoastJustin
> _Technical teams from CloudFlare, OpenDNS and Google jumped on a conference
> call and discovered what appeared to be malware on the site to which the
> NYTimes.com site was redirected._

On the HN post _" Google.ps domain was hacked (google.ps)"_ [1], HN user
_biot_ predicted this exact scenario, although not a zero day most likely. He
talked about submitting hacked sites to HN _"... and thousands of HN readers
get infected by a zero-day exploit. Maybe. If you're thinking of submitting a
known compromised site to HN, consider instead submitting a third-party site
which explains/documents the compromise. Ideally from a respected security
research company"._ [2]

[1]
[https://news.ycombinator.com/item?id=6278737](https://news.ycombinator.com/item?id=6278737)

[2]
[https://news.ycombinator.com/item?id=6279253](https://news.ycombinator.com/item?id=6279253)

~~~
Cyranix
I'd suggest extending the idea to non-responsive sites as well. Instead of
submitting a link to a company's homepage when they're being DDOSed or are
otherwise unavailable, submit a link to their status page if they have one or,
failing that, use a third-party indicator. Off the top of my head I can't
think of a third-party indicator that would capture point-in-time
availability, but a manually crafted URL like
[http://isup.me/example.com?1970-01-01T00:00:00Z](http://isup.me/example.com?1970-01-01T00:00:00Z)
would get the point across just fine.

~~~
joshschreuder
Maybe something like Zapier's API Status Board, though I'm not sure how
realtime and whether it only applies to site's APIs rather than their general
websites.

[https://zapier.com/status/](https://zapier.com/status/)

~~~
bryanh
It applies to their APIs specifically. It is realtime within ~5 minutes.

------
pavs
Basically zero information. They keep telling us how MelbourneIT is usually
more secure but doesn't do on to tell us how it is any more secure than other
registrars. More importantly, even with admin access to to their control panel
how can it be so easy to change registry information of such high profile
sites with a click of a button?

~~~
throwaway86
Devilishly clever marketing for Cloudflare, though. Clearly I need to spend my
days on more bridge calls for situations affecting other ops teams that have
nothing to do with me, so my company can put out a PR piece from a position of
authority about how awesome we are. What exactly did a team of people at
Cloudflare do today? Consult? Do you bill hourly or is it a friendly NYT
discount? What was your plan connecting end users with recursive operators?
Want them to manually flush their resolvers out of the normal DNS TTL
protocol? Is that a service that comes with my Cloudflare subscription?

Next time a startup goes down, ask yourself: if I were on a bridge call with
their ops team, could I use this to sell my company's reliability product?
Clearly, the answer is yes.

Classy, too, jumping out in front of MelbourneIT's response then speculating
on it. I would be furious about Cloudflare writing a details-thin
"postmortem," headlining it as a postmortem, analyzing my initial statement to
customers in it, then getting it on HN before DNS caches are even cold from
the incident itself. It's not even subtle.

This is the sort of thing I remember in discussions about using Cloudflare.
There's lots of choices for CDNs, a market growing surprisingly full of
ambulance chasers: one CDN startup had the fucking courage to email me
directly after a hellish multi-hour outage and say "want to set up a call to
discuss how our product could have prevented this outage?" I was still awake
from fixing the problem overnight and no, your CDN is not going to fix my
catastrophic DB failure. Get bent.

This is a disgusting move by Cloudflare. The little human network signoff made
me gag; don't forget, small ops teams, you will only get things done if you
know people. Notice HuffPo wasn't on the call? Exactly.

~~~
solistice
Didn't they pull a similar story telling people an attack on them by
Cyberbunker impacted the London Internet Exchange, prompting quite some
pandemonium?

I remember there being a more somber post after the whole incident by another
blog detailing just how little fluctations there were on the alleged day of
the incident, and how the numbers didn't stack up.

Cloudflare is tricky, isn't it?

~~~
jacquesm
This is known as 'inserting yourself in the news story' and it works well as a
marketing trick but in this case cloudflare is actually part of the story
because the NYT (one of the affected sites) and cloudflare did communicate on
the subject. The more peripheral the link the trickier it is, in this case (a
first order contact with the affected party which was initiated by cloudflare)
I think it is fine to issue some statement, but not necessarily this
statement.

~~~
jgrahamc
Not only did CloudFlare (where I work) and the New York Times communicate, the
CTO of the paper has said the following:
[https://twitter.com/rajivpant/status/372559771960098816](https://twitter.com/rajivpant/status/372559771960098816)

"I'm super impressed by the operations, incident/crisis management & expertise
of the @CloudFlare and @OpenDNS teams."

~~~
throwaway86
Before leaving my comment, I searched and searched for any shred of reason for
CloudFlare to release this inappropriate statement, including reading all of
Rajiv's timeline. Obviously, since I left the comment, I came up empty.

Can you point to what you feel makes this statement appropriate on behalf of
your company? I can't identify what annoys me most about it, because there are
many things: the "it's who you know in ops" attitude that I've been fighting
for my entire career, the creation of a Batman-esque hero at a startup CDN
provider who assembles a team to guide the lesser ops teams through a crisis,
the overdramatizing of a DNS hijack that happens countless times daily (just
with an interesting vector this time, but certainly not the first of ITS kind,
either), speculating on another company's statements, preempting an official
response with your own "postmortem" to score some traffic...

It's particularly frustrating because I've been in this exact scenario, to the
T and including a registrar compromise, before. But because my personal side
project doesn't have name pull, I didn't get a CloudFlare Crack Squad on
speakerphone calling in a dozen courtesy phone favors to score my contract.
And I had to wait for tickets and TTLs like _everyone else_. That sounds
bitter -- and I hate bringing it up for that reason -- but that's why this is
ethically shitty. Either you're playing favorites or capitalizing on something
for sales. There is no third option, not even an altruistic one.

~~~
eastdakota
No good deed, it seems, goes unpunished by those upset they're not getting
enough attention. May I suggest you read the end of the NYT CTO's recently
updated blog post:

[http://www.rajiv.com/blog/2009/12/10/tech-ops-
irc/#2013Aug28](http://www.rajiv.com/blog/2009/12/10/tech-ops-irc/#2013Aug28)

~~~
throwaway86
That wasn't remotely the thrust of my comments and you know it. I also
(correctly) predicted you would hop on the bitter swan song instead of, you
know, the half-dozen reasons why this sucks immediately prefacing it. Also,
that's two employees who have posted Rajiv's words as rationale for the blog
post; can we go for three? Shouldn't you be hiring Rajiv at this point, as
hard as you're riding him?

Address something smaller and bite-sized, like preempting MelbourneIT's
statement with your own and speculating on their behalf. Can you at least
defend that inappropriateness? Can we start there?

Your company provided guidance and connections, which makes this statement
inappropriate. Or did CloudFlare do something that has been left out of all
statements?

I am _not_ annoyed by your "good deed". I'm annoyed by how hard and how
inappropriately you are capitalizing upon it as a PR coup, before the ashes
have even settled. The victim tone is discouraging for this conversation, I
have to say, and it's quite unbecoming.

------
willvarfar
> At 1:19pm (PDT) today, a researcher noticed that the New York Times' website
> wasn't loading.

So if the content on the redirected page had been more subtle - for example,
mirroring NYTimes but editing stories etc - then things would have taken a lot
longer to have been noticed?

------
signed0
Are there any registrars that allow one to set serverDeleteProhibited,
serverTransferProhibited, and serverUpdateProhibited?

~~~
otterley
I'm not sure it would have helped in this instance. If the attacker got access
to the administrative interface for the registrar, all he'd have to do is
unset the relevant flag first, using the same interface, before changing the
name server records.

These flags are the functional equivalent of forcing you to break a piece of
glass before pushing the fire alarm button.

~~~
tjohns
Some registrars require you to send in a hardcopy request, along various forms
of ID, in order to clear these flags. I know mine (PairNIC) does.

It's not unspoofable, but it is an time consuming extra step that involves a
human on the receiving end.

 _Edit:_ I was thinking of the client(Update|Transfer|Delete)Prohibited flags,
which is a registrar lock. I'm not even sure how one goes about setting the
"server" version of those flags for a registry lock, but it's probably even
more complicated.

~~~
eli
And someone reviews that hard copy and flips a switch in an admin interface
like the one that supposedly got hacked?

~~~
vxNsr
Probably sends out a call to verify first.

------
holdenc
So, if my DNS is hacked, I can call Google and OpenDNS and have them correct
my records upstream? And then contact Verisign for a registry lock? And expect
a personal response from MelbourneIT (even though it's likely their reseller's
fault)? This is great news!

~~~
eastdakota
If you're the paper of record, yes.

------
martin_
The details actually look pretty sparse. I'm looking forward to MelbourneIT
letting us know the specifics (if they do!).

~~~
pfraze
Particularly the malware. Is this related to the Google Palestine hacking
yesterday? Somebody linked the hacked site and it hit top of HN, so a number
of us had to have clicked through to it.

~~~
tomrod
Uh oh. What was that HN link?

~~~
pfraze
[https://news.ycombinator.com/item?id=6278737](https://news.ycombinator.com/item?id=6278737)

------
damian2000
I'm amazed that Melbourne IT seem to be held in high regard these days. Going
back to the 1990s, they had a monopoly on Australian domain registration, they
charged the earth, and had really crap customer service.

~~~
WatchDog
I've only heard bad things about them.

------
alien_acorn
> The correct name servers should have been DNS.EWR1.NYTIMES.COM and
> DNS.SEA1.NYTIMES.COM.

How does this work? How would you get to DNS.EWR1.NYTIMES.COM without first
knowing where nytimes.com is?

~~~
zhoutong
Nameservers have their IPs registered with the registry, and they are returned
in the additional answers section. These are called "glue records".

------
nly
How would setting the registrar lock have helped in this case? The registrar
lock can be unlocked by the current registrar... which was the target in this
case.

It's good advice, but seems kind of irrelevant.

> It's worth noting that while some of Twitter's utility domains were
> redirected, Twitter.com was not -- and Twitter.com has a registry lock in
> place.

~~~
eastdakota
registry lock != registrar lock

The former is with Verisign and cannot easily be removed by the registrar. The
latter is with the registrar and can be removed by the registrar. In whois
status codes "clientXXX" = registrar lock (weak). "serverXXX" = registry lock
(stronger).

------
peterwwillis
I'll bet five dollars the credentials were stolen by a botnet the SEA runs or
has access to. You wouldn't believe the shit that pops up sometimes. (It's
also incredibly trivial to take over botnets run by jackasses who took a
tutorial in setting up Zeus) Less likely but still highly possible would be
spear phishing of registrar resellers.

Edit: I don't know why, but the nameservers I use don't resolve any address
for nytimes.com now. If I query 8.8.8.8 directly I get a response. So, could
be they're still suffering from this attack, which sucks.

------
agwa
> MelbourneIT has traditionally been known as one of the more secure
> registrars

They were one of the registrars compromised back in May as part of Hack the
Planet[1]. If I recall correctly, they were the only registrar where the
attackers actually got shell access on a server. That's when they lost any
reputation for security in my eyes.

[1]
[http://www.theregister.co.uk/2013/05/09/melbourne_it_hacking...](http://www.theregister.co.uk/2013/05/09/melbourne_it_hacking/)

------
dotBen
I don't think I understand why CloudFlare was involved - do they provide
services to NYT, it isn't clear from the post that they do.

~~~
throwaway86
No, they're the concerned citizen that performed first aid on the motorist,
then hung around to take questions from the media.

------
dibbsonline
Good to see the MelbsIT product using two factor auth.

