
A Note from one of Cloudflare's upstream providers - jauer
http://cluepon.net/ras/gizmodo
======
lsc
> This is definitely on the large end of the scale as far as DoS attacks go,
> but I wouldn't call it "record smashing" or "game changing" in any special
> way. It's just another large attack, maybe 10-15% larger than other similar
> ones we've seen inthe past

Heh. Nice. Yeah, I expressed skepticism that 300G/sec qualified as "largest
ever" - I mean, I personally have been hit by 10G+ attacks, and Cogent mostly
shrugged. (I mean, my cogent side was down until the target was blackholed at
the cogent border.) I know 10 gigabits is a lot less than 300 gigabits, but I
am a nobody compared to the people involved in this little kerfuffle.)

~~~
cft
Can you describe Cogent's reaction to your request to blackhole the route? We
had a very small incident with XO recently (perhaps 500Mbps reflected DNS
flood), and it took them 24 hrs to get back to me to blackhole the target! We
just packet filtered in our edge switch. Was Cogent any faster?

~~~
dsl
If you run a network of any size you should be speaking BGP with your
upstream, even if its using a private AS. You can then announce a prefix to
them specially tagged with a "blackhole community" that drops traffic at the
edge of their network.

The exact details vary by network, but here is how Hurricane does it
<http://www.he.net/adm/blackhole.html>

~~~
lsc
Yup. The details are slightly different for Cogent (I think we had it setup as
a separate BGP session rather than as a community tag like he.net does it, but
that was because my customer requested it that way.)

But yeah, you give them /32s to null, and they drop those /32s at the network
edge.

It stops the attack, well, almost immediately, but the problem is that it
kills the target site completely.

(well, often people have web frontends to this, which, well, work poorly when
your pipe is completely full, and for that matter, just getting the bgp data
to your peer can take a few tries. but yeah, it's still pretty quick and
effective, compared to calling someone to whine.)

What we really need is to get everyone to implement bcp38 anti-spoofing rules.
If everyone did that, we'd be able to block the sources of the problem, rather
than the destination. But, well, that's unlikely to happen, so for now, you
just, ah, finish the job.

~~~
eps
This is what I don't understand about DoS mitigation. So i'd null route the
target, but then I have to move the service to another IP. I will also
announce this IP to the customers, but that would simply redirect the DoS to
the new address and take it down too... I mean, there seems to be no way to
deal with DoS without throwing a baby (customers) with the water (attackers)
or am I missing something?

~~~
codesuela
Well services like Cloudflare or Blockdos try to mitigate this problem by
absorbing the bad traffic and filtering it so that it is spread out across
nodes and then dropped by firewalls with custom rules.

~~~
eps
I don't follow. How do they know which traffic is "bad" and which is legit?

~~~
codesuela
With ordinary DDoS attacks an effective method (which Cloudflare uses) is to
prompt you with an captcha before letting you pass or dropping your connection
when you fail often enough. As far as I understand it was not that hard to
block this attack because it follows traffic patterns (DNS responses from open
resolvers). The actual problem was that the attack was so massive that it
clogged the pipes before it could reach a router (belonging to Cloudflare)
that would've been able to drop the packages.

Disclaimer: I am no network engineer so don't rely on my reply being factually
100% correct.

~~~
eps
Captcha. Right.

------
e40
I never read the Gizmodo piece, because I think they're mostly trolls, but I
really liked this response. Kudos to him.

~~~
guelo
The Gizmodo piece was better journalism than the nytimes or bbc articles which
regurgitated Cloudfare's over the top PR uncritically.

~~~
devindotcom
I disagree. Calling the story a straight-up lie was a mistake, and link bait
to boot. I thought Biddle failed to understand what the risk to "The Internet"
truly was, and how an attack like this (based on a known architecture
vulnerability), if repeated or made more common or effective, could cripple a
tier 1 provider and cause serious routing issues, among other things.

------
zobzu
That was a very nicely worded response that echos what, I hope/believe, most
of us think.

Plus; that was in ASCII. Damn. :)

~~~
tiedemann
Word. Respects given to serving in text/plain (as is proper in this context).

~~~
edmond_dantes
Uber respect if the file had gzip content-encoding.

~~~
lucb1e
Or if the output was gzip without encoding header. The FBI will be all over
your place the next day with a warrant for the encryption key :D

------
nullrouted
For those of you who don't know Ras is one of the people that run/ran nlayer
(now part of GTT) which is Cloudflare's primary provider.

~~~
sargun
RAS has been pretty awesome in the community. He was actually the cofounder of
the company we called nLayer. He's given a bunch of very valuable "101" talks
at NANOG (think of this as the Hacker News for networking people), and until
the acquisition, he was pretty active on the mailing list.

------
polarix
What if this had been Akamai and not Cloudflare? Would we even hear about it?
Why is Cloudflare in the news regularly and never Akamai? Do they just like
drama? Is that good or bad for their business?

~~~
kyrra
Cloudflare uses it as advertising. "Look at this attack we mitigated for our
client!", which then gets picked up by HN and similar sites because Cloudflare
does write fairly good blog posts about it. As others have said, Cloudflare
does a fairly good job explaining parts of the internet that most of us don't
get to see, so it gets attention.

~~~
youngtaff
It gets picked up by HN as CloudFlare post it to HN themselves.

That said they're interesting articles if you ignore the hype.

------
finnh
> But, having a bad day on the Internet is nothing new.

That's my new quote of the year.

------
dudus
It amazes me how little I know about overral internet traffic infraestructures

~~~
kilovoltaire
Same here. Can anyone point to a good starting point for someone who
understands TCP/IP to start understanding the Internet topology?

~~~
WestCoastJustin
You might want to check out the NANOG mailing list. There is also a list of
Internet exchange points [1], these pages are packed with details about how
many of them function.

TCP/IP and Internet topology are different things. Think of it this way,
internet topology is the highway, and TCP/IP is a vehicle that travels the
highway (along with UDP, ICMP, etc).

[1]
[http://en.wikipedia.org/wiki/List_of_Internet_exchange_point...](http://en.wikipedia.org/wiki/List_of_Internet_exchange_points)

~~~
dsl
Just please don't post to NANOG when your cablemodem/DSL stops working. _sigh_

~~~
Swannie
I had to post a response to this:

LOL for 1 minute...

Also: don't post if your business DSL stops working. Or your ISP. But maybe if
you find your BGP from your upstream is poisoned/severely b0rken.

------
eastdakota
This is a great writeup and aligns with what we saw at CloudFlare. The most
interesting part of the attack was that the attackers went after the IXs.

~~~
eastdakota
PS - This was written by nLayer/GTT's CEO, one of CloudFlare's bandwidth
providers. They have been a terrific partner and were extremely helpful as we
mitigated this attack.

~~~
spiantino
Would you mind explaining this piece of the story in a little more detail?

"When the attackers stumbled upon this, probably by accident, it resulted in a
lot of bogus traffic being injected into the IXP fabrics in an unusual way,
until the IXP operators were able to work with everyone to make certain the
IXP IP blocks weren't being globally re-advertised."

It's pretty fascinating and I think most of the HN audience, myself included,
would be able to understand the actual technical detail.

~~~
rasatnlayer
Updated with a few more details for you (but still trying to keep it in
laymen's terms for those who don't do advanced networking). I wasn't really
expecting this thing to take off or get linked anywhere, it was just a dump of
the e-mail I sent this morning so I could link it to Facebook. :)

~~~
darkarmani
Thanks for the very informative write up!

I have to smile when people are praising you for the plaintext writeup whose
purpose was to link from facebook. It's like saying you finally got your house
fully off-grid using your hand made windmill that generates power so you can
watch the Kardashians. ;)

~~~
petenixey
This may be the only HN comment I've ever committed to Evernote. Well played
:)

------
malachismith
Fantastic response. Thank you so much for writing and sharing. Cuts through
all the spin and FUD effectively.

------
stevewilhelm
This incident drives home the fact that there is no one entity responsible for
"The Internet." It is run by a network of for-profit companies, governments,
and non-profit public and private standards bodies.

------
smackfu
>The next part is where things got interesting, and is the part that nobody
outside of extremely technical circles has actually bothered to try and
understand yet.

Isn't this talked about in CloudFare's write-up?

------
barkingcat
I think this just reminds everyone that as large as google, facebook, etc seem
to be, they are just a small part of this huge global network we humans have
created.

As large as Google, Facebook, Amazon, etc is on the web, the major telcos have
to be even larger (in terms of network size, capacity, amount of fibre,
switches, datacentres) in order to carry the traffic.

~~~
EricBurnett
That's only true to a point. In addition to the consumer-directed packets,
large volumes of traffic for Google and Amazon never leave their networks.
Shuttling data between datacenters for Google; moving data within the many
cloud services of Amazon; or transferring between the two companies (e.g. GCS
to S3). This means it's no longer a given that telcos must be larger.

The closer we get to living "in the cloud", the more our traffic can be seen
as a window into operations taking place within and between cloud services.

------
morganwilde
My first reaction to these kinds of news is fear. As someone who builds stuff
for the web, I really hate the idea of some malicious being trying to
purposefully ruin your work.

My second reaction is that of "bring-it-on". Basically this is an impulse for
improvement, and as with any major threat you either stand your ground or get
run over.

------
mpchlets
Nice response, nice sentiment, nice format. I often dislike the sensationalism
surrounding attacks or viruses - it makes people distrust and gives others
excuses for issues.

------
Paige
Richard you can hit me with your cluepon any day.

------
donavanm
Related, <http://cloudscare.com>

------
drakaal
Straight up Lie. Talking with Google Engineers, and Amazon Engineers they say
this is not happening. So unless someone picked a fight with CF, it is more
likely CF is just having a bad day.

