
The DDoS that almost broke the Internet - jgrahamc
http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet
======
ChuckMcM
In one of the earlier attacks I discovered I was running an open resolver on
my home network. I fixed it, and now just get a bunch of 'recursive lookup
denied' messages in my logs.

But the key here is the source. And this from the article:

 _"The attackers were able to generate more than 300Gbps of traffic likely
with a network of their own that only had access 1/100th of that amount of
traffic themselves."_

And this is key, so we could hunt them at their source if there was a way of
deducing their launch points (it may be a botnet but it may also just be some
random server farm)

I've got log records of the form:

    
    
       Mar 27 09:19:21 www named[295]: denied recursion for query from [212.199.180.105].61604 for incap-dns-server.anycast-any2.incapsula.us IN
    

Which suggests that 212.199.180.105 is somehow being used, and according my
latest GeoIP database that is an IP address in Tel Aviv.

    
    
       $VAR1 = {
              'longitude' => '34.7667',
              'city' => 'Tel Aviv',
              'latitude' => '32.0667',
              'country_code' => 'IL',
              'region' => '05',
              'isp_org' => 'Golden Lines Cable'
            };
    

So can we create a service where the recurse requests send the IP trying to do
the recursion to a service which then inverts the botnet/privatenet? Everytime
this level of co-ordination is undertaken it potentially shines a bright light
on the part of the Internet that is compromised/bad.

~~~
peterwwillis
DNS requests are UDP, so you can just spoof the source address (actually
that's what makes the attack work; you spoof the target as the source so
replies 10x bigger than the query go to the target). Nobody can really know
where they came from except at border routers of the source.

~~~
pedrocr
Does this mean that proper egress filtering would solve this? Because that
seems like a more feasible solution. Having a bunch of large ISPs and
datacenter operators block outgoing packets with sources outside their own IP
ranges sounds much easier than having many more home users reconfigure their
routers.

~~~
bodyfour
Yes, egress filtering is the _only_ solution.

I personally think that the focus on open recursive resolvers is misplaced.
All authoritative nameservers have to be "open" to queries for the domains
they serve. So instead of the amplifying by doing: $ host -t any random-
site.com. ns1.example.org. you can just as well do: $ host -t any example.org.
ns1.example.org. Even if all nameservers supported good per-IP throttling (far
from the case), there are still enough valid nameservers on the internet to
stage a decent amplification I think. So once all of the open resolvers are
shut down, the DDoS pricks will just target more important infrastructure to
accomplish the same goal.

It might be that we'll have to switch all authoritative DNS requests to be
TCP-only but I can't imagine what a pain that transition will be.

The worse news is that egress filtering has been something we've clearly
needed for 15+ years and it doesn't seem we've gotten very far. Part of the
problem is that these amplification attacks usually don't cause much pain to
the real source of the attack. Plus since it's very hard to tell where the
true source is, they don't even get publicly shamed for it. It's so much
easier to point a finger at the middle man in this case.

For egress filtering to be effective protection, it needs to cover nearly all
of the network. As long as a botnet can get their hands on a decent amount of
unfiltered bandwidth to amplify it's game on.

I'm not optimistic.

~~~
codexon
Egress filtering is not a solution.

I've had attacks come from Asia and Russia and none of the hosts respond. I've
also tried contacting their upstream providers.

The DNS system needs to be changed.

~~~
Daniel_Newby
Egress filtering would happen within a few days if someone DDoSed every open
resolver with other open resolvers. In fact, saturation of the outbound links
would effectively _be_ egress filtering.

~~~
jessaustin
Haha this would be kind of awesome. Obviously this would be bad news for the
open resolvers' home networks, but could their upstreams handle it (i.e. maybe
they would have to drop resolver traffic but not anything else)? If so maybe
this Cloudflare should prepare such a response for the next time someone pulls
this.

------
apawloski
Cloudflare always does an excellent job of optimizing their writeups for
large, diverse audiences. The prose of this article reminds me of an equally
accessible discussion of BGP from a few months ago [1].

[1] [http://blog.cloudflare.com/why-google-went-offline-today-
and...](http://blog.cloudflare.com/why-google-went-offline-today-and-a-bit-
about)

~~~
thatthatis
Whether intentional or not, this is some of the best PR marketing copy I've
ever seen. Large, diverse audiences who are afraid of their blog/app/site
getting DDOSed can understand enough to know that Cloudflare has a credible
solution to that which they fear.

I really can't figure out from their writing whether this is altruism
accidentally leading to great PR or phenomenal PR leading to the appearance of
altruism.

~~~
SoftwareMaven
Good marketing is honestly communicating about a great product. Cloudflare has
good marketing because they have a great product and their communications are
honest. Marketing is only slimy when the product can't back up the boasts
(which, unfortunately, is way too often).

~~~
duaneb
MMM, I think that honestly communicating about a great product can be great
marketing. I don't think that all great marketing is honest, in fact some of
the best marketing makes people doubt their own intelligence, see the DeBeers
campaign to make a diamond engagement ring an American necessity by equating
it with a love that lasts forever (which is silly). Yet, I know for a fact I
will buy an engagement ring when I propose.

~~~
precisioncoder
Why would you buy a diamond ring even knowing that? Personally I opted for a
silver ring with an inlaid golden pattern, it was beautiful and unique and was
very well received by my wife. My sister also received an non-traditional ring
and loves it. Sure it can be a conversation starter but often that's a plus
not a minus. All in all I find something beautiful and unique is a wonderful
token of your love, if it comes from the heart I can't see any girl you'd want
to marry rejecting it.

~~~
philbarr
Exactly. Personally, I had to let my wife buy her own ring because I was
skint. If it's the right person the ring thing is kind of just a tradition
that can be observed as you wish.

In contrast, a friend of mine was ordered by his fiance that she would only
accept a ring costing thousands of pounds. It didn't work out. Lucky escape in
my opinion (especially as she stopped him drinking beer too).

~~~
precisioncoder
Heh, certainly sounds like it. Ideally imho a wife should be a partner in all
the trials and tribulations as well as happiness that life brings. Once she
starts giving orders it seems more like she wants to be a boss, that was
usually the point in the relationship I would try to gracefully bow out...

------
binary11
Yes, my DNS server is listed on openresolvers.org. Here's why:

I've a smallish home network, 5 machines, one of them running handful of VMs,
some devices (printer, scanner).

I wanted to have a local DNS server to name all these things, but mostly to
learn about DNS and how to set up Bind.

So i installed bind on a Debian machine, set up a local domain, promptly named
.fia.intra. As an added benefit, I now had a local DNS caching server too, and
since my machines use this as their primary DNS server, it needs to be
recursive and not just respond to queries for my internal fia.intra. network.

Now, all this is running on an internal 192.168.1.0 network, and bind is set
up to only respond to queries from 192.168.1.0/24, and I'm behind an ADSL NAT
gateway, so noone from outside should be able to query my internal DNS server.

I ignorantly assumed that the ADSL modem wasn't completely broken and having a
moronic way of operating.

Now, I've _not_ set a port forwarding rule in the modem that forwards port 53
to my internal DNS server, the only port forwarding rule I have is one for
SSH.

However, I have this setting on the ADSL modem:
<http://i.imgur.com/dlL9LKV.png>

The ADSL modem as shipped from the ISP acts as DHCP server on the LAN side, as
most modems would do, and by default the DHCP server hands out a DNS server
that is my ISPs DNS server. I changed that to my internal DNS server,
192.168.1.20.

In the image you will see the DHCP server isn't even enabled, I moved that to
my same Debian machine and turned it off on the ADSL modem, but didn't erase
the DNS settings.

As it turns out, because of that setting the ADSL modem listens on port 53 on
the WAN interface(which has a routable IP address), and forwards/reverse-NATs
queries to my DNS server at 192.168.1.20. I'd never guessed it to do that.

I did a "dig google.com @<my.public.ip>" from an EC2 instance I have, and
indeed it responded nicely..

I've now changed the setting to read "Primary DNS Server= 0.0.0.0" and have
verified I no longer respond to DNS queries from the WAN side.

Stuff sucks.

------
tptacek
Worth pointing out: I'm presuming that the 300gbps of reported traffic† was
not generated by DNSSEC resolvers, because DNSSEC isn't that widely deployed.

Which is bad, because DNSSEC dramatically increases the amplification effect
you get from bouncing queries off open resolvers (the DNSSEC RRs are Big).

 _Adam Langley notes on Twitter that Cloudflare reports 3k DNS responses,
apparently containing the zone contents of RIPE.NET; I guess these were EDNS0
UDP AXFR requests? That's worse than DNSSEC._

This has been one of Daniel Bernstein's big critiques of DNSSEC. It's not one
of mine, but I'm still happy to see his argument validated.

† _(at a tier 1 Cloudflare doesn't have a business relationship with, which
makes this kind of a "my cousin's best friend told me" number, but still)_

~~~
morsch
How does this validate DJB's criticism? He's saying we shouldn't adopt DNSSEC
because of traffic amplification. But we're _already_ in an untenable
situation with the amplification caused by bog standard DNS -- according to
your quote, it's even worse than DNSSEC -- so we need to solve the problem
anyway.

~~~
tptacek
I am batting .000 on analysis for this situation and debated removing the
comment and just replacing it with a link to DJB's talk. We don't have all the
details but there's a notion that the attack already implicates DNSSEC because
of RIPE's RRSIG records. I don't know, either way.

~~~
davidu
Correct. DNSSEC just makes amplification worse.

It's like: Do you want to be shot with 50 bullets or 100 bullets. DNSSEC is
100 bullets, regular DNS might be 50 bullets. Either way, you're going to die.

I don't like DNSSEC, but amplification isn't my argument against it. The fact
that it doesn't provide encryption, puts keys in the wrong hands, and is
bizarrely complex for reasons that don't fit with the model of DNS is why I
don't like DNSSEC.

~~~
floody-berry
DNSSEC amplification is still pretty high. When I ran dnssecamp[1] last year,
I got similar numbers to the example run cited (2000+ servers providing 30x
amplification, scaling up to 95x amplification for the worst offenders).

[1] <http://dnscurve.org/dnssecamp.html>

------
Bootvis
News is that the dutch hoster CyberBunker[1] is responsible[2] for the attack.

[1]: <http://en.wikipedia.org/wiki/CyberBunker>

[2]:
[http://translate.google.com/translate?sl=nl&tl=en&js...](http://translate.google.com/translate?sl=nl&tl=en&js=n&prev=_t&hl=nl&ie=UTF-8&eotf=1&u=http%3A%2F%2Fwww.nu.nl%2Finternet%2F3382530%2Fstrafrechtelijk-
onderzoek-aanvallen-nederlandse-cyberbunker.html&act=url)

~~~
chiph
I'd still keep an open mind -- it could just as easily been someone seeing the
opportunity to deflect blame away from themselves.

~~~
speleding
Well, it's their own spokesman admitting it. Although they later rescinded it,
probably after realising that there could very well be legal implications to
it.

------
jre
As a programmer with little knowledge of internet-scale networking, this was a
very interesting read. Thanks !

------
DangerousPie
So, given that there is already a list of open resolvers and the problem is
that they can be used to DDoS a server - why doesn't someone just make them
attack each other? From what I have read one could easily forge packages
appearing to come from DNS A and send it to DNS B-Z. Rinse, repeat and take
down the servers one by one.

Obviously this is probably illegal, but there would definitely be a beautiful
irony to it. :)

~~~
MostAwesomeDude
These open resolvers are largely run by ISPs and resolve addresses for their
customers. Taking them down would cripple the Internet access of their peers.

They do not need to be taken down; they need to be reconfigured. An open DNS
resolver is (arguably) misconfigured, not malicious.

~~~
goldfeld
By being made public they can be used maliciously by third parties. If such
ISPs are slow to respond and reconfigure, they are a liability to the whole
internet and the fault is theirs. A suggestion such as GPs, while probably
illegal, sounds like an interesting way to hasten them a bit. Better to have
their customers on their heels urging them to get a grip, than the whole
internet in jeopardy.

------
alanbyrne
This is why I pay CloudFlare each month. They repeatedly publicly show that
they know exactly what they're doing - and they do it without any sense of
smugness.

------
lucaspiller
I'm a bit confused about the 'open resolvers' bit. I searched for the static
IP range assigned by my ISP, and a number of results came up:

[http://openresolverproject.org/search.cgi?mode=search4&s...](http://openresolverproject.org/search.cgi?mode=search4&search_for=109.224.131.0%2F24)

This range has a description of "Static IP Pool for xDSL End Users", so is it
also home users who have open resolvers?

~~~
nwh
Yep, my ISP has three open resolvers in my assigned range, and another 6 in
the alternate range. It it grounds to give them a slight prod?

~~~
noselasd
Your ISP doesn't have 3 open resolvers. Your neighbors have.

Your ISP could terminate the contract with those 3 customers, but they won't -
they're customers. They could block inbound trafic on port 25 to those 3
customers (if they knew about it), but they have no incentive to. Or they
could block all inbound traffic on port 25, which will likely break DNS for a
lot of customers.

~~~
baq
you meant port 53, i presume?

~~~
nadinengland
I don't believe he/she did as for my ISP blocked port 80 and 443 making me
resort to port 25 to send emails~

------
richardjordan
Doing a DDoS attack in the cause (however questionable the commitment to that
cause is let's put it to one side for now) of Internet freedom is a ridiculous
strategy. The more this sort of thing becomes inevitable the more TPTB will
clamp down on such things and eventually we'll find ourselves on an Internet
with far fewer freedoms and it'll all be far more locked down.

Whether you like it or not society tends to react like high-school - when
enough people abuse a privilege eventually that privilege gets taken away. You
can argue that a free Internet is a right (as some do) but you won't win that
argument in the public sphere if that right is used to stop everyone else from
getting done what they want to do online.

~~~
qu4z-2
I suspect the people DDoSing spamhaus aren't doing it for internet freedom.

~~~
np422
Or perhaps they are?

We really need places like cyberbunker to keep internet free and open.

The day the the last piece of w4r3z, pr0n and other 1337 stuff is taken away
from the internet the infrastructure would have to be in place to pretty much
remove anything you want at will.

I wonder what will be next thing to get removed after that?

"First they came..."

~~~
driverdan
Cyberbunker hosts spammers. They're free to make that choice. Spamhaus runs an
IP blacklist of known spammers and put Cyberbunker on that list. They're free
to make that choice. Companies who don't like spammers subscribe to the
blacklist and use it to block offenders. They are free to make that choice.

I don't see where the internet is becoming less free and/or open.

------
davidschein
Not nearly almost: [http://gizmodo.com/5992652/that-internet-war-apocalypse-
is-a...](http://gizmodo.com/5992652/that-internet-war-apocalypse-is-a-lie)

~~~
notatoad
I don't really know about these things, but I know enough to trust cloudflare
over gawker.

------
Ntrails
Thanks for a really cool lesson about the nature of the internet :)

I was, however, interested to see no mention whatsoever of cloudflare in other
reports of this[1]. Is this something that bothers you?

[1] e.g <http://www.bbc.co.uk/news/technology-21954636>

~~~
eastdakota
No. When we're doing our job right, no one should know that CloudFlare even
exists.

------
eah13
"..but first a bit about how the Internet works"

My favorite part of a Cloudflare post.

------
kevinburke
What are the incentives for the maintainers of open DNS recursors? How can we
alter their incentives so that they can no longer be used in DNS amplification
attacks?

~~~
bradleyland
There are some benefits (incentives) to running your own DNS server with the
ability to perform recursive queries, but best practices dictate that these
servers should only accept queries from "trusted clients". So, it's not so
much that there are incentives to run an _open_ recursor as it is there are
very few negative incentives to running an _open_ DNS recursor. I keep
highlighting the word "open", because the open state of a DNS server is often
not an intentional decision.

I'd speculate the most common reason for using DNS recursion is to allow a
non-authoritative name server to return results for any query. This non-
authoritative name server usually sits on a network that serves clients with
low latency and high bandwidth, like a LAN. The network is typically private,
but private is not always the same as secure/closed. Some benefits of running
your own DNS are:

* The ability to cache DNS lookup results (speed increase)

* Placing the name server closer to clients (speed increase)

* The ability to blacklist certain zones (security)

And many more, most of which relate to control, speed, and security.

The thing is, none of these advantages are related to running an "open" DNS
with recursive queries enabled. I think the core problem is twofold:

1) Many amateur sysadmins don't recognize that running a name server with
recursive queries enabled is a security issue.

2) Enabling recursion doesn't _automatically_ require any configuration to
secure the client-trust relationship.

Unfortunately, I'm not really smart enough to propose any changes that would
help the situation, but I think this represents a high-level overview of the
most common problem scenario.

EDIT: It's also worth noting that some DNS servers enable recursive queries by
default. Anyone running their own DNS for their zone, but who don't have
knowledge of the issues related to recursive DNS will likely be running an
open DNS recursor as well. These are commonly servers at web hosts, which have
much faster internet connections as well. So it's a matter of getting everyone
on board for changing defaults.

~~~
rip747
why not just use opendns.com? you get all of these `benefits` and more without
you having to configure and maintain a DNS server.

~~~
Cymen
One reason I don't use OpenDNS.com or Google DNS is that it means I might not
hit the optimal CDN for me. Maybe they have improved things but last time I
used it, I often ended up hitting CDN end points that were not optimal. For
example, downloading MSDN content was slow. If I switched to my internet
service providers DNS servers, I would hit a more optimal CDN end point that
was faster. I had a similar experience with Netflix.

Note my example with MSDN took place a while ago so it might be replicable
today.

~~~
halfasleep
It might be worth trying again, at least in theory, you should get correct CDN
endpoints whatever happens. I suppose there might be an exception to this if a
CDN has edge nodes within your ISP though. There's a bit more detail at
<https://developers.google.com/speed/public-dns/faq#cdn>

~~~
Cymen
My solution is to use my provider's DNS servers first and Google DNS after
those.

------
sakopov
This was an immensely interesting read even though only high-level details
were discussed. Always impressed with CloudFare architecture. Thanks for a
great read!

------
pragmatic
Did I miss something in your article (I skimmed), who are the gentlemen in the
photo at the top?

~~~
jgrahamc
I didn't write TFA, but it appears that that is an image of the group Massive
Attack.

~~~
solarflair
After seeing the headline and the photos, I kept expecting to read that it was
some sort of composite "sketch" of suspects in the DDoS attack compiled from
real photos. I am not familiar with what the members of Massive Attack look
like and did not make that connection. Seems like poor photo usage.

~~~
error54
They were trying to be clever. "Largest DDoS attack"..."Massive Attack". But I
love Massive Attack but yes, most people wouldn't recognize the muscians and
assume that it was the suspects behind the attacks.

------
drchaos
I'm not a networking expert, but how would turning off recursive DNS queries
mitigate this kind of attack? A nameserver must still answer queries for the
domains it is authoritative for, so what prevents the bad guys from using only
authoritative queries for their attack? Wouldn't it be much better to just add
some rate limiting to every DNS (recursive or not)?

On a side note, I think that especially in times where messing with DNS is
used as a censorship tool by a lot of governments and regulators, there is
some value in being able to ask someone else's DNS for any domain, but that's
a different issue.

~~~
uxp
I'm not a networking expert either, but I do believe the recursive nature
turns this amplification DDoS to just a "normal" DDoS where the only target
would be the nameservers of the target service. At the end of the day, if
there wasn't a single nameserver that was turning around and asking 2, 10, or
100 other nameservers the same fraudulent question they didn't have the answer
to, the only thing you could do with the same exact attack would be to take
out someone's authoritative servers. No collateral damage by slowing down the
internet connections of home users in California when the attack is being
targeted at some server in Germany.

~~~
drchaos
I don't think the nameserver (or the upstream NS) is the target of the attack,
but the (spoofed) IP which appears to be sending the queries:

* Attacker sends (small) query to a lot of resolvers, spoofing the source adress to be the IP of the target

* Each NS replies with a (large) response, thus flooding the target with a lot of data

As long as there is no rate limiting in the nameservers used for the attack,
this would work regardless of whether the answer is authoritative or not.

------
neumino
Thanks for the post Clouflare. It's way more interesting to read from your
perspective than from a random journalist.

------
dubcanada
I don't understand something...

How is 300GBPS a lot? If we take London which has 8million and say roughly
half of them are on the internet (4million). Wouldn't that mean that if
everyone was using 78kbps we would reach 300 gbps(roughly)?

I just don't understand how a tier1 or internet exchange router can only
handle 100gbps. That seems extremely low to me considering I have like 1mbps
for just my house?

~~~
lelandbatey
You have 1mbps, but you very, very rarely actually use all of that. It doesn't
actually really matter if everyone had 1Gbps internet connections, because
most everyone is requesting data so infrequently that the points that must
bear all the load can handle it.

Pretty much, 100Gbps is about 100 thousand times your internet connection. So,
a single 100Gbps line could handle 100 thousand people like you requesting a
1mb file at the EXACT same moment. Spread that out by even a few seconds and
that router can handle way more people than that 100 thousand.

This is a GROSS oversimplification, but the idea stands.

------
ancarda
I feel like there's a shocking amount of laziness and incompetence rife in the
industry. How else would so many open resolvers exist? It's like the thousands
of nodes with default passwords that were used for the IPv4 census.

How exactly do we combat this?

~~~
kfcm
Laziness? Possibly.

Incompetence? More likely.

Incompetence due to having 45 #1 priorities and developing a deep
understanding of secure configurations is a #3 priority? Most likely.

------
calbear81
I wonder what the political implications of this type of "collateral damage"
might be.

Governments have recognized the need to defend against direct attacks on their
networks and develop their own offensive attack capabilities from a national
security perspective but I haven't seen the same level or response to these
sorts of events.

When the damage spills over to impacting services that tens of millions of
people rely on and cause economic damage, should we treat this as the same as
an attack on our critical infrastructure (electricity, water, etc.) like an
act of terrorism? If that becomes the case, would it warrant the use of deadly
force against the attackers?

~~~
redblacktree
All interesting questions, and now quite pressing ones.

------
chisto
Today one guy in my local news (Monterrey, Mexico) talk about it, that "tech
guy" said that netflix and other services were intermitent or fail to access,
and also that this "war" were between cyberbunker and Spamhaus. I think this
is not completly true, but he spoke if this kind of things would be the next
kind of wars in the following years and recommendo to the people a good
antivirus and a firewall.

I really admire what cloudflare did, and help the not too tech guys to
understand how this things works, if cloudflare were promoting himself with
this posts well they need to eat and educate their children is normal their
behavior.

------
colbyolson
I thought this was an excellent writeup and I would like to learn more more.

Are there any recommended books on learning about the Internet/DNS on a global
scale?

~~~
devdas
[http://www.amazon.com/Cisco-Essentials-Press-Networking-
Tech...](http://www.amazon.com/Cisco-Essentials-Press-Networking-
Technology/dp/1587050412) (Networking) <http://www.amazon.com/DNS-BIND-5th-
Cricket-Liu/dp/0596100574> (DNS)

There are always the standards texts, Computer Networks by Tannenbaum and
Internetworking by Radia Perlman for the theory.

------
summerdown2
I understand the issues with DNS reflection, but why are open resolvers the
issue? Isn't the point of DNS to respond to requests with correct information?

Surely if random people can't connect to DNS resolvers and get information,
they can't surf the net either? Someone has to resolve DNS for people for the
internet to function, don't they?

~~~
yuliyp
DNS runs over UDP, which means the source of requests for information can be
spoofed. Also, the amount of data of a response is significantly larger than
the request, so you can use DNS resolvers to send significantly more data to a
victim than you yourself need to generate by sending DNS queries with your
victim as the source IP.

~~~
summerdown2
Yes, I understand the attack. My point is that somewhere, there have to be DNS
servers that respond to public requests ... otherwise the internet will not
work.

Hence, some DNS servers have to be open. By saying it's openness that's the
problem, we're blaming the victims, rather than the issue, which is that DNS
is flawed. Simply moving to TCP would be better, surely?

~~~
AdrenalinMd
UDP doesn't require a handshake hence is easy to spoof unlike TCP where a
full-duplex connection must be established for a successful connection.

~~~
summerdown2
Yes, that's my point. If we move to TCP we fix the issue. At the moment I
can't see how closing open servers is a real fix.

~~~
pixl97
Moving from UDP back to TCP on large packets is a mixed bag. TCP is slow, very
slow. At one time DNS packets were limited to 512 bytes and had to use TCP for
more data, but over time the number of UDP packets over 512 bytes increased
greatly. Going back to the smaller packet size would impact a large number of
users with longer load times, especially on wireless devices.

Closing open DNS servers isn't a real fix. The people who need to fix it are
the lest likely to have a clue there is a problem in the first place.

------
omegant
So, that´s why HN has been doing 503´s for the last week?, I am from Spain and
it has been failing all the time.

~~~
ceejayoz
HN isn't hosted on CloudFlare, and a slowdown wouldn't cause 503s like that.
HN's likely just having server trouble.

------
kronholm
From the article: "If the Internet felt a bit more sluggish for you over the
last few days in Europe, this may be part of the reason why."

Hm, as opposed to normal days, when our Internet is just normal sluggish. Not
that fond of that phrasing. And I must be bored to even comment on that.

------
senthilnayagam
I don't think the attack has stopped, it will come back even bigger and would
take down many networks.

Hope that kind of pubic outcry and media visibility will get the networks and
governments to take notice and fix the core Internet infrastructure of the
known vulnerabilities

~~~
senthilnayagam
[http://m.cnet.com/news/egypts-military-arrests-divers-
cuttin...](http://m.cnet.com/news/egypts-military-arrests-divers-cutting-
undersea-internet-cables/57576689) now it feels like a conspiracy to takedown
Internet

------
jwr
What I don't understand is why someone doesn't finally write a piece of
malware that destroys botnets? These unmaintained machines cause the entire
world so much grief.

It doesn't have to be mean and destroy data, just incapacitate the machines
and force the users to upgrade.

~~~
brazzy
Existing botnet malware already protects itself against removal or
interference from other malware and anti-virus software. And it usually has
auto-update functionality. It would just be extended to defend against this
new "anti-malware" as well.

Besides, if this hypothetical "anti-malware" actually incapacitated machines,
it would be highly illegal itself.

------
trumbitta2
And here's another interesting one from the past (7 / 13 root servers got
shutdown or blocked): <http://c.root-servers.org/october21.txt>

------
nitins
That Internet War Apocalypse Is a Lie! [http://gizmodo.com/5992652/that-
internet-war-apocalypse-is-a...](http://gizmodo.com/5992652/that-internet-war-
apocalypse-is-a-lie)

------
sikhnerd
I'm happy to see CloudFlare actually include some technical details in this
post, though as always, I'd be happy to see more. It's always more interesting
when we can follow along technically.

------
k__
"Tier 1 networks don't buy bandwidth from anyone, so the majority of the
weight of the attack ended up being carried by them."

"We're proud of how our network held up under such a massive attack."

wat?!

------
wglb
This site <http://www.internetpulse.net/> is useful to check if you suspect a
global slowdown.

~~~
lostlogin
I know very little on this subject, so please excuse my ignorance, but is that
a world list or US list - just reading the names makes it seem US centric.
Thanks

~~~
wglb
You are right--just US. Sorry for misdirection.

------
radio4fan
Great post, and thanks for introducing me to the Open DNS Resolver Project.

<http://openresolverproject.org/>

------
vermontdevil
No wonder I had difficulty accessing Google a few times during that time
period.

------
AdrenalinMd
Hold on, isn't it the time to force DNS to be TCP only ?

~~~
graylights
No. Currently a significant amount of latency in opening a web page is the DNS
resolution (for half a dozen domains). TCP adds a lot of overhead compared to
UDP.

But a case can be made that large requests should be TCP only.

~~~
lucb1e
> _TCP adds a lot of overhead compared to UDP._

Not if you simply keep a TCP connection open and use SYN cookies on the
servers. Many DNS servers already support TCP (though I'm pretty sure you need
a new connection for every request).

------
propercoil
nic.fr is down so most .fr domains are down due to unresolved hostname.

------
ateev23
Mine was OK. In India

~~~
munimkazia
Unfortunately, we in India are severely affected by the Mediterranean cable
cut, which coincidentally also happened at roughly the same time

------
criley
Gizmodo is calling out Cloudflare's claims that this affected anything more
than Dutch networks: <http://gizmodo.com/5992652>

They point out, for example, that the IX's in question routinely see 2.0+Tbps
peaks, so a 0.3Tbps attack would not be likely to shake a single IX, little
like "the internet" itself.

Interesting rebuttal, although certainly not comprehensive.

~~~
TranceMan
Indeed they should, Cloudflare offers to protect you [and charge you for the
privilege] from DDoS attacks like this.

If they _really_ know how the _Internet_ works they shouldn't make claims like
this.....

~~~
guiambros
That's called _marketing_.

They are crafting the stories methodically (and successfully I'd say, given
the number of times we all talk about CloudFlare on HN) to drive more exposure
to their brand. It's too optimistic and hyperbolic sometimes, but no really
different than any product announcement from Apple or Facebook.

In all fairness to them, a positive side effect is that we're all discussing
now how to solve the root cause of this decades-old - but still threatening -
problem (open networks and dns attacks). I guess it will really take an
apocalyptic event to convince millions of operators to configure better their
own networks.

This is not the first time CloudFlare uses hyperbolic headlines. Case in
point: _"Why Google Went Offline Today and a Bit about How the Internet
Works"_ <https://news.ycombinator.com/item?id=4747910>

~~~
ubercow13
How is that at all hyperbolic?

~~~
ceejayoz
Headline: "Why Google Went Offline Today"

Article: "Looking at peering maps, I'd estimate the outage impacted around
3–5% of the Internet's population."

