
Global Internet slows after 'biggest attack in history' - laumars
http://www.bbc.co.uk/news/technology-21954636
======
jgrahamc
This story doesn't mention that Spamhaus is protected by CloudFlare and we
took a beating from this attack. At some point I'm hoping the full technical
story about how the attack morphed from our infrastructure to Internet
infrastructure can be told.

Also, <http://openresolverproject.org>

PS Technical details: [http://blog.cloudflare.com/the-ddos-that-almost-broke-
the-in...](http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet)

~~~
tomjen3
What I am more interested in is their comment 'spamhaus should not be allowed
to decide what goes on the internet'.

I abhore censorship. Does Spamhause engage in it?

~~~
npsimons
It's not censorship; it's simple property rights and basic freedom. I don't
consent to someone else using bandwidth that I have paid for, space on my hard
disk, or my attention and time, for advertising. Hosting is cheap enough; they
can get their own damn website and opt-in mailing list. And spamhaus
subscriptions are completely voluntary and optional.

That being said, the problem with many BL's is that they are run by
incompetents or extremists. They usually either end up blocking things that
are not spam by accident (see lists of supposedly dynamic IPs), or block whole
subnets (sometimes entire ISPs) to try and "teach them a lesson." or blackmail
them into fixing the problem.

~~~
lazyjones
> That being said, the problem with many BL's is that they are run by
> incompetents or extremists.

Unfortunately, that includes Spamhaus.

It's a bit sad to see how many companies will blindly support such entities
because they've "heard" that they somehow help fight spam. As someone who's
had issues with them because of their badly configured hosts and shady
practices (e.g. using domains previously used by mail providers as "spam
honeypots", meaning anyone who emails someone with an old address can be
banned [all content mailed there is considered spam, regardless of what it
actually is]), I am disappointed (yes, looking at you cloudflare).

------
xSwag
Funny story from the Hosting company[1]:

 _"Before the break of dawn on a morning in April, a full SWAT team was sent
to execute a search warrant on CyberBunker's property."_

 _"It must not have occurred to the officers that the blast doors were
designed to withstand a 20 megaton nuclear explosion from close range. When
the SWAT team realized that the door was not being opened for them, they throw
flashbangs and take other actions to draw attention."_

\---

And from the NYT article:

 _“Dutch authorities and the police have made several attempts to enter the
bunker by force,” the site said. “None of these attempts were successful.”_

Haha, this is too funny.

More detailed article on NYT[2]

\---

[1]<http://cyberbunker.com/web/swat.php>

[2][http://www.nytimes.com/2013/03/27/technology/internet/online...](http://www.nytimes.com/2013/03/27/technology/internet/online-
dispute-becomes-internet-snarling-attack.html?pagewanted=all&_r=0)

~~~
peterwwillis
I looked up a video with an actual Dutch SWAT team and their uniform looks
different. The one in the picture is pretty derpy, and they're carrying what
looks like tiny medieval shields. I'm not convinced.

~~~
ohwp
The picture is clearly Photoshopped. Light in the background is coming from
the right, while light on the team is coming from the left.

Edit: talking about this picture: <http://cyberbunker.com/web/images/swat-
bunker.jpg>

Edit2: I think it's more difficult to determine if it's real than I thought.

~~~
knowaveragejoe
It's definitely photoshopped, look at the ground they're standing on, where it
fades into the wooded area.

------
unimpressive
I don't have much time to write this comment. But before I head to school I'd
like to posit that these sorts of attacks are largely our fault.

When I say "our", I mean the loose knit group of sysadmins, self proclaimed
"computer people", hackers, phreakers, security experts, and government
officials trying to quell the increasing lurch of botnets and malware that has
gone on since the Eternal September.

Botnets get big because users don't know any better, users don't know better
partly out of laziness, partly because they feel they _can't_ know any better.
I don't know of a single site I can point to and say "If you really give a
shit about not getting your credit card data stolen, go here." Instead as far
as I can tell the majority of users in this demographic have their needs "met"
by fraudsters selling bogus antivirus packages and weird proprietary
utilities.

If you want a computing environment that can survive open, it needs users who
can use open.

~~~
jiggy2011
I agree, the problem however is that threats are constantly evolving and
getting more complicated. Even most IT people don't understand the threats
properly (I know I struggle).

It used to be that could just tell people to install a security suite on their
computer and they'd be mostly OK. I don't think that's really true any longer.

You could also partly lay the blame at Microsoft's door in getting users to
start connecting to internet with an OS designed without any reasonable
security (Windows 95).

Now that we have operating systems with better security it's hard to change
people's usage patterns to take advantage of that.

~~~
jessriedel
If she has Microsoft Security Essentials installed, doesn't run unknown
software, and doesn't give out her password, what more could my mother, as a
layman, reasonably be expected to do? I understand there are all sorts of
complicated steps she could take if she had good intuition about sniffing out
bad guys, but she doesn't. Isn't the problem with the crappy software, not
her?

~~~
jiggy2011
The fundamental problem is that bad guys can come up with new attack methods
faster than we can educate or produce reliable user friendly software to
counter their methods.

Even a sophisticated user is just as vulnerable in many cases. If I give
personal information to a third party site that I presume to be trustworthy
(say a government site) there's no way I can know if someone is going to find
SQLi vulnerabilities in that site next week and exfiltrate all of that data.

~~~
peterwwillis
New attack methods are not the problem. If there is new technology there will
always be a new attack method. Right now the _existing_ attack methods are the
problem. Specifically, that technology is being developed using the same lack
of basic security standards and thus the same old attacks keep working.

SQLi should not be a thing. At all. It's the most trivial fucking thing in the
world to validate data before you use it in an SQL query, and people get it
wrong, every single day. Security isn't hard, it's just tedious.

------
lucb1e
I'm sorry, but what actually slowed the internet down and was the biggest
attack in history? It doesn't even make a dent on the charts of the Amsterdam
Internet Exchange (ams-ix.net): [https://stats.ams-ix.net/cgi-
bin/stats/16all?log=totalall;pn...](https://stats.ams-ix.net/cgi-
bin/stats/16all?log=totalall;png=monthly)

As wel as the daily stats by the way: <https://ams-
ix.net/technical/statistics>

Reading on through the article, they continue about Spamhaus. What's that got
to do with slowing down the internet? And "But we're up - they haven't been
able to knock us down." is factually incorrect, Spamhaus did go down. They're
winning in the end, but they did go down.

> _He added: "These attacks are peaking at 300 gb/s (gigabits per second)._

Source? 300gbps would definitely be visible, and I think I remember hearing
about something between 60 and 100gbps.

> _Spamhaus is able to cope, the group says, as it has highly distributed
> infrastructure in a number of countries_

AKA cloudflare

> _We can't be brought down_

We've seen that. Am I missing information or is this a lie?

~~~
TranceMan
_Source? 300gbps would definitely be visible, and I think I remember hearing
about something between 60 and 100gbps._

From Cloudflares response [1 - nice graph in blog] ~75Gbps extra traffic was
hitting part of their network.

Obliviously there would have been much more traffic floating around and
getting dropped by ISPs that have correctly configured their outgoing traffic
filters.

Many [not all] ISPs that were affected only have themselves to blame, the
'Internet' didn't slow down - the part that they are responsible for did - and
it was their fault....

1\. [http://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-
of...](http://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-offline-and-
ho)

------
derrida
Wikipedia seems to suggest that Spamhaus blocked a large chunck of
Cyberbunkers IP allocation, when the problem originated with a subset. I
suppose given the conditions, Wikipedia is perhaps not to be trusted, but it
does make me think less of Spamhaus.

 _In October, 2011, Spamhaus identified CyberBunker as providing hosting for
spammers and contacted their upstream provider, A2B, demanding service be
cancelled. A2B initially refused, blocking only a single IP address linked to
spamming. Spamhaus retaliated by blacklisting all of A2B address space. A2B
capitulated, dropping CyberBunker, but then filed complaints with the Dutch
police against Spamhaus for extortion._

<https://en.wikipedia.org/wiki/CyberBunker>

~~~
enimodas
Spamhous has a history of blocking much more than the offending IP's,
sometimes whole ISPs. It effectively makes the issue a high priority one for
the ISP, some call it blackmail.

example: <http://edpnetissues.blogspot.be/2012/10/22-october-2012.html>

~~~
TeMPOraL
For more examples of blackmail-ishy behaviour, see
<https://news.ycombinator.com/item?id=5450049>.

------
error54
_"We can't be brought down."_

I understand taking pride in your work but isn't bragging like this kind of an
invitation for more things like this to happen to Spamhaus?

~~~
ihsw
I was under the impression that their statement was less of a challenge and
more of a declaration as such:

"We mustn't be brought down."

~~~
phaus
You are almost certainly mistaken. The statement was followed immediately by
the claim that Spamhaus has the largest network of DNS servers in the world.

~~~
zinkem
I think that claim could also support that they "mustn't" be brought down.

------
ohwp
_"a Dutch web host which states it will host anything with the exception of
child pornography or terrorism-related material"_

Since it's a Dutch company I highly doubt they host anything illegal (as the
article implies). The same rules apply to them as they do to other hosting
companies in The Netherlands (and EU).

Related:
[http://en.wikipedia.org/wiki/Onafhankelijke_Post_en_Telecomm...](http://en.wikipedia.org/wiki/Onafhankelijke_Post_en_Telecommunicatie_Autoriteit)

------
belorn
_"These attacks are peaking at 300 gb/s (gigabits per second)._

Is that around like 3000 compromised computers? Maybe 2-5 botnets worth? I
might be a bit off on the prices here, but that sound like maybe ~$1k/day on
the market? would be nice to get a price tag on the "'biggest attack in
history'".

~~~
criley
3000 computers means each one is putting out a full 100Mb/s, which unless
those 3000 computers are in data centers seems unlikely.

Seems like 30,000 nodes at 10Mb/s would be more likely?

But I don't have experience in botnets, just curious.

~~~
random_ind_dude
From what I understood, these attacks used DNS amplification. I am no expert
on botnets either, but here is the basic idea: they basically send a small
request to a DNS server with the source spoofed. The server sends a much
larger response to the spoofed source, which in this case is Spamhaus. This
happens on those DNS servers that don't check whether the request originated
from inside their own network.

So the botnets involved don't have to send 300 Gbps of traffic to Spamhaus.
The DNS servers being much more powerful will take care of that. I have no
idea about the going rate for a botnet, though.

~~~
entropy_
I think the onus for this one is on ISPs who don't properly filter outgoing
traffic. It's pretty simple, really, you have a block of IP addresses you
allocate to your customers, any outgoing traffic with a source outside of this
block should be dropped. A simple iptables rule on the router handling that
block would suffice.

There is no legitimate use case for sending traffic with a spoofed source IP.
I'm simply amazed that ISPs who should have the technical knowhow still
haven't eradicated all kinds of network attacks that rely on spoofed source
addresses(of which DNS amplification is only one).

------
garretruh
Relevant blog post from CloudFlare:

[http://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-
of...](http://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-offline-and-
ho)

------
ck2
[http://www.nytimes.com/2013/03/27/technology/internet/online...](http://www.nytimes.com/2013/03/27/technology/internet/online-
dispute-becomes-internet-snarling-attack.html?pagewanted=all&_r=0)

 _Cyberbunker brags on its Web site that it has been a frequent target of law
enforcement because of its “many controversial customers.”_

 _“Dutch authorities and the police have made several attempts to enter the
bunker by force,” the site said. “None of these attempts were successful.”_

If this happened in the USA - the police would never leave - they'd call in
tanks or bunker busters from the military.

Did the Dutch just turn around and go away and say "oh well" ?

~~~
tomjen3
This isn't just any bunker. It is a 5 story cold war bunker, with blast doors
designed to take a hit from a freaking 20 megaton bomb at 5 km.

The US may have a bunker buster that can open that thing, but it would kill
everybody inside, and possibly around it. I am not quite sure that the US
bunker busters of that size are non-nuclear either.

Would the US FBI really drop a nuke to serve a warrent? That seems, excessive
and counter productive (as you would destroy whatever it was you wanted).

~~~
ck2
Bunker busters are not nuclear. They use turrets from tanks made of depleted
uranium which is unbelievably strong - dropped from altitude to piece just
about any underground dwelling and deliver a non-nuclear bomb.

I have no idea how I know or remember this, I must have read it somewhere but
I am very anti-war, so _why_ I know it is strange to me.

------
jobigoud
"In this case, Spamhaus's Domain Name System (DNS) servers were targeted"

I'm not sure to understand why this should slow down the whole internet. It
seems to be only for email filtering, not for the web, and only those ISP that
use their service should be impacted, and only when their DNS cache is not
triggered. Am I missing something ?

~~~
TranceMan
_"In this case, Spamhaus's Domain Name System (DNS) servers were targeted"_

This is blatantly wrong, the DNS system and poorly configured networks were
used to target and attack Spamhaus.

------
curveship
Here is a (claimed) account of the dispute from the attackers:
[http://stophaus.com/entry.php?5-The-Real-story-on-the-New-
Yo...](http://stophaus.com/entry.php?5-The-Real-story-on-the-New-York-Times-
Article-and-all-the-SPAMHAUS-stuff) .

I got there from the comments on the CloudFlare blog posts, where a user named
"STOPhaus" posted taunting CloudFlare, with a link to the stophaus.com
website. Apparently it's a meeting place for anti-Spamhaus sentiment and
includes such classy stuff as personal information on Spamhaus employees.

Wild stuff, and thanks to CloudFlare for their writeup.

~~~
spangborn
I wonder if they realize how unreadable those posts are - they look like
mindless ramblings. It took me a few minutes to realize it was an
interview/chat.

------
srj
I realize it's a news headline but I feel obligated to point out that this
doesn't rank as the 'biggest attack in history' except maybe for Spamhaus. It
doesn't appear to come close to what the SQL Slammer, Blaster, or Sasser worms
did. Slammer in particular increased latency globally and the impact was
easily visible to anyone browsing the web.

------
byerley
'Security experts' are full of it. Root DNS servers have seen attacks on the
same scale regularly.

------
Nux
RBLs are a bad idea, they often end up in abuse.

To this day - with v4 exhausted and despite numerous delisting attempts - I
have a /21 listed in Sorbs because it happened to be in the past part of an
ISP's /18 dynamic range for customers.

They deserve all that's coming to them and more. Too bad other's get affected
in the process.

~~~
lazyjones
> RBLs are a bad idea, they often end up in abuse.

Indeed, all they should be used for is perhaps 0.5 points worth of
spamassassin weight. Sysadmins who use spamhaus as a blacklist are just as
incompetent as those who bounce virus e-mails to the address in "From:" ...

~~~
Nux
Hehe, true. It takes 2 ...

------
hhw
When it comes to DoS attacks, bandwidth is a much less meaningful metric than
packets/s. 300Gbps could be anywhere from 200,000,000 PPS to 4,687,500,000
PPS. High bandwidth attacks just cause congestion issues, while high packets/s
actually take networks and servers down.

------
InclinedPlane
It's great that all of our criminal justice resources in the US are dedicated
to stopping real crimes like TOU violations of academic journals instead of
things like investigating and stopping industrial espionage, sabotage, etc.

------
zozu
This discussion is quite intense. Frankly it's scary that a single attack
manages to slow down popular sites like Netflix. What consequences for future
attacks will this attack have?

------
darkarmani
Now we have a list of every zombie in their arsenal. Is that information we
can use to reduce their impact in the future?

~~~
MertsA
No because it was a DNS amplification attack so it was a bunch of DNS requests
with spoofed source IPs. Best way to reduce their impact in the future would
be to threaten to null route ISPs that don't do anything to stop spoofed
packets.

------
AdamN
Exactly why is this affecting non-spamhaus services? Is it just shared dns
servers or actual IP traffic being throttled ?

~~~
jiggy2011
A lot of spam filters rely on spamhaus.

~~~
leethax0r
This is true. However the reason this affects non-Spamhaus servers is because
there is so much traffic that it is literally clogging the backbone.

------
rikacomet
for the biggest attack in the history, its kinda looking boring from here
(India). :/

1 billion+ people won't hear of it much.

I would be interested to know, what did Spamhaus paid google to use its
resources, and would such a type of cooperation on global scale means the end
of DDoS in the long term?

------
dreamdu5t
This is all advertising doubling as news, seeded by CloudFare.

------
jussij
Really? I never noticed.

My internet down here in Oz has been as slow as ever!

------
orenmazor
the internet is a series of on/off ramps.

------
OGinparadise
The main problem is that some people decide what's good and what's not online
and paint with the broadest brush possible. Spamhaus, sadly I say, is used by
a lot of providers as gospel and a lot of innocent sites are hurt.

~~~
leephillips
You seem to be taking the line of the attackers' spokesman, who accused,
rather hysterically, Spamhaus of deciding what goes on the internet. Of
course, all Spamhaus does is supply a list of hosts who are sending email
spam, and other things like lists of dynamic IPs. Sounds like this hosting
outfit was making money hosting spammers and their business is threatened by
legitimate countermeasures.

~~~
TeMPOraL
He's not the only one to do so. Spamhaus has engaged in some shady behaviour;
even pg wrote about it once:

<http://paulgraham.com/spamhausblacklist.html> (2005)

 _I wanted to believe him. But before I could reply to his mail, I got first-
hand evidence that the SBL has in fact gone bad._

 _As of this writing, any filter relying on the SBL is now marking email with
the url "paulgraham.com" as spam. Why? Because the guys at the SBL want to
pressure Yahoo, where paulgraham.com is hosted, to delete the site of a
company they believe is spamming._

EDIT

Wait, there's more!

<http://paulgraham.com/sblbad.html>

<http://paulgraham.com/spamhaussbl.html>

~~~
tquai
_any filter relying on the SBL is now marking email with the url
"paulgraham.com" as spam._

Impossible. The SBL lists only IP addresses; there is no content filtering at
all.

<http://www.spamhaus.org/sbl/>

Furthermore, there's a lot of FUD in this thread about Spamhaus listing people
who don't emit spam. IF this is true, then Spamhaus would have an unacceptably
high false positive rate, and we would be able to observe this. In reality,
Spamhaus has the lowest FP rate in the industry. Occam's Razor suggests those
who claim to have been wrongly blocked are mistaken about the reason for their
listings (if they ever existed in the first place).

~~~
eli
You are incorrect. (Well, you're correct that Spamhaus doesn't filter content
-- but they don't filter anything, they publish lists that various filtering
software uses.)

<http://www.spamhaus.org/faq/section/Spamhaus%20SBL#270>

 _I hear the SBL can also block domains, how? What is "URIBL_SBL"?_

 _Yes, the SBL can also be used as a URI Blocklist and is particularly
effective in this role. In tests, over 60% of spam was found to contain URIs
(links to web sites) whose webserver IPs were listed on the SBL. SpamAssassin,
for example, includes a feature called URIBL_SBL for this purpose. The
technique involves resolving the URI's domain to and IP address and checking
that against the SBL zone._

And of course they also have the DBL (Domain Block List), though I don't know
if that existed back when PG ran into problems.

Do you have a link to the false positive rankings? I'm curious as to how that
is measured.

~~~
tquai
Good point; I think both of our statements are true due to ambiguous wording
upstream. I also took it literally, "any filter relying on the SBL" -- I use
the SBL (via ZEN) but don't use SpamAssassin. And so my mail servers wouldn't
block any domain that resolves to an IP address in the SBL, as described in
the link you provided.

As for DNSBL false positive rates, I haven't seen statistics in a few years,
and by now they wouldn't be worth much. The only ones I saw were from 2005 or
2007. This one (linked to from the below article) from 2011 doesn't even test
Spamhaus:

[http://www.spamresource.com/2011/05/dnsbl-safety-
report-5142...](http://www.spamresource.com/2011/05/dnsbl-safety-
report-5142011.html)

This is just my personal experience saying (in 2013) that Spamhaus has the
lowest FP rate, which isn't scientific. I'm kind of surprised there haven't
been more FP comparison reports of major DNSBLs in recent years. If anyone has
a link I'd love to see it.

------
np422
Spamhaus can be a real PITA to deal with, all in attitude "squeal like a pig,
or you'll end up on the blocked list - bitch!"

Been there, done that, got the t-shirt.

What can I do to provide extra firepower in the ongoing ddos against them?

~~~
entropy_
Could you elaborate on what happened in your case that you'd be so vehemently
opposed to spamhaus(to the point of being willing to commit crime(s) to hurt
them)?

I'm truly curious on why the reaction to spamhaus being DDoS is so polarised.

~~~
np422
A long time ago in a galaxy far away I was a sysadmin at a local university.

Trying to keep mail-servers running and keeping up with the different spam
clearing houses different policies that kept changing without notice was a lot
of work back then.

Once you got black-listed getting removed wasn't always an easy process no
matter how quickly you tried fix whatever caused it. Methods of communicating
were not always available and when it were, responses were not always helpful
or even very polite.

I haven't managed mail servers for over ten years, and really hope that the
conditions for being included on a blacklist and process for getting removed
is more transparent by now.

Given the amount of trust that most people running mail servers are putting
into the different blacklists organizations like spamhaus get a lot of power
over the internet.

Judging from my experience with spam clearing houses it looks like that power
have once more corrupted when I read the news stories about cyberbunker.

We need places like cyberbunker to keep the internet free and open. When all
the pr0n, w4r3z and 1337 stuff have been cleaned out from the internet the
infrastructure to stop anything at will on the internet will be in place and
functional.

I wonder what would be the next thing to be removed from the internet?

~~~
tquai
_Once you got black-listed getting removed wasn't always an easy process no
matter how quickly you tried fix whatever caused it._

I took a job in the year 2000, at a company with 3000 email users, listed by
Spamhaus. First thing I did was close the open relay they were running. The
listing was promptly removed, and the mail queue was back to normal within
only a few days. I'm skeptical of your claim. I've never seen a confirmed case
of Spamhaus aggression, but I've seen a lot that were disproven, and even more
that sound like they were written by miscreants. Like the kind who would
advocate DDoS attacks _cough_.

~~~
np422
(English is not my native language)

Open relays were at the time manageable, even the ones that suddenly appeared
when someone installed an old OS-version, as were the process for getting
removed from the blacklists due to open relays.

Once you had one a computer lab workstation hacked and used for spamming - not
so easy to get whiteliested anymore.

The university had a class B-network, trying to get the staffs subnet
whitelisted while keeping the computer-labs blacklisted was apparently not
possible according to the spam clearing houses. Blocking port 25 for outgoing
traffic not possible to check from the outside and didn't help.

I can understand that organizations like spamhaus are overworked don't have
the resources to handle every non-standard case on the internet as quickly as
the blocked ip-range would like, but the replies we got were truly unhelpful.

The fact that someone bothered to register the domain stophaus.com seems to
indicate that my experiences isn't uniqueue.

