
Ask HN: How would you deal with DDoS? - samratjp
In my recent memory - Posterous dealt with it once, the 4Chan and Tumblr fiasco and in the last few days, the Visa, MasterCard and PayPal mess.<p>What's a good way to deal with DDoS? Bonus points if you can elaborate more than "buy a stinkin' Cisco firewall"<p>EDIT: Do tell us your war stories as well.
======
tptacek
I have what Patrick would call "a wee bit" of experience with this problem; I
led a dev team on a successful anti-DDoS product from 2001-2003. For now, I
have two pieces of advice:

Advice #1:

Every major ISP has a tier of network engineer that is equipped to handle the
DDoS problem. They're the ones with access to the traffic analysis tools,
they're the ones with the scripts to deploy ACLs, they're the ones that can
reroute traffic to a regional scrubbing center. If you're dealing with a real
ISP, they have an anti-DDoS product (probably one I've very familiar with)
deployed.

There aren't many of them and they are never the person answering the phone
when you call the ISP. Nobody at your ISP has any incentive to escalate you to
that person. Your ISP may deny that the person exists (I've seen that happen
at ISPs where I know the right person by name).

Find that person. Be persistent. If I knew who mine was right now I'd send
brownies and belgian ale. Weekly. Advance planning and cheap insurance can't
hurt.

Advice #2:

Nobody knows what "I'm under DDoS attack" means. It doesn't mean anything. You
have to be able to describe the attack precisely in technical terms. That
doesn't mean "it's a SYN flood!"; it means, "My link is saturated, I'm getting
N million packets per second, an unusually high number of connections sourced
from TCP ports 15030 - 19012, it started 9 minutes ago, and doesn't coincide
with a spike in requests to my DNS server."

In all likelihood, nothing you have deployed today is going to generate that
information for you, so your job today is to get that infrastructure set up. I
recommend getting NetFlow turned on and using Argus, which is free; NetFlow
also happens to be a language ISP network engineers speak readily. You have
other alternatives, like ntop. Just have something that can characterize
traffic and ideally tell you (either directly, or via graphs) when things are
out of the ordinary.

You will have much better luck getting help from your upstreams if you can
write the ACL for them and make it easier to find what places in the network
need it.

~~~
bradleyland
I can vouch for nTop. Really cool project. I did a bandwidth monitoring
deployment for a customer who runs their own layer-3 network atop a carrier
ethernet solution, but didn't have any plan in place for monitoring what goes
over the wire. We found all kinds of fun stuff once we installed a few nTop
nodes. Among the things we found: a Halo server to which employees from
several different states were connecting and playing on a daily basis.

~~~
lhnn
Please tell me you didn't shut that down. Work/life balance! Stress relief!

------
jacquesm
The best way is to have an attack mitigation strategy worked out in advance.
Once the traffic is hitting your servers you are very much too late, the best
spot to filter it out is as far upstream as you can get. Getting to know your
upstream providers and having contact information is probably the single most
effective thing you can do in terms of ROI when it comes to dealing with large
scale DDoS.

Chances are they already have that big Cisco or Force10 router complete with
FPGA based filtering magic.

~~~
dazzawazza
This is great advice. At a poker startup I worked at the most important thing
was our relationship with the pipe owner.

It can be expensive to get them to act but for a poker site it's a lot cheaper
than loosing hands per minute.

We also paid a lot of money for boxes at our end that did real time traffic
analysis for various attack vectors. Expensive but again cheaper than loosing
money.

~~~
wlievens
Was it necessary at some point? I'm not being sarcastic, just interested.

~~~
jacquesm
Typically anybody that makes more than a few million per month on the internet
and is not under the protection of the authorities for stuff like this has by
now found out about the dark side of the net.

~~~
chopsueyar
What about streaming video sites? Has ww seen any distributed attacks on 1935?

I am running Wowza currently, and worry about this.

~~~
jacquesm
We have not seen any attacks on the video infrastructure but we're small fry.
Let me ask a much larger site using wowza and get back to you on that one.

~~~
chopsueyar
Thanks! Look forward to it.

~~~
jacquesm
Ok, here is the answer:

"If there has been we haven’t noticed. Taking down one wowza server wouldn’t
cause us any problems. They would have to take down 5+ servers for us to
notice."

------
catlike
One of the most effective answers is not "buy a stinkin' Cisco firewall" but
rather "buy a stinkin' Arbor"

<http://www.arbornetworks.com/>

If you want to survive large scale DDOS you need equipment that can scrub the
incoming unvalidated data in real-time and keep up.

Combining Source Based Remotely Triggered Black Holing (RTBH) with uRPF
affords you the ability in a sophisticated network to drop a large amount of
undesired traffic (especially if it's from simpler DDOS strategies). If you do
in fact have the $$/need for the Arbor then inside of black-holing the traffic
you send it to the Arbor and let it scrub the packets.

~~~
tptacek
I was the lead developer on Arbor Peakflow DoS (not the traffic scrubber they
acquired from Ellacoya; the NetFlow analysis engine) and then a product
manager at Arbor. This is fine advice if you're an Internet service provider
(except that virtually every ISP has already taken that advice). But if you're
buying connectivity, the Arbor box isn't going to do you any good; your links
are going to be saturated before the scrubber can do anything about it.

------
robotkad
Most clients participating in DDoS attacks are plain dumb.

They will continually perform a GET request on a single page and not parse the
response. This means they wont respond to javascript, images, cookies or
redirects correctly.

You can devise a test that identifies attacking clients and then blacklist the
IP addresses for a while.

While I'm sure this strategy isn't perfect, it has been sufficient for the two
attacks I have been subjected to.

Edit: vladd's reply articulates this idea better than I was able to :)

~~~
xentronium
Effectively banning robots is the last thing you want to do.

~~~
bdonlan
It's better than having the site be completely inaccessible. You'd only deploy
this when you come under load. Additionally, many of the more important robots
can be identified reliably (eg, googlebot has a DNS handshake that can
positively identify legitimate googlebots) - and even if you get a false
positive, if you filter out all their packets, they're likely to assume a
temporary failure and come back later.

------
snorkel
On your outward-facing servers

* turn down the TCP handshake timeouts * turn on TCP SYN cookies * watch incoming network connections and traffic logs and add firewall rules to block each suspected attacker subnet.

In a real DDoS you have to block the attack traffic at the TCP level, not at
the application level (the web server) which means adding firewall rules or
iptables.

You will end up blocking a lot of legitimate traffic but better than being
100% down.

Still getting hammered? On to phase 2:

* Launch mirror servers, change your DNS records to point traffic at the mirror. (Most attack bots don't follow DNS changes) * If it's an application-level attack then put your site in static delivery mode.

Still getting hammered? That'll teach you to hate on Justin Bieber.

------
vladd
0) Be prepared

Be proactive. Plan for the solutions below in advance, not when traffic is
hitting your servers and SSH along with everything else is unresponsive.

1) Availability for investigation

Renice sshd to make sure you can get to your servers when they are hit with
CPU usage due to the DDOS. Reserve some connections for the root user on your
SQL server (Postgresql already defaults to 3 for this) so you can still
connect, investigate and analyse what's happening.

2) Pattern matching

Every solution against DDOS can be described more or less as "pattern
matching". There are two steps involved:

-> identify a pattern associated with DDOS requests.

-> match them as early as possible (iptables, upstream provider, hardware firewall) and reject them.

Easiest is to match offending IPs, those that make repeated queries (i.e. once
every second). Set the threshold in the right spot - too high and you'll ban
valid users, to low and you fail to ban enough in order to restore service
availability.

Other things to watch for are at the connection level: either SYN flood
behaviour ( <http://en.wikipedia.org/wiki/SYN_flood> , deploy
<http://en.wikipedia.org/wiki/SYN_cookies> ) or KeepAlive connections that
keep httpd busy (always reserve in httpd.conf for example a number of
connections that aren't KeepAlive enabled).

Identify pages on your website that are slow to load or generate large, time-
consuming SQL-queries. Those are the main target during DDOS attacks; pattern-
match their URLs and disable them during attacks if they're not critical for
your website, or have a fall-back for them.

Check for errors in the attackers' requests: either they have the same weird-
looking user-string, or lack the referrer, or something else that you might be
able to find. Pattern-match and reject based on that.

Pattern-match clients that keep requesting the .html without loading the
corresponding .css or images -- most probably they are attackers calling the
GET API (corresponding to wget-ing the HTML) but don't parse and download the
corresponding images on the page. Sure, you'll ban text-browser users but
that's a small price to pay in order to restore your website.

Have a low-resource version of the website where you can turn off the requests
for CSS, images and so on and just serve plain functional HTML. Pattern-match,
if any, those IPs that keep requesting CSS and images previously associated
with non-cache-able HTML pages.

And lastly... be proactive, be prepared, have a plan.

------
arondeparon
Some related sources with answers to this question:

[http://serverfault.com/questions/32361/how-to-best-defend-
ag...](http://serverfault.com/questions/32361/how-to-best-defend-against-a-
slowloris-dos-attack-against-an-apache-web-server)

[http://serverfault.com/questions/3476/what-are-the-best-
tech...](http://serverfault.com/questions/3476/what-are-the-best-techniques-
for-preventing-denial-of-service-attacks)

[http://serverfault.com/questions/142280/is-syn-flooding-
stil...](http://serverfault.com/questions/142280/is-syn-flooding-still-a-
threat)

------
robotkad
I also forgot to mention in my earlier reply the one thing you should
DEFIANTLY not do: pay an attacker to stop.

I often hear of attacks that are accompanied by an email demanding thousands
of dollars via Western Union.

Attackers like this pray on small, non-technical businesses (eg blogs, small
retailers) who often feel they have no option but to pay the ransom.

Guess what happens when you pay for the attack to stop? The attack stops, only
to start again in 8 weeks time with a new, higher ransom email.

~~~
dedward
Never negotiate with terrorists, right?

THe thing is, this all depends on scale. If you are small scale, look at you
cost per day. They won't attack you forever - their resources are limited.

So if you're in that situation - okay, it sucks. First, calculalte the cost of
doing business. If they're asking for 10 grand and you're thinking of paying
because you'll lose that much in a few days, you're better off getting hooked
up with a real DDOS protection provider - it will cost a lot less than that in
the long run.

If you make tons of money daily (some affiliate sites, etc) then it may (may)
be worth doing both - if one day's business will cost you a quarter million in
profits, and you havent' already set yourself up for DDOS, it might be worth
letting hte FBI know, paying them off anyway, and then getting protection
services in place so it never happens again (cause now you know)

It's the small fish who just can't afford DDOS protection services and might
have sites who make _just_ enough money to live on off their site that are in
the tough spot, and could be a prime target for the extortionist-type
DDOS'ers. I wonder how common that is.

------
phillian
As a web publisher and affiliate marketer, I've had domains DDoSed at least 4
times in FY10. The worst of it being twice in two days on two separate domains
both targeting the same campaign by the same botnet based in Turkey.

Each one cost me money, but the two in two days cost me thousands. I was
hosting a VPS on Rackspace and did everything I could to mitigate it but
ultimately the attack stopped because they wanted those resources directed
elsewhere.

After the first one, I hooked up with a smart guy on Twitter
(<http://twitter.com/sempersecurus>) that tracks botnets. He allowed me to
sinkhole the domain so that he could collect and aggregate the data.

To stay defensive, I put some .htaccess blocks in for countries that are
clearly well outside normal buyer patterns. In each case, I know a user (read:
a human) saw the campaign and then put the DDoS in place so I hoped, by the
.htaccess blocks, to prevent them from seeing the site altogether. But I know
I really need to be a much more knowledgable coder to be proficient at
deflecting these in the future.

~~~
dedward
AThere isa point where coding won't help you at all, and sheer bandwidth wins
out. These guys can saturate 10gigs without worrying about it.

If your business is really worth at least $1000/day then, a real-life
protection service that proxies your traffic at a high bandwidth location and
sends you back only clean traffic would seem easily within your budget, and
worth it ot protect your assett. Let them worry about the constantly changing
attack patterns and let you worry about increasing the value of your site.

------
tzs
One of the calls I saw for the current round of DDoS attacks listed Amazon as
one of the targets. It seems to be having no effect. Can I infer from this
that Amazon knows how to handle DDoS attacks?

If so, would one reasonable answer to the question then be "host with Amazon"
and let them deal with it for you? Or at least have a backup ready at Amazon
to switch to? (With the recent announcement that Amazon's hosting is now PCI
level 1 compliant, hosting at Amazon becomes possible for many for whom it was
not an option).

~~~
lsc
>One of the calls I saw for the current round of DDoS attacks listed Amazon as
one of the targets. It seems to be having no effect. Can I infer from this
that Amazon knows how to handle DDoS attacks?

keep in mind that you are paying for that DDoS traffic. $0.08 per gigabyte
means you are paying something north of $350/hr for a 10Gps DDoS.

Now, I guess if you only expect the traffic to continue for a short while
that's not a bad deal at all, but if it goes on for any period of time, that
can get expensive fast.

For comparison, one of my providers has offered me a 10 gigabit transit link
for around $10.5K/month, so assuming I could use that to DDoS you, (and
really, that sort of thing would be... difficult. Far easier to hire out a
botnet.) I'd be spending about a dollar to cost you twenty dollars.

~~~
dedward
Amazon operates on a scale where they can handle this type of thing I don't
mean automaticlaly, but they have the access points and gear in place to do
it, and the levearage with the right ISPs.

Buying a 10 gig link woulnd't be a distributed denial of service attack, and
itwould be far too easy for the providers to not only find out who it was and
filter it out, but to find out who to sue for all the wasted time and
bandwidth and other criminal charges.

~~~
chopsueyar
First, you are assuming both parties are in the same country, and second, you
think the provider would notice instantly, or charge ~$6 per minute until
filtered.

~~~
borism
and lsc is assuming in his calculations that he'd be able to attack for a
month.

------
troels
ddos is a very broad category of attacks, so there isn't one single solution.
What really matters, is that you have some sysadmin type on board who knows
his network protocols.

~~~
lsc
why was this modded down? There are many kinds of DoS attacks, with many
mitigation strategies. If it's a pipe-filling attack, there's not much you can
do besides get a bigger pipe, or have your upstream do filtering for you
(which is quite often more expensive than just getting the bigger pipe... see,
/someone/ needs the capacity to soak up the attack, you or your upstream, and
that capacity costs money.)

I personally got hit with a flood of many small packets, which overwhelmed my
router. The obvious solution here is to buy a better router. (it was 200Mbps
of rather small packets on a 1000Mbps pipe.) I ended up just having my
upstream blackhole all traffic to the target IP, which is sad, as it 'finishes
the job' for the attacker, but on a $12/month account, what do you want?

Now, for what I provide (VPS hosting) an application-level DoS, slowlaris or
just hitting your app in an expensive way isn't really my problem. That's on
the application programmer, and as this is mostly a form for application
programmers, it does make sense that the highest modded comments on this page
describe mostly application level DoS mitigation strategies.

------
sl_
Since we are regularly the target of DDoS attacks on rather small scale (below
10GB/s) we have compared a number of machines:

[http://dev.esl.eu/blog/2010/09/10/in-search-of-the-anti-
ddos...](http://dev.esl.eu/blog/2010/09/10/in-search-of-the-anti-ddos-device/)

Just having one of these machines won't solve all of your problems, but it can
certainly help.

------
shorbaji
A Cisco whitepaper is
[http://www.cisco.com/en/US/tech/tk59/technologies_white_pape...](http://www.cisco.com/en/US/tech/tk59/technologies_white_paper09186a0080174a5b.shtml)

This was posted earlier on HN <http://news.ycombinator.com/item?id=1986867>?

------
carl_
You only mitigate attacks you don't stop them.

You either get massive uplinks (multi 10GE) and cisco nexus (or similar) or
proxy/spread/diversify across multiple facilities separate to the core and
only forward legit traffic (or null route targetted proxies).

------
swalberg
I've dealt with a couple over two different employers, both had a similar
pattern. The same page was being requested over and over, but there was no
discernible patterns in the headers.

On the first one, I already had a Squid reverse proxy in front of the servers,
so I configured that page to cache and all was good.

On the second one my load balancer was a POS, so we ended up getting the
developers to configure the page to 302 to a new page, and the bot didn't
follow the redirect.

In both cases, the thing requesting the page wasn't a real web browser, so it
didn't take cookies or follow redirects. I ended up getting an F5 shortly
after and wrote an iRule to do a cookie check should another DDOS come by, but
never had to use it. I was also fortunate that we had enough bandwidth to
serve the requests and that it was a resource starvation type attack.

------
roadnottaken
For anyone interested in this topic, 'Fatal System Error' is a pretty good
non-fiction book that tells a LOT of great DDoS war stories:

[http://www.amazon.com/Fatal-System-Error-Bringing-
Internet/d...](http://www.amazon.com/Fatal-System-Error-Bringing-
Internet/dp/1586489070/)

Good read, IMHO.

------
tabman
It cost a lot of money to get the necessary equipment to defend against an
attack, but DOSarrest can handle the job rather well. They defend against any
kind of DDOS attack and they do not care about the size of the attack. I had a
forum which came under attack one day and I did all kinds of things to try and
defend against the attack. I wrote my own script to grab IPs, use automated
attack detection and block tools, bought expensive cisco hardware, and still,
I had a LOT of problems. I finally went and looked for experts in DDOS
protection which is when I came across DOSarrest. They where nice and helped
get everything set up and optimized for my site. I gave them my IP address and
a payment, within 5 minutes I was given an IP to point my DNS to and then my
site was back up as soon as my DNS migrated. I turned my TTL down in
anticipation for this, so it was only about 10 minutes from the time I gave
them my IP to the time I was back online. Then to be safe I changed my IP to a
secondary IP, had them update their system which took less then 5 minutes, and
then blocked all traffic but theirs coming to my system.

These 4Chan guys attacked MPAA which went to DOSarrest as well and within
minutes of being set up there they where back online. They protect a bunch of
high end clients who have come under attack and bring on new people all the
time.

They also have a nice customer panel where I can log in and change all kinds
of settings for my site and view the statistics of the attack. I saw the
attack get up to over 500mb/s and have no effect what so ever. I highly
recommend them.

------
karl9231
I too had to go to a specialist I was attacked last year and my provider Time
Warner could do nothing in our case. We went to Dosarrest.com and they fixed
everything in about an hour.....if your stuck its the fastest way to get back
online.

------
pbhjpbhj
I'd have thought you could counter-attack? Assuming that the bots have poor
security and so have been compromised - I guess that the botnet could be
hardened before use to mitigate against this.

Do any ISPs profile their clients to test for bots?

------
smountcastle
If you have the money, pay someone else to deal with it. e.g. VeriSign's
offering: <http://www.verisign.com/ddos-protection/index.html>

------
monological
hmm what if you 301 redirected (if http) the ddos flood back at 4chan, would
that end up disabled their site?

~~~
monological
but that would mean that you only redirect packets based on filtering out the
botnet IPs, which would be pointless because if you could distinguish between
botnet IPs and valid users, then you'd have the ddos problem solved

------
Keyframe
Send a submarine to cut off submarine communications cables.

------
brianwillis
DDoS attacks aren't aren't technical problems - they're political ones.
Political problems need political solutions. Technology isn't the cause of a
DDoS attack, just the medium.

~~~
csomar
A competitor can attack you to take your service down.

~~~
wlievens
Has this ever happened?

~~~
dedward
Yes it has. Unfortunately I can't really say much about the parties involved
or what happened (I wasn't the bad guy though, promise), but it absolutely
has, and if I've seen it, then either I'm a unique little snowflake (yay!), or
it's happened a LOT more.

