
Ask YC: Dealing with DDoS - PStamatiou
How do you do it?<p>At least once a week my load averages will stay at 5 or higher for about 30 minutes at a time. It's mysql that's eating up all the cycles even though page caching is setup for frequently accessed pages. netstat shows many blocks of IPs, each trying to connect to port 80 many times. I think it's coming from a zombie computer network.. one set of IPs included a Purdue connection.<p>My CS friends have told me about things ranging from iptables and manually adding offenders to my firewall - any automatic solutions that append offending IPs after X connections?<p>thanks
======
dazzawazza
I'll preface this with "I used to work for an online poker company and every
mofo in the planet used to attack our network"

Firstly consider what type of target are you. Most DDoS attacks are launched
for a lot longer the 30 minutes and they are certainly not regular. They are
looking to see if they can bring you to your knees so they can blackmail you.
Is your site that valuable? If so forget everything and get yourself a cisco
firewall and pay someone who knows what they are talking about. Cisco are
expensive but if you have a valuable site that 1% of real traffic needs to get
through.

Assuming your not in a position to be blackmailed: As other ycombo's have
mentioned logging and blocking are your friends but be careful. You say it's
mysql taking the time despite caching. Looks to me like you've found a bug in
your code (or at least your caching). Log what these IP blocks are requesting.
If it looks algorithmic then the chances are you've got a crawler ignoring
your robots.txt. Contact Purdue. Call them up (they will ignore your email)
and ask them what's going on.

You could chose to block these ip ranges but if you make your site weather
this storm it will be stronger in the future.

Good luck

~~~
mattmaroon
Which poker site?

~~~
dazzawazza
I was lead programmer at PKR <http://www.pkr.com>

~~~
mattmaroon
I've never even heard of that one. Very odd. Did they have a lot of action?

~~~
dazzawazza
It's one of the fastest growing poker networks in Europe. Only been going
three years, I left after two.

------
gduffy
Are they accessing pages, or just opening connections? Use netstat to see what
state the connections are in.

Make sure syncookies are enabled if they are just opening lots of connections
(<http://cr.yp.to/syncookies.html>).

You can limit parallel connections per host with iptables. See 'connlimit'.
Drop any invalid SYN packets. There's also 'recent' which you can use to keep
a dynamic list of ip addresses sending n SYNs over the past m seconds (then
drop new connections). Bonus is that's the list of IPs is
accessible/modifiable from /proc. Be careful not to kick out legitimate
clients by setting too low a limit, though. Iptables can log, too, so maybe
you can sample connections for a few seconds, 'sort | uniq -c' the ips, and
decide on a cutoff.

There are also network appliances that will do similar things without loading
your web server.

SHOW PROCESSLIST on your mysql, figure out what queries are happening. It
could just be that you need a new index, more appropriate configuration, or
better queries. In any case, at least it will give you a clue as to what is
causing the load.

Use mod_status (or similar for your web server) to figure out what your Apache
workers are doing. Modify keep-alive times.

If all else fails, see if your ISP can enable TCP Intercept on your nearest
router.
([http://www.cisco.com/univercd/cc/td/doc/product/software/ios...](http://www.cisco.com/univercd/cc/td/doc/product/software/ios112/intercpt.htm)).

------
tlrobinson
Could be web crawlers for search engines (or research, at Purdue?)

They're supposed to play nicely, especially if you have a robots.txt file set
up. Try that first if you don't.

------
gojomo
Squid as a reverse proxy can rate-limit inbound connections of various kinds
of groupings, which may help, especially if you can define a grouping which is
exclusively the kinds of remote IPs/user-agents/URIs that tend to become
abusive. I believe Squid 3 even has a way to delegate the grouping decision to
an external script, so you could write your own kind of triggering logic.

------
jbyers
This does the automatic iptables manipulation you're looking for based on
inspecting netstat:

<http://deflate.medialayer.com/>

That said, it's not really DDOS protection. It is good for blocking bad
crawlers or misconfigured / casually malicious clients. When attacks get to be
true DDOS - hundreds or thousands of zombies intent on taking you down -
you'll almost certainly need purpose-built hardware and an expert network
admin to deal with it.

------
imsteve
Use cache priming and serve slightly stale content out of the cache when the
most recent data isn't cached and then update the cache in the background.

And check your indexes!

Plus, write a decorator around your database query function to log all queries
and make a list of longest queries and longest average query times.

------
tlrobinson
If you want to go the iptables route (have fun...) check out the --limit
option.

Here's an example: <http://www.webservertalk.com/message1796841.html>

------
comforteagle
I'm a fan of turning on apf's anti-dos whenever I see load spike.

------
cosmok
This question has been on my mind for while.Hope to hear some great responses.

------
dawnerd
Are you sure you are not being "dugg" or anything else similar?

~~~
PStamatiou
I was looking at apache-top referrers the whole time - nothing crazy there.

