
Apache HTTP DoS tool released - timf
http://isc.sans.org/diary.html?storyid=6601
======
joshu
Put a reverse proxy in front. You should do this anyway. Modem and other pokey
connections effectively do the same as this DOS.

<http://joshua.schachter.org/2008/01/proxy.html>

------
gojomo
There's really nothing new about this sort of depletion attack, and it's easy
to launch in dozens of different ways in a line or two of code. (As far as a
'tool' to do it, I think the 'ab' benchmarking utility _bundled with Apache_
suffices.)

In response, you block/throttle origin IPs, and decrease timeouts so zombie
connections are dropped more quickly, and increase the capacity to serve or
shake-off troublesome connections without locking out legitimate traffic. An
inbound reverse proxy, as joshu suggests, often makes this easier -- and keeps
such enforcement cruft from complicating the primary server.

~~~
rbanffy
I think ab will send a full set of headers and not tie the server connection
indefinitely.

~~~
gojomo
Yes, but with concurrency/repetition/keep-alive, you can still tie up
essentially all of the connections. (And it's plenty easy to open any number
of hanging connections without 'ab', too -- a few lines of shell script, at
most.)

------
defied
More info about this attack is available on
<http://ha.ckers.org/blog/20090617/slowloris-http-dos/>

~~~
pmjordan
After reading the main article and the link you provided, my
interpretation/understanding is:

* The idea is to trick the web server into keeping a connection open and waiting for data, e.g. by keeping on sending pointless headers, not sending enough (< content-length bytes of) data in the body, etc. Unlike a SYN flood or so, this doesn't require much traffic from the attacker.

* The reason this works is that apache has a limit of one thread or process per active connection, and there is typically an upper bound on

* The way to fix this once and for all in apache et al would be to handle socket I/O asynchronously, thus lifting the 1:1 ratio of connections to threads/processes.

Is that accurate? I'm mostly curious so that I can try to avoid any such
pitfalls in any server software I might write.

One way to fix this in apache without breaking too many existing modules and
extensions might be to use fibers to hide the asynchronicity and schedule them
in the "blocking" I/O functions which would actually use non-blocking I/O
underneath. (I don't actually know the apache architecture, so I might be
wrong about the blocking I/O)

~~~
ryah
> I'm mostly curious so that I can try to avoid any such pitfalls in any
> server software I might write.

If you start a thread for each connection, you're doing it wrong.

------
eli
So... you basically just hold open all the TCP connections at once? How
anticlimactic.

~~~
rbanffy
Yes, but since it's Apache, anything like this is big news.

~~~
moe
It was big news 10 years ago (literally, 2001ish) when these kind of attacks
were initially discussed. Nowadays it's more of a big yawn as the script
kiddies rediscover old playgrounds.

~~~
jsteele
10 years ago was literally, 1999ish.

~~~
moe
Well, I _know_ I have seen tools released in 2001. And I'm quite sure the
discussion was older than that. Anyways, put 8 years if you want, this isn't
news either way.

------
jsn
And this is news somehow? Color me unimpressed. First time we'd been subjected
to this kind of attack was, what, a year ago? Maybe two. It's a trivial next
step after the old boring flood of http get requests.

~~~
Locke1689
It's also not that difficult to counter. The easiest way is to just do packet
inspection - if something looks wrong (repeated bad/incomplete headers), just
blacklist the IP for however many minutes. Repeat ad nauseam. Any IDS worth
its salt shouldn't even hiccup at this.

~~~
jsn
It's trivial to counter indeed; you don't need any packet inspection for that.
The main anomaly is not in the header structure, it's in the number of
sessions opened simultaneously by one host. This particular tool uses broken
headers for some reasons; other ones don't. But they all open sessions
excessively.

So you just use iptables connlimit match block the clients with more than e.g.
10 connections.

~~~
rbanffy
Actually, just blocking originating IPs would not stop a DDoS based on this.
The simplest approach would be to timeout the connection if a credible set of
headers is not received within a certain time.

This attack is based on the idea the attacker commands more resources than the
target server. It doesn't fly very well.

~~~
jsn
Sorry, most of that is wrong :)

1\. No, it's not the simplest approach. The attacker can trivially send
perfectly credible set of headers, then start receiving the response,
acknowledging 1 byte per 30 minutes. It's routinely seen in the wild, too.
Header-based approach completely misses the point of anomaly.

2\. Blocking originating IPs is perfectly effective as soon as all existing
connections from the blocked IPs time out. Which is fairly soon.

3\. connlimit approach doesn't just block originating IPs, it prevents anyone
from opening a lot of connections.

4\. In any real DDOS scenario attacker does command an order of magnitude more
resources than the target servers.

[edit: formatting]

~~~
rbanffy
1\. I don't think the client can acknowledge one byte per minute - I think the
minimum would be one network packet. Anyway, the whole response would timeout.
Apache has a config setting for that, does it (I have been using mod_proxy a
lot and mod_proxy does timeout)? Perhaps, there should be a minimum bandwidth
threshold that forces the connection to abort. I see your point, however. Even
sending a real complete header should not prevent the request from being
treated as hostile.

2\. I am not sure I got what you say. In a real DDoS attack, there should be a
lot of different IP addresses and blocking thousands of them (many of whom
would reset connection and come back with a different IP) would make the life
of the firewall very uncomfortable.

3\. I know this has problems with proxies and firewalls with large number of
computers behind them. I don't remember proposing this idea.

4\. It all depends on what is being attacked and how. I don't think any
serious DDoS attacker would use this approach.

~~~
jsn
1\. You're wrong, see rfc793. TCP flow control acknowledges streamed bytes,
not packets.

2\. I just tell you what works in real life. Blocking 500 supernodes (out of
5000 machines botnet) is basically a non-issue for linux iptables on usual
modern processes. Other 4500 are usually pretty tame in terms of firewall
processing power.

3\. You didn't propose that, it's what we do (and what works best for these
attacks). Its rate of false positives is sufficiently low. Folks behind NATs
don't use sustained 4 connections per user, so it's usually a non-issue
either.

4\. Well, you're wrong if you don't think so. Mitigating DDOS-es is one of the
things i do for living [i'm in charge of web infrastructure of certain
political opposition websites in Russia, if it rings any bells]. The scenarios
i described is what actually happens.

~~~
rbanffy
Interesting. Acknowledging receival on two layers (packets and butes) makes
little sense. Wonder why such decision was made.

About 4, are the DDoS aimed at your servers or at your office infrastructure?

~~~
jsn
They don't acknowledge packets, just bytes. It's pretty reasonable for a
reliable streaming protocol.

Their DDOS attacks target our servers. For example, Russia has some issues
with Estonia, opposition sites post some independent statements and news on
that. Then comes DDOS.

~~~
rbanffy
Still, the overhead is large if it employs single byte confirmations. It turns
symmetrical what could be an asymmetrical communication.

~~~
jsn
Sorry, wrong again. Stream byte acks are _way more_ effective than packet
ones. Tcp endpoint can receive several packets of data and then send just one
very small ack packet (you only have to send the byte stream position after
the last received packet). I'm not sure what you mean by "asymmetrical", but
that's probably as asymmetrical as it gets.

It's called "delayed ack" or something. Iirc, it's all described in rfc793.

~~~
rbanffy
What would be then the ideal way to fix Apache?

~~~
Locke1689
Apache doesn't need "fixing" - this is the job of a firewall or IDS or some
other such anti-DDoS software. That doesn't mean they couldn't incorporate
some protocol patch or use asynchronous I/O or some other crap, but the
reliance shouldn't be on Apache to block this. If you're running a website
important enough to DDoS, you should have a heavy duty firewall and IDS and
possibly reverse proxy in place as well.

------
pibefision
what if we use NGINX as a reverse proxy in front of apache? could this be a
good workaround?

------
jimmyrcom
It's a pretty short code written in perl. I'd be scared if it was modified
from curl or written in erlang. The author admits being corrected about the
type of attack not being new but claims thats the first tool. I don't see how
its new though. Whether you keep alive by incomplete requests, by valid
requests or by sending post data very slowly, it doesn't seem to make a
difference. Just eat up as many connections as possible.

~~~
gojomo
That perl script is about 100x longer than it needs to be (even disregarding
the embedded docs and ASCII graphics). This is a super-simple (and old)
attack; many DoS attacks based on large amounts of traffic first cause
problems by depleting this same limited pool of threads/connections.

------
foppr
woohoo!

