

Ask HN: How to deal with SYN FLOOD and extortion - xal

We made a big mistake in our server setup. We bought beefy as app and db servers but went for ultra reliable sun ultrasparks and openbsd for firewalls. This was fine until the DDOS started. After 3 days of many different attacks we now how much better firewall hardware installed thanks to our fantastic colocation which went the extra mile and helped us when we needed their help most but we are now surviving on total brute force.<p>Our firewalls are faster then the SYN Floods that hit us. This is an arms race that we cannot win in the long run. Yes we can buy more hardware but it's much easier to infect more machines with bots over time.<p>How do people protect themselves against extortion and malicious ddos attacks? What software / hardware protects the bigger sites on the net?
======
tptacek
Any large ISP should have a process and gear in house to mitigate DDOS
attacks; you should escalate, first to your hosting provider and then to their
upstream provider.

Every ISP does something a little bit different for large-scale attacks. Some
of them off-ramp traffic to scrubbers, some of them have inline devices that
can characterize and block SYN floods. My previous employer, Arbor Networks,
now has gear deployed at something like 90% of the worldwide tier 1 and tier 2
ISPs that will detect any major flooding event and generate a report which
might be helpful to you.

What I'd watch out for is the dozen odd fly-by-night operations that are
promising you that, for a monthly fee or a one-time purchase of some $100k
box, you can block these attacks on your own network.

~~~
lsc
I'd extend that warning... watch out for any company, even if it's large and
reputable and the salesmen have nice suits, that promises to solve your DoS
problem by selling you an expensive firewall. If your traffic is below
100Mbps, there's nothing the fancy box can do that your OpenBSD box can't.
(this changes as your traffic climbs significantly above 100Mbps, but most of
us little guys are below that line.)

~~~
iuguy
I'd go a bit further and say that even over 100Mbps you're not going to get
much protection over an OpenBSD box or a Sun system running a decent Firewall.

The key thing is to try to block it upstream. Your firewall (and indeed
anyone's firewall) is only really designed to restrict IP-layer access to
specific source and destination hosts, ports and datagram types. Unless it was
built by Cisco, in which case it was probably designed to do a whole load of
other stuff, none of which will help you right now.

There are limited things you can do to the IP stacks of most OSes these days
that can help discard unsolicited packets, and reduce resource consumption on
the local boxes (if that's high) but an upstream solution is likely to be more
optimal in most cases.

------
mpk
The big boys seem to be a little reluctant to publish information on their
DDoS strategies. (If anyone has information that says otherwise, it's much
appreciated).

I spent some time googling around and in case anyone's interested, these links
seem to cover most of the mitigation patterns.

A slideshow covering common mitigation techniques,
[http://www.slideshare.net/intruguard/10-ddos-mitigation-
tech...](http://www.slideshare.net/intruguard/10-ddos-mitigation-techniques-
presentation)

A Cisco whitepaper (which, as usual, has en emphasis on Cisco kit and a long
URL),
[http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5879/ps...](http://www.cisco.com/en/US/prod/collateral/vpndevc/ps5879/ps6264/ps5888/prod_white_paper0900aecd8011e927_ns615_Networking_Solutions_White_Paper.html)

A fairly comprehensive PDF document - A Survey of Active and Passive Defence
Mechanisms against DDoS Attacks

<http://www.fbi.cqu.edu.au/FCWViewer/getFile.do?id=17921>

~~~
tptacek
I don't know how secret this stuff is; I think if you poke around NANOG,
you'll find a lot of material. If there's something that makes DDoS mitigation
mysterious, it's how ad hoc all this stuff is.

What I've had firsthand experience with is:

* Tier-1's profile all their traffic, either directly or with flow export, and will get alerts if they see spikes to certain netblocks, or spikes to specific /32s with specific characteristics.

* Inside a Tier-1, the third-tier support people usually have additional monitoring they can enable for customers if an incident has been escalated to them. They get hundreds of these calls a week.

* A lot of Tier-1's can quickly reroute traffic to a specific /32 to run through special-purpose filtering setups, which might be a DDoS box like a Cisco/Riverhead, or an IPS like TippingPoint, or even just a Cat with lots of TCAM space set aside for filtering.

* A lot of Tier-1's --- maybe all at this point --- have some mechanism set up to share signatures of attacks so they can push filtering further upstream. When I watched that stuff happening, the sense I got of it was that this was really reserved for things like global botnet C&C.

What I'd add to the discussion on products is, most of what's built to combat
DDoS is really only useful if you sell transit. If you're a Fortune 500, I
know there are ISPs where you can get Cisco or Arbor gear deployed on the head
end specially for you. But that's a Fortune 500, not 37 Signals.

------
axod
I've had a few DDOS attacks in the past, first when hosting with slicehost,
and then when at linode. Running IRC related stuff, these attacks are pretty
much expected from time to time.

Both dealt with them fantastically, keeping me updated on progress etc.

As far as I remember, there was only one instance when they decided
(slicehost) to take my server 'offline' and just wait for it to pass/upstream
providers to deal etc.

The other times they were quickly able to escalate to upstream providers with
minimal impact. I've had a few at linode, and whilst it does use up some
bandwidth allowance, it hasn't been a big deal at all.

Obviously if you're able to, get more IPs, use a few data centers, etc.
distribute yourself so that an attack on any one part doesn't have a big
impact.

------
Brushfire
Move yourself into the cloud and let someone else deal with it. Seriously.

I'm a big fan of outsourcing anything you arent 'great' at. In this case, its
running firewalls. Unless of course your app is something related to server
maintenance or firewalls or security, etc.

Most software companies really shouldnt try to deal with this stuff -- I had
to learn the hard way. You should only spend your time on your applications,
not anything else that doesnt directly benefit you. You can get lost in the
maintenance.

~~~
Tichy
Just curious, but how would "the cloud" help with such a problem? Sure,
supposedly you could summon infinite computing power to handle the DDOS
requests. But the computing power would not come for free, so the DDOS would
still do serious financial damage.

~~~
Brushfire
You are paying either way. Thats just the way it works.

However, under one option, you pay and Amazon deals with the actual load,
downtime, server stress, hardware improvements, etc.

Under the other, you get to deal with all of that AND get to be pissed off AND
accomplish nothing for your business at the same time.

~~~
xal
I'm curious, anyone has experience dealing with DDOS attacks on EC2? It does
seem like a good option...

~~~
wmf
If you're using an instance as a load balancer it's limited to 1 Gbps.

------
patrickg-zill
If they are never connecting, then why not set up a SYN "honeypot"? As soon as
a valid user connects, redirect them to a different site or IP address that
you do not advertise, that actually hosts your app and db servers.

Thus you could rent or colo a small system on a different network, maybe even
in a different part of the country and then redirect valid users to an
entirely different location with a cookie in the URL. No cookie no service.

~~~
moe
The attacker would simply move the syn-flood to where the candy is.

------
datums
If they have already attempted to extort $$. You should keep the mitigation
device in place. There are a few devices that do packet inspection and can
handle multi gigabyte size attacks.But like many have mentioned contact the
upstream provider

------
lsc
your provider (or your network guy, possibly you) needs to blackhole the
source of these attacks (preferably notifying the ISPs in question.) most of
the better co-location providers will give you a way to blackhole traffic from
certain IPs. Really, you should be able to do that from your OpenBSD firewalls
without help from anyone else, assuming the problem is too many syn packets
rather than too much traffic. dropping all packets from a particular source IP
is not expensive, even when you multiply that by a large number of source IPs.

<please ignore this comment, I'm completely wrong. see the next comment.>

~~~
asmithmd1
Black listing the source IP's is NOT going to help. The attacker can just
spoof any IP, he doesn't care about actually connecting he just wants to tie-
up your server

<http://en.wikipedia.org/wiki/SYN_flood>

~~~
daveungerer
Makes you wonder why ISPs are not discarding packets with spoofed addresses,
instead of letting them escape into the greater internet. Is it too expensive
to do? My knowledge is a bit rusty, but isn't this a common router function?

~~~
ericwaller
There's no way to know the address is spoofed without replying to it. It's the
same idea as "spoofing" the return address on a letter.

~~~
daveungerer
Sorry, that's not what I meant, and I may have forgotten some terminology, so
let me attempt to make myself clearer: The first edge router that a packet
encounters, coming from the source, knows what the valid addresses are. The
router simply needs to ask itself: "If I were to receive a return packet
destined for the claimed return address, would I know what to do with it?"

If invalid ones were dropped, then you can at most spoof another address in
the same network. But either way, it can be traced back to the correct edge
router.

I'm looking for someone experienced in these matters to explain why this
theory isn't / can't be put into practice.

~~~
lsc
It's called BCP38 and decent ISPs and competent network admins do it. I do it
on the port level for all my IPv4 addresses (that is, all packets leaving a
xen vps of mine are dropped unless they come from the address I assigned
them.) and I will do it for IPv6 when I build out the routing infrastructure
for my new location.

It's really pretty easy to do... especially just edge level like you are
talking about. The problem is that it costs me some time/money (even if it's
only a little) to do it, and it mostly benefits other people. So some ISPs
still don't do it.

<http://www.faqs.org/rfcs/bcp/bcp38.html>

but it's really easy. on my router I look at all outgoing packets, and I ask
"Does this IP have a source address that is reasonable? (one of my IPs, or an
IP of someone for whom I am carrying traffic.)" If my router sees a outgoing
packet with a source address not associated with me, it's obviously spoofed,
so it drops the packet. If everyone did this, spoofing would only be possible
within your own network.

Every time an ISP does this, the world becomes a little better for all of us.
But spoofing will still be a problem until all ISPs do this, and that probably
won't happen for a while.

(you also do this to incoming packets, if it is an incoming packet coming in
from the internet and it has a source of one of my IP addresses, something has
obviously gone horribly wrong. drop the packet. but this only protects you
from the most obvious spoofs. It is very important if you do IP based
security.)

~~~
daveungerer
Thanks, exactly what I was looking for. Now, if only we could start a movement
to only use BCP38 compliant ISPs. And then convince our ISPs to reject all
packets from other ISPs that have been shown to be non-compliant, in case any
survived.

EDIT: The way I put it may be a bit extreme, but the idea is there.

~~~
lsc
I'm not sure it's too extreme. take it to NANOG. We'd need the support of big
players, much more than one-and-a-half rack operations like me. (though bcp38
compliance is much more common amongst the big players. Heck, the provider I'm
moving away from doesn't do it, something I didn't know before I signed a
contract. For that matter, I only do it on outgoing IPv4. I don't do it on
incoming packets or IPv6. This will be corrected in the network upgrade that
is in progress now, but still.)

The big problem with your proposal is verification. It seems... difficult to
verify that another network properly implemented BCP38 without actually
putting a probe on that network.

