What's a good way to deal with DDoS? Bonus points if you can elaborate more than "buy a stinkin' Cisco firewall"
EDIT: Do tell us your war stories as well.
Every major ISP has a tier of network engineer that is equipped to handle the DDoS problem. They're the ones with access to the traffic analysis tools, they're the ones with the scripts to deploy ACLs, they're the ones that can reroute traffic to a regional scrubbing center. If you're dealing with a real ISP, they have an anti-DDoS product (probably one I've very familiar with) deployed.
There aren't many of them and they are never the person answering the phone when you call the ISP. Nobody at your ISP has any incentive to escalate you to that person. Your ISP may deny that the person exists (I've seen that happen at ISPs where I know the right person by name).
Find that person. Be persistent. If I knew who mine was right now I'd send brownies and belgian ale. Weekly. Advance planning and cheap insurance can't hurt.
Nobody knows what "I'm under DDoS attack" means. It doesn't mean anything. You have to be able to describe the attack precisely in technical terms. That doesn't mean "it's a SYN flood!"; it means, "My link is saturated, I'm getting N million packets per second, an unusually high number of connections sourced from TCP ports 15030 - 19012, it started 9 minutes ago, and doesn't coincide with a spike in requests to my DNS server."
In all likelihood, nothing you have deployed today is going to generate that information for you, so your job today is to get that infrastructure set up. I recommend getting NetFlow turned on and using Argus, which is free; NetFlow also happens to be a language ISP network engineers speak readily. You have other alternatives, like ntop. Just have something that can characterize traffic and ideally tell you (either directly, or via graphs) when things are out of the ordinary.
You will have much better luck getting help from your upstreams if you can write the ACL for them and make it easier to find what places in the network need it.
Chances are they already have that big Cisco or Force10 router complete with FPGA based filtering magic.
It can be expensive to get them to act but for a poker site it's a lot cheaper than loosing hands per minute.
We also paid a lot of money for boxes at our end that did real time traffic analysis for various attack vectors. Expensive but again cheaper than loosing money.
However as it started to climb the poker ranks and get more players the early start we had made meant that the network guys were ahead of the hackers/criminals from the start. We had developed a good relationship with our providers (of bandwidth and hardware) and internally stress tested the system.
By the time I left it was WELL worth it. A reasonable sized poker site can easily rake a million dollars a week which is a revenue stream worth defending.
My advice would be that if you think your are going to be vulnerable invest early, make mistakes early, when the big boys come at best you have hours to respond. Investing doesn't have to mean buying the most expensive hardware but planning for how much you can afford to loose will put things into perspective. Deciding how much to spend of defence is important and must be continually reviewed as hardware/software change and new threats pose themselves.
It's also important to remember that the big attacks like DDos are annoying but can normally be defended against. Intrusion detection is as important but much harder to defend against.
I am running Wowza currently, and worry about this.
"If there has been we haven’t noticed. Taking down one wowza server wouldn’t cause us any problems. They would have to take down 5+ servers for us to notice."
If you want to survive large scale DDOS you need equipment that can scrub the incoming unvalidated data in real-time and keep up.
Combining Source Based Remotely Triggered Black Holing (RTBH) with uRPF affords you the ability in a sophisticated network to drop a large amount of undesired traffic (especially if it's from simpler DDOS strategies). If you do in fact have the $$/need for the Arbor then inside of black-holing the traffic you send it to the Arbor and let it scrub the packets.
You can devise a test that identifies attacking clients and then blacklist the IP addresses for a while.
While I'm sure this strategy isn't perfect, it has been sufficient for the two attacks I have been subjected to.
Edit: vladd's reply articulates this idea better than I was able to :)
Still even an simple test could be too slow to counter a big enough attack.
* turn down the TCP handshake timeouts
* turn on TCP SYN cookies
* watch incoming network connections and traffic logs and add firewall rules to block each suspected attacker subnet.
In a real DDoS you have to block the attack traffic at the TCP level, not at the application level (the web server) which means adding firewall rules or iptables.
You will end up blocking a lot of legitimate traffic but better than being 100% down.
Still getting hammered? On to phase 2:
* Launch mirror servers, change your DNS records to point traffic at the mirror. (Most attack bots don't follow DNS changes)
* If it's an application-level attack then put your site in static delivery mode.
Still getting hammered? That'll teach you to hate on Justin Bieber.
Be proactive. Plan for the solutions below in advance, not when traffic is hitting your servers and SSH along with everything else is unresponsive.
1) Availability for investigation
Renice sshd to make sure you can get to your servers when they are hit with CPU usage due to the DDOS. Reserve some connections for the root user on your SQL server (Postgresql already defaults to 3 for this) so you can still connect, investigate and analyse what's happening.
2) Pattern matching
Every solution against DDOS can be described more or less as "pattern matching". There are two steps involved:
-> identify a pattern associated with DDOS requests.
-> match them as early as possible (iptables, upstream provider, hardware firewall) and reject them.
Easiest is to match offending IPs, those that make repeated queries (i.e. once every second). Set the threshold in the right spot - too high and you'll ban valid users, to low and you fail to ban enough in order to restore service availability.
Other things to watch for are at the connection level: either SYN flood behaviour ( http://en.wikipedia.org/wiki/SYN_flood , deploy http://en.wikipedia.org/wiki/SYN_cookies ) or KeepAlive connections that keep httpd busy (always reserve in httpd.conf for example a number of connections that aren't KeepAlive enabled).
Identify pages on your website that are slow to load or generate large, time-consuming SQL-queries. Those are the main target during DDOS attacks; pattern-match their URLs and disable them during attacks if they're not critical for your website, or have a fall-back for them.
Check for errors in the attackers' requests: either they have the same weird-looking user-string, or lack the referrer, or something else that you might be able to find. Pattern-match and reject based on that.
Pattern-match clients that keep requesting the .html without loading the corresponding .css or images -- most probably they are attackers calling the GET API (corresponding to wget-ing the HTML) but don't parse and download the corresponding images on the page. Sure, you'll ban text-browser users but that's a small price to pay in order to restore your website.
Have a low-resource version of the website where you can turn off the requests for CSS, images and so on and just serve plain functional HTML. Pattern-match, if any, those IPs that keep requesting CSS and images previously associated with non-cache-able HTML pages.
And lastly... be proactive, be prepared, have a plan.
I often hear of attacks that are accompanied by an email demanding thousands of dollars via Western Union.
Attackers like this pray on small, non-technical businesses (eg blogs, small retailers) who often feel they have no option but to pay the ransom.
Guess what happens when you pay for the attack to stop? The attack stops, only to start again in 8 weeks time with a new, higher ransom email.
THe thing is, this all depends on scale.
If you are small scale, look at you cost per day. They won't attack you forever - their resources are limited.
So if you're in that situation - okay, it sucks.
First, calculalte the cost of doing business. If they're asking for 10 grand and you're thinking of paying because you'll lose that much in a few days, you're better off getting hooked up with a real DDOS protection provider - it will cost a lot less than that in the long run.
If you make tons of money daily (some affiliate sites, etc) then it may (may) be worth doing both - if one day's business will cost you a quarter million in profits, and you havent' already set yourself up for DDOS, it might be worth letting hte FBI know, paying them off anyway, and then getting protection services in place so it never happens again (cause now you know)
It's the small fish who just can't afford DDOS protection services and might have sites who make just enough money to live on off their site that are in the tough spot, and could be a prime target for the extortionist-type DDOS'ers. I wonder how common that is.
Each one cost me money, but the two in two days cost me thousands. I was hosting a VPS on Rackspace and did everything I could to mitigate it but ultimately the attack stopped because they wanted those resources directed elsewhere.
After the first one, I hooked up with a smart guy on Twitter (http://twitter.com/sempersecurus) that tracks botnets. He allowed me to sinkhole the domain so that he could collect and aggregate the data.
To stay defensive, I put some .htaccess blocks in for countries that are clearly well outside normal buyer patterns. In each case, I know a user (read: a human) saw the campaign and then put the DDoS in place so I hoped, by the .htaccess blocks, to prevent them from seeing the site altogether. But I know I really need to be a much more knowledgable coder to be proficient at deflecting these in the future.
If your business is really worth at least $1000/day then, a real-life protection service that proxies your traffic at a high bandwidth location and sends you back only clean traffic would seem easily within your budget, and worth it ot protect your assett. Let them worry about the constantly changing attack patterns and let you worry about increasing the value of your site.
If so, would one reasonable answer to the question then be "host with Amazon" and let them deal with it for you? Or at least have a backup ready at Amazon to switch to? (With the recent announcement that Amazon's hosting is now PCI level 1 compliant, hosting at Amazon becomes possible for many for whom it was not an option).
keep in mind that you are paying for that DDoS traffic. $0.08 per gigabyte means you are paying something north of $350/hr for a 10Gps DDoS.
Now, I guess if you only expect the traffic to continue for a short while that's not a bad deal at all, but if it goes on for any period of time, that can get expensive fast.
For comparison, one of my providers has offered me a 10 gigabit transit link for around $10.5K/month, so assuming I could use that to DDoS you, (and really, that sort of thing would be... difficult. Far easier to hire out a botnet.) I'd be spending about a dollar to cost you twenty dollars.
Buying a 10 gig link woulnd't be a distributed denial of service attack, and itwould be far too easy for the providers to not only find out who it was and filter it out, but to find out who to sue for all the wasted time and bandwidth and other criminal charges.
My point is that by "handle this sort of thing" in the context of ec2, means "charge you for the traffic and pass it through" which some times is just fine; but quite often it still means a denial of service, if it doesn't bring down your servers, it will eventually bring down your bank account. I mean, I don't know about you, but at north of $350 an hour, my bank account isn't going to last long.
I personally got hit with a flood of many small packets, which overwhelmed my router. The obvious solution here is to buy a better router. (it was 200Mbps of rather small packets on a 1000Mbps pipe.) I ended up just having my upstream blackhole all traffic to the target IP, which is sad, as it 'finishes the job' for the attacker, but on a $12/month account, what do you want?
Now, for what I provide (VPS hosting) an application-level DoS, slowlaris or just hitting your app in an expensive way isn't really my problem. That's on the application programmer, and as this is mostly a form for application programmers, it does make sense that the highest modded comments on this page describe mostly application level DoS mitigation strategies.
Just having one of these machines won't solve all of your problems, but it can certainly help.
This was posted earlier on HN http://news.ycombinator.com/item?id=1986867?
You either get massive uplinks (multi 10GE) and cisco nexus (or similar) or proxy/spread/diversify across multiple facilities separate to the core and only forward legit traffic (or null route targetted proxies).
On the first one, I already had a Squid reverse proxy in front of the servers, so I configured that page to cache and all was good.
On the second one my load balancer was a POS, so we ended up getting the developers to configure the page to 302 to a new page, and the bot didn't follow the redirect.
In both cases, the thing requesting the page wasn't a real web browser, so it didn't take cookies or follow redirects. I ended up getting an F5 shortly after and wrote an iRule to do a cookie check should another DDOS come by, but never had to use it. I was also fortunate that we had enough bandwidth to serve the requests and that it was a resource starvation type attack.
These 4Chan guys attacked MPAA which went to DOSarrest as well and within minutes of being set up there they where back online. They protect a bunch of high end clients who have come under attack and bring on new people all the time.
They also have a nice customer panel where I can log in and change all kinds of settings for my site and view the statistics of the attack. I saw the attack get up to over 500mb/s and have no effect what so ever. I highly recommend them.
Good read, IMHO.
Do any ISPs profile their clients to test for bots?
And they're definitely technology, they're a tool used to attack the owner/normal beneficiary of the tool with that tool (your uplink).
It's when they try to collect money that law enforcement has the best opportunity to catch them. There is a monetary limit over which it becomes internationally interesting to the FBI,HTCU, etc, to get involved and start using international resources to track the money back and make arrests.