Probably worth bookmarking this for when you [hopefully never] have to deal with this same situation.
Customers, especially non-technical ones, don't give a crap. What they want to know is when the service will be back up, and what steps you're taking to prevent it happening in the future, although I'm sure a certain percentage would be interested in why this is happening in the first place (not as in the technical breakdown, but why you didn't have a contingency plan).
If I'm a customer of Basecamp it looks to me like 37Signals is couching this as if they are the victims here, when really I am the victim. They're business isn't being disrupted... mine is! I pay them to abstract me away from the gory details... if I wanted to deal with that stuff I'd pay people to build it in house. My job as a customer isn't to sympathize with an outage, it's to move to a service that won't have one.
After turning in a term paper a day late a wise professor once told me "It doesn't matter if your excuse is true, it's still an excuse." The basic facts are the job didn't get done, and the person to blame is the person who didn't get the job done. Any modern web service that doesn't take the simple effort to sign up for cloudflare or their ilk to reduce attack surface doesn't deserve my money. (Admittedly a harsh perspective to take, but one many do take)
There is an entire movement in Sicily dedicated to highlighting and frequenting businesses that refuse to pay protection money, because in the past, paying was the norm.
Since that's not the kind of society I want to live in, I'd rather stand firm behind a company that refuses to deal with criminals. If companies give in as a matter of convenience to retain customers who turn a blind eye, that will only make the criminals stronger.
Now, certainly, there are measures they can take to mitigate the problem, but with all the things to do in a business, I suppose it's the kind of thing that might not be on the front burner until it happens. There are all kinds of bad, destructive things that could happen in the world, but if you spend all your time worrying about what could happen, you won't have a viable business. It's a tricky balancing act, and I'm willing to cut some slack to someone being targeted by criminals.
If 37Signals was a bitcoin exchange, aka a known target of DDOS attacks, the mood here would be drastically different... yet we've hit a tipping point where it seems everyone is equally at risk. DDOS attacks have become a sad cost of doing business on the internet, and just because you acknowledge that fact and try to prevent yourself from being a target doesn't mean you're capitulating to the criminal enterprise.
In fact, I don't see a better way of sticking it to the thugs than responding with "Hahaha, do your worst. We'd love to see if the money we're paying X COMPANY is worth it." And then you get to write a totally different blog post, one where you get to brag about your excellent foresight and how you have proven to your customers that the money they pay you buys a top-notch service.
you're seriously comparing handing in a term paper late to being targeted for extortion by an international crime syndicate?
of course handing in a term paper late is unexcusable - it's just a fucking essay and there's no reason why it should be late because you probably had weeks to do it.
waking up to find your entire network infrastructure under siege (and anything ELSE you put up as a contingency, because it's on the internet, remember?) is not some shit you can be "no excuses" hardcore about because this is in the real world which is complex, unlike slacking on a paper, which is very simple.
reasonable people know this, which is if you read their TOS and other SLA agreements, this is all spelled out for you. nobody wants ot hear "NO EXCUSES!" from some guy paying $50/month while gigabits worth of malicious traffic is pounding at your door.
the truth is it's YOUR business, just like basecamp is THEIR business which they are QUITE obviously in the middle of running. if you're concerned your $50 saas product is not delivering the goods, it's on YOU to find an alternative.
That's not wise, it's just being an asshole. Reasonable people understand that things happen sometimes despite our best efforts. You can spend your life railing at people getting hit by metaphorical meteors, until you're hit by one yourself, or you can take a minute to work with people, be a little flexible, and win your time "investment" back many times over in return.
And Cloudflare is hardly a panacea for DDOS attacks.
I think most Basecamp users are savvy enough to understand that there's nobody to blame except for the extortionists responsible for this attacck.
Straight up murder doesn't quite fit the situation here.
The fact that this is on a Github Gist, as opposed to a static page (like on s3), suggests an audience that would understand those subtleties.
Basecamp is actually the name of the company now, they aren't 37Signals anymore.
I prefer Github's recent response , clear and helpful but without the rhetoric.
I blackmail my kids all the time... ("Wash your hands after using the bathroom or you will put 25 cents in this jar")
Calling someone "a criminal", meanwhile, degrades their status to that of a common mugger; someone in the lower class who needs to commit crime to survive, and who doesn't have the intelligence required to come up with a clever crime.
Hackers are generally aesthetes--we tend to value our intelligence, curiosity, etc. more than we value our moral fibre. We can appreciate stories like "A hacked into B to see if it was possible, and reported the vulnerability all responsible-like, but then they threw him in jail! How horrible!" because we think the positive-status from the display of intelligence makes it less likely, rather than more, that they were genuinely seeking to harm the people they hacked.
Because of this, I think we here are scared of being potentially associated with dumb, low-status, lower-class criminals more than we are of just being considered evil. People hire "evil, black-hat" hackers. Nobody hires a dumb hacker.
There is moral judgement involved with calling someone a criminal, and rightfully so. Taking what other people have created by force or extortion degrades society.
I was actually marveling at how precise the wording is in this piece. Curious how different these things can come across.
(and as a message to the DDOSer - they're likely to be reading this too and reminding them it is criminal and law enforcement is involved might make them reconsider the attack)
Burglary and murder happen too. No reason to hold your language back. Not even lawyers and prosecutors do, and they deal with those everyday.
For the company loosing millions or the Basecamp client whose unable to enter his account, that "those things happen" is not much of a response.
> This attack was launched together with a blackmail attempt that sought to have us pay to avoid this assault.
The original comment said that because DDoS could be illegal, is was different from openly carrying a fire-arm; that assumes that openly carrying a firearm isn’t illegal. It often is, outside of the US -- hence my response.
I would have appreciated you didn’t downvote me before you understood that.
There's certainly a bit of knowing your audience here.
> This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way.
It is. Only 4 words into the DDoS announcement and I rolled my eyes. I think that's a record for DHH.
I am not completely sure what this even means, but I am sure there is irony in there!
>This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way.
If this is truly 110% true, they couldn't even ssh into their servers (in before, "You don't ssh each into individual shards"). Which I'm betting they can, which means their are still attack vectors to exploit.
I'm surprised that on a technical forum there is still this complete misconception of what a DoS is.
Each level of the OSI model can be attacked in a DDoS, and its still of Denial of Service attack. I.E.: You can hold down a pre-2009 windows server with as little as 10-15 packets per second, totally less then a kilobyte per second.
Yes if your have a 10/100/1000 nic and your getting 20Gb/s then yeah nothing's getting in. Or if you have a 10Gb/s router, then 20Gb/s will freeze the whole data center. But if you have a 100Gb/s router with a 4x bounded 10Gb/s nics. And your service is down, then it isn't your hardware, but your software. Your kernel, sockets.h, are still processing packets, you can still be attacked.
There are a few gotchas, including if I understand it correctly the need to "retry twice" when you try to SSH in your server when a DDoS is going on but...
OVH doesn't even feel a 85 Gbps attack (let alone a 20 Gbps one like in the article). They can deal with attack much larger than that automatically.
They seem to have very good DDoS protection against the "flood" type of DDoS. And this is pretty much transparent to users.
I hope more and more hosting company start implementing similar anti-DDoS features: more competition would bring better protection against flood-type DDoS and cheaper price.
Here's the explanation as to how their system works (in french but there are several graphics):
Basically as soon as a DDoS trying to saturate your server(s) is detected the attacker faces the problem of needing to DDoS... OVH itself.
And the DDoS doesn't even make it to your server while the legitimate trafic still does.
I find it great that there are people actually looking for solutions to the DDoS issue.
Though a friend at another related service had been kicked from two VPS providers due to receiving a few DDoS attacks.
These providers claimed it was against their Terms of Service and ejected him as a customer.
That day he learned it is best to keep offsite-cross-company backups of everything, since he did not get a single byte from his machines.
Storage is so cheap these days there is no excuse not to keep client data for at least a month.
I'm a big fan of OVH.
It just shouldn't be a surprise anymore that DDOS's happen.
In practice, there is. If we're talking about an HTTP flood, the other endpoint address is always validated (due to the 3-way handshake) so it's plausible to rate limit and block individual addresses. (But without validated client addresses, the rule is to NEVER create state off those, because spoofing is too easy.)
We've taken some steps since then to hopefully reduce our vulnerability. I'd be really interested in a DDoS protection best practices guide for small SaaS businesses.
I'm sure a few of you out there are readying this thinking "too bad for Basecamp, but this will never happen to us because we aren't an interesting target." That's what we thought too...
I was hired as a CEO at an <unnamed> company ($200m+ revenue) and we were hit by this type of attack.
Every second of being down cost us literally $10k, so we quickly negotiated with criminals for $5k one time payment and they stopped the attack.
Unfortunataly a few weeks later we were hit by 3 new attacks. Apparently the word had spread and these new attackers demanding $50k.
We were not going to pay $50k but I was also unable to stop the attacks. I was let go a few days later as we had a down time of 2 days and I wasn't able to fix this problem.
Source: 3rd world south america
Usually it was because they were hired to do so by a competitor. Every time they would claim to demand a ransom, although they didn't expect for it to be paid. It just made people less suspicious.
The attack is pretty clever, being indirect it is hard to trace and because bittorrent allows arbitrary ports you can hit a specific ip & port pair.
The one downside is the victims can be sure it is a bittorrent DDOS by checking the attacking connection's requests. The attacker's packets will contain bittorrent's magic connection bits.
Please confirm my understanding: this would be by inserting yourself into the DHT with an address near/equal to a target high-volume torrent, so that you're frequently queried by clients looking for peers?
If so, I guess it could be possible in some cases to identify the peers who initiated the attack. The non-malicious peers attempting to make BitTorrent connections to your server will provide the infohash of the torrent they think you're downloading, which you might be able use to find the malicious DHT peer who's directing them.
At first I thought you were suggesting that it's possible to for malicious peers to insert invalid IP/port pairs into non-malicious DHT nodes, which I don't believe is possible. (The mainline DHT protocol  requires that peers provide a "token" value, previously sent to their IP address, to verify themselves when being listed for a torrent.)
ISTM that once you've determined bittorrent is the attack vector, the hard part is done? Is dropping by "magic bits" harder than dropping by ip/port?
a strong and fast-moving stream of water or other liquid.
"rain poured down in torrents"
2. a sudden, violent, and copious outpouring of (something, typically words or feelings).
"she was subjected to a torrent of abuse"
Edit: Sorry, misread the post.
Otoh that's where the opportunity is. The fact that "customers have no clue". People pay you for something that they can't do themselves or that you make easier for them to do.
 - https://www.quakenet.org/articles/102-press-release-irc-netw...
If so, how did that go?
Did Amazon help?
I don't know how you would get feds to pay attention?
Or you could've just typed FBI, like a normal person.
It's a different matter if the attacker is based in a country where DOS are legal or that doesn't have any extradition treaty with the US, but that still needs to be established.
Properly ignoring invalid requests can be a challenge, the process of doing so will depend on the type of attack being used. SYN floods can difficult since the src IP is most likely invalid. The attacks we've seen with DNS and NTP amplification are difficult as the attack isn't trying to get your servers to respond, they are just flooding your incoming pipe with data. If they are trying to abuse some page within your application you can more easily mitigate it as you'll know the source IP of the request so it can be blacklisted.
EDIT: a few more details:
SYN flood: http://en.wikipedia.org/wiki/SYN_flood
DNS Amplification: http://blog.cloudflare.com/deep-inside-a-dns-amplification-d...
As for mitigation, while we hear about Cloudflare a lot, AT&T and other large providers can provide DDOS protection for leased lines. Basically what happens, before the data gets to your leased lines, traffic headed to you will go through AT&T's DDOS detection/prevention systems that attempts to filter bad traffic. This type of service would apply more to companies like Linode or possibly the datacenter that they are housed in.
It appears Basecamp only has a /23, so even if they redirected traffic through Cloudflare, the attacker could still find their direct servers fairly easily and attack that IP. It's still possible to block, but not quite as easy as setting up Cloudflare.
Why would it be easier for the attacker to find their direct servers if they only have a /23 - doesn't Cloudflare obscure the identity/location/IP of the server on the other side?
Can the upstream to the actual server restrict traffic to known Cloudflare blocks?
The ISPs will help during a DDOS but response times are slow and we haven't tried getting them to put this type of block in place yet.
I'll leave it to others to answer this (for this situation) but keep in mind also that adding cloudflare also adds an additional layer that can fail for different reasons.
That tradeoff may well be worth it for certain high visibility web properties but maybe not if you are a low value target.
There are pros and cons to any decision you make that depend on specific circumstances.
That's obviously very much dependent on the kind of attack and whether CloudFlare has more network capacity than Basecamp (which I would imagine is highly likely).
Because of the unique design of our network, I'm unaware of any other service that has as much capacity that can be utilized in aggregate to mitigate large-scale attacks.
Co-founder & CEO, CloudFlare
I've seen articles before saying online gambling websites often do pay up as the downtime isn't just lost revenue but customers going elsewhere.
Apart from being distributed, the insidious power of DDoS appears to lie in "subscriber-calling-server". Why not go the other way around? At least only for specific subscription services, not general purpose web access.
The situation of a DDoS attack is first communicated by the web service provider texting a subscriber, who texts back their present IP address. The web service provider then "calls" the subscriber from a hitherto unknown IP address. Of course, that address could be leaked too, but at least it's not obvious public knowledge like a DNS entry.
Sounds like circuit switched telephony/modems rather than packet switching, but can it be implemented in software?
Set up CF, only allow traffic from CF.
On another note, having CF monitor an attack like this could help them do more research into mitigating these attacks in general and allow them to try and hunt the attacker. They tend to make things like this public which would benefit everyone.
> As I noted in a talk I gave last summer with Lance James at the Black Hat security conference in Las Vegas, a funny thing happens when you decide to operate a DDoS-for-hire Web service: Your service becomes the target of attacks from competing DDoS-for-hire services. Hence, a majority of these services have chosen to avail themselves of Cloudflare’s free content distribution service, which generally does a pretty good job of negating this occupational hazard for the proprietors of DDoS services.
I could post more, but why bother?
CloudFlare is firm in our belief that our role is not that of Internet censor. There are tens of thousands of websites currently using CloudFlare's network. Some of them contain information I find troubling. Such is the nature of a free and open network and, as an organization that aims to make the whole Internet faster and safer, such inherently will be our ongoing struggle. While we will respect the laws of the jurisdictions in which we operate, we do not believe it is our decision to determine what content may and may not be published. That is a slippery slope down which we will not tread.
As a result, both the Israeli Defence Forces and Hamas are CloudFlare customers. Unless one of their customers is doing something that is unambiguously illegal (e.g. hosting child pornography), CloudFlare won't cut them off just because they're doing something that some people regard as "bad".
It's a very principled stance and one that I respect.
Information isn't really the question here. These aren't sites telling people how to conduct DDOS attacks, these are sites where you pay them, and they run a DDOS for you. This effectively silences someone until they either give up on their message, or sign up for expensive DDOS mitigation packages (or Cloudflare).
You may consider that to be free speech. I don't.
You clearly don't and you're entitled to your opinion.
The krebs story was interesting thanks, the forum posts less so. I understand why cloudflare are reluctant to start rejecting customers based on content, but surely it's illegal to sell DDOS services? Perhaps they should change their TOS to exclude any sites which sell attack tools/services, because it looks really bad for them to be protecting sites that promote DDOS, which then provides them with repeat business.
Are there still sites up protected by cloudflare which promote this sort of activity?
Selling attack tools, however, is explicitly legal in most places, it's just software just as a port-scanning tool, DeCSS or zero-day vulnerability data.
"Promoting this sort of activity" again is free speech issue, no matter what "that sort" is. For example, there are posts right here in HN that "promote this sort of activity", and it would be ridiculous if having such content is even close to allowing someone to take down a server.
In short, unless the actual site is performing illegal activities (implementing the DDoS or uploading childporn&stuff), I'd say that they're correct in explicitly ignoring whatever else the site is doing.
a great many of today’s DDoS attacks are being launched or coordinated by the same individuals who are running DDoS-for-hire services (a.k.a “booters”) which are hiding behind Cloudflare’s own free cloud protection services.
I don't see Matthew Prince's post quoted above as a satisfactory response to this. This is morally and legally shady because cloudflare directly profit from the continued existence of DDOS, so they should be very careful to offer not a shred of evidence that they currently support people who carry out DDOS IMO, it would just be good business and current customers are going to get restless if they find cloudflare protects DDOS sites knowingly.
They've obviously taken a different stance (based on not wanting to filter customers on content), which I'm sympathetic to, but if the content is illegal and directly benefits them by facilitating more DDOS attacks, that equation changes.
I'm sure there's tons more, but why bother compiling a list when nothing will change. If you're curious, a good place to look would be the hackforums 'DDOS as a service' section. I bet a lot of the active ones would go to cloudflare.
And I don't think Basecamp is technically "cloud", but collocated. They appear to own most or all of their servers.
Like something that tries to find exploits on the machines used in the attack and try to shut them down, close their internet connection or inject a self-targeting DNS or something of the sort?
This is a huge constraint for the people (e.g. at Microsoft) who work to identify and take down botnets: they expose themselves to significant legal/PR risk if they do anything harmful to the bots.
And even if it were legal, you'd still have to deal with all of the "$SELF_DEFENDER broke my web site" PR unpleasantness from the innocent bystanders.
That doesn't sound like self-defense at all :)
I any case let me clarify what was my purpose as it seems I'm not good at analogies. The point is that you're attacked using compromised computers so it is incredibly stupid to retaliate to the source of the attack.
Hope that clarifies!
Hope that destroys your absurd misconception!
I'm interested about what he means by quarantine.
Does it mean that ISP's will stop accepting traffic going to their servers?