Hacker News new | comments | ask | show | jobs | submit login
Basecamp was under network attack (gist.github.com)
266 points by ibsathish on Mar 24, 2014 | hide | past | web | favorite | 185 comments

Some great language there: framing it as an attack by criminals (gains sympathy from users), explains in plain-terms what a DDOS is (front door analogy), emphasizes (twice!) that user data is safe, apologizes for the likely downtime, informs people where to get updates.

Probably worth bookmarking this for when you [hopefully never] have to deal with this same situation.

I'm going to play devil's advocate and completely disagree with you here :)

Customers, especially non-technical ones, don't give a crap. What they want to know is when the service will be back up, and what steps you're taking to prevent it happening in the future, although I'm sure a certain percentage would be interested in why this is happening in the first place (not as in the technical breakdown, but why you didn't have a contingency plan).

If I'm a customer of Basecamp it looks to me like 37Signals is couching this as if they are the victims here, when really I am the victim. They're business isn't being disrupted... mine is! I pay them to abstract me away from the gory details... if I wanted to deal with that stuff I'd pay people to build it in house. My job as a customer isn't to sympathize with an outage, it's to move to a service that won't have one.

After turning in a term paper a day late a wise professor once told me "It doesn't matter if your excuse is true, it's still an excuse." The basic facts are the job didn't get done, and the person to blame is the person who didn't get the job done. Any modern web service that doesn't take the simple effort to sign up for cloudflare or their ilk to reduce attack surface doesn't deserve my money. (Admittedly a harsh perspective to take, but one many do take)

Reasonable people realize that unforeseen things happen, and might empathize with someone being targeted by a criminal enterprise a bit more than someone who just forgot to pay the electricity bill.

There is an entire movement in Sicily dedicated to highlighting and frequenting businesses that refuse to pay protection money, because in the past, paying was the norm.


Since that's not the kind of society I want to live in, I'd rather stand firm behind a company that refuses to deal with criminals. If companies give in as a matter of convenience to retain customers who turn a blind eye, that will only make the criminals stronger.

Now, certainly, there are measures they can take to mitigate the problem, but with all the things to do in a business, I suppose it's the kind of thing that might not be on the front burner until it happens. There are all kinds of bad, destructive things that could happen in the world, but if you spend all your time worrying about what could happen, you won't have a viable business. It's a tricky balancing act, and I'm willing to cut some slack to someone being targeted by criminals.

I more or less agree with you, but that's kind of a false dichotomy, isn't it? Signing up for cloudflare or using a CDN isn't giving in, it's taking measures to protect yourself (and that's ignoring the other benefits you get). The unfortunate fact is DDOS attacks are becoming a daily occurrence, and if you have something to lose you should probably take measures to counteract any possible threats.

If 37Signals was a bitcoin exchange, aka a known target of DDOS attacks, the mood here would be drastically different... yet we've hit a tipping point where it seems everyone is equally at risk. DDOS attacks have become a sad cost of doing business on the internet, and just because you acknowledge that fact and try to prevent yourself from being a target doesn't mean you're capitulating to the criminal enterprise.

In fact, I don't see a better way of sticking it to the thugs than responding with "Hahaha, do your worst. We'd love to see if the money we're paying X COMPANY is worth it." And then you get to write a totally different blog post, one where you get to brag about your excellent foresight and how you have proven to your customers that the money they pay you buys a top-notch service.

That's a bit naive though. People can always find ways to hurt you - it's a very asymmetric fight. With a complex application such as Basecamp, you can't really put everything behind a cdn.

That's why I actually think that their thrust on pursuing the legal/FBI route is a good one, especially if they achieve any success there. This extortion/racket is indeed criminal and not tolerable. It would be good to catch the racketeers and make an example of them.

Disagree. Understanding the root cause helps even non-technical customers make the right decision. For example - "If I move to a different service (competitor of Basecamp), is there a chance that I will run into this issue there too? Answer is yes, based on how DHH explained the problem." Customers understand that shit happens. Particularly because many Basecamp users are business owners and can relate to shit happening in their business too. Explaining the root cause in plain language, and emphasizing that the user data is safe is a great way to deal with this situation.

> "It doesn't matter if your excuse is true, it's still an excuse."

you're seriously comparing handing in a term paper late to being targeted for extortion by an international crime syndicate?

of course handing in a term paper late is unexcusable - it's just a fucking essay and there's no reason why it should be late because you probably had weeks to do it.

waking up to find your entire network infrastructure under siege (and anything ELSE you put up as a contingency, because it's on the internet, remember?) is not some shit you can be "no excuses" hardcore about because this is in the real world which is complex, unlike slacking on a paper, which is very simple.

reasonable people know this, which is if you read their TOS and other SLA agreements, this is all spelled out for you. nobody wants ot hear "NO EXCUSES!" from some guy paying $50/month while gigabits worth of malicious traffic is pounding at your door.

the truth is it's YOUR business, just like basecamp is THEIR business which they are QUITE obviously in the middle of running. if you're concerned your $50 saas product is not delivering the goods, it's on YOU to find an alternative.

> It doesn't matter if your excuse is true, it's still an excuse.

That's not wise, it's just being an asshole. Reasonable people understand that things happen sometimes despite our best efforts. You can spend your life railing at people getting hit by metaphorical meteors, until you're hit by one yourself, or you can take a minute to work with people, be a little flexible, and win your time "investment" back many times over in return.

And Cloudflare is hardly a panacea for DDOS attacks.

It's not as if this service failure is due to incompetence. And we don't know what counter-measures they used to mitigate this attack. It's impossible to be unaffected by a DDoS unless your Google or Facebook (with warehouse-sized server facilities).

I think most Basecamp users are savvy enough to understand that there's nobody to blame except for the extortionists responsible for this attacck.

So if a pizza delivery guy gets shot on the way do you still demand better service? Just trying to see if you believe in the principle or just the practical aspect. :)

Better analogy would be if the "criminals" flooded the streets with bicycles or cars preventing the pizza delivery guy from delivering your order.

Straight up murder doesn't quite fit the situation here.

Yes, that would be a better analogy. However, I was not trying to make an analogy. I was testing if the person held a principled (absolute) or practical (relative) view.

I'm guessing relative.

Customers, especially non-technical ones, don't give a crap.

The fact that this is on a Github Gist, as opposed to a static page (like on s3), suggests an audience that would understand those subtleties.

> If I'm a customer of Basecamp it looks to me like 37Signals is couching this

Basecamp is actually the name of the company now, they aren't 37Signals anymore.

Not sure why you got voted down (hopefully my vote will put it back at 1). I think it's a legitimate point of view. I can certainly imagine some company out there mad at 37signals because they can't get work done because of the attack, wasting thousands of dollars of labor.

I always liked how the Japanese apologised. There is no excuse as its irrelevant, all you get is an apology, compensation and how/why it wont occur again. Not sure if that was an industry specific thing but it sure was effective.

Some customers don't care. Many do. I personally do. When a business can explain what happened it makes me not only like them more, but become a little more loyal.

I agree, though blackmail seems inaccurate. I've always understood blackmail to be a demand backed by a threat to reveal secret information[1]; this sounds more like an extortion racket[2].

[1] http://en.wikipedia.org/wiki/Blackmail#United_States

[2] http://en.wikipedia.org/wiki/Extortion_racket

Yea, extortion seems more apt here. It's like a square and rectangle. All blackmail involves extortion, but not all extortion is blackmail.

It is extortion, not blackmail. While blackmail is a form of extortion, it's as you say - threats to reveal potentially damaging information.

They use "criminals" 5 times in that short statement. IMO the overuse of emotive language is unnecessary and belies the emotional state of the author. Stay professional and detached—it's a DDoS, I've no doubt it's frustrating but they happen.

I prefer Github's recent response [0], clear and helpful but without the rhetoric.

[0] https://github.com/blog/1796-denial-of-service-attacks

Rhetoric? You've got people who just attempted to blackmail you and then take your service offline when you refuse. The descriptive term "criminal", i.e. one who breaks laws, is perfectly valid IMO.

Of course it's valid, rhetoric != lies.

While I agree, the term blackmailer or extortionist would had been better.

Which are just specific types of criminals. I don't see the problem.

Criminals are just specific types of people, and people are just specific types of mammals. Being more specific sometimes aids understanding.

I think I get what you're saying here; it's an example of the non-central fallacy[1]. Calling someone "a criminal" calls to mind a set of stereotypes, to which blackmail/extortion don't quite fit (most crimes for which one gets called "a criminal" as a generic term are violent, for just one thing.) Calling them "a blackmailer" or "an extortionist" calls to mind a more accurate set of stereotypes, clustered more closely with how you'd react to kidnappers, con-artists, etc. than how you'd react to, say, a mugger.

[1] http://lesswrong.com/lw/e95/the_noncentral_fallacy_the_worst...

Not all blackmail is a crime.

I blackmail my kids all the time... ("Wash your hands after using the bathroom or you will put 25 cents in this jar")

That's not blackmail ... "wash you hands or I'll tell your sister that you killed her pet fish" ... is blackmail. What your describing is more like extortion.

it's just framing the scenario in good guys vs bad guys terms, it's childish regardless of how accurately the term describes the actors involved..

Why is it childish to point out when someone is acting criminally — in a literal sense being a bad guy? Is it somehow more adult to act as though you are morally equivalent to an extortionist?

I think people might be being offended-by-proxy by a sort of status-shift 37s is trying to work into its language. Calling someone "an extortionist" still implies a sort of high-status white-collar cunning-and-intelligence, of the kind you'd expect of a person in the tech industry. An evil person, surely, but the respectable, movie-villain-you-love-to-hate kind of evil.

Calling someone "a criminal", meanwhile, degrades their status to that of a common mugger; someone in the lower class who needs to commit crime to survive, and who doesn't have the intelligence required to come up with a clever crime.

Hackers are generally aesthetes--we tend to value our intelligence, curiosity, etc. more than we value our moral fibre. We can appreciate stories like "A hacked into B to see if it was possible, and reported the vulnerability all responsible-like, but then they threw him in jail! How horrible!" because we think the positive-status from the display of intelligence makes it less likely, rather than more, that they were genuinely seeking to harm the people they hacked.

Because of this, I think we here are scared of being potentially associated with dumb, low-status, lower-class criminals more than we are of just being considered evil. People hire "evil, black-hat" hackers. Nobody hires a dumb hacker.

Calling someone a criminal degrades their status from someone who doesn't commit crime to someone who does. It degrades them from someone who adds value to someone who takes value.

There is moral judgement involved with calling someone a criminal, and rightfully so. Taking what other people have created by force or extortion degrades society.

I agree with your general sentiment wrt to good and bad, but saying these people are criminals is just plainly accurate, and specifically not attributing "badness" at all. It's extortion, which is forbidden for very good reasons and as far as I know, uncontroversially so.

I was actually marveling at how precise the wording is in this piece. Curious how different these things can come across.

That is the scenario, it's not just framing it that way, it actually is that way. There's nothing childish about it.

It is criminal behaviour. It reinforces to clients that the attack is not legal, and that they are not to be tolerated.

(and as a message to the DDOSer - they're likely to be reading this too and reminding them it is criminal and law enforcement is involved might make them reconsider the attack)

Yes, but you don't have to repeat it five times; it seems that you are pushing the thing

>Stay professional and detached—it's a DDoS, I've no doubt it's frustrating but they happen.

Burglary and murder happen too. No reason to hold your language back. Not even lawyers and prosecutors do, and they deal with those everyday.

For the company loosing millions or the Basecamp client whose unable to enter his account, that "those things happen" is not much of a response.

I like the "criminals" language. It's unfortunate they need to use it, because it points out that many people think this sort of thing is more like youthful hijinks--a type of vandalism, say--as it was when the Internet was younger. Repeating the word criminals is an excellent way to change the tenor of discussion on this topic in the public mind. I hope all companies that are ddos'd will do it, until it becomes redundant.

Actually, it’s not ‘a DDoS’ but a blackmail attempt, using a DDoS. That’s like confusing someone open-carrying a gun and an armed robbery.

> This attack was launched together with a blackmail attempt that sought to have us pay to avoid this assault.

While I know this is a little pedantic, I'm pretty sure the analogy falls down a bit -- denial of service attacks are often illegal (for instance, in the US it's possible for them to be prosecuted under the Computer Fraud and Abuse Act or even under trespassing or contract laws). Even without the blackmail attempt this could still be considered a criminal act.

So are open carry in most countries. You don’t come off as pedantic, just US-centric.

The US is far from the only country to make DDoS a crime or tort in various situations.

Of course not, and that was not my point.

The original comment said that because DDoS could be illegal, is was different from openly carrying a fire-arm; that assumes that openly carrying a firearm isn’t illegal. It often is, outside of the US -- hence my response.

I would have appreciated you didn’t downvote me before you understood that.

Fair enough, but the comment you had replied to had assumed quite reasonably that DDoS is a criminal act, and I can only assume your response quibbling about that was $SOMETHING-centric. Apologies for the downvote but that impression changed my interpretation of your later comment.

Personally I would rather them show some emotion, as it shares their frustration and anger at these idiots. Also, "criminals" reiterates to the attackers and potential copy-cats, this really is a criminal act and you can be punished.

Github's response would be a whole lot of technical, unhelpful nonsense for most basecamp users.

There's certainly a bit of knowing your audience here.

Yes, "criminals" is much too harsh. Let's replace it with "unfortunately misguided souls xoxo".

Except GitHub's audience is very different from Basecamp's. The first rule of any communication is - know your audience. Well played by DHH.

Yup, they're doing a great job in couching it in language that their customers can understand regardless of technical background. It's easy to forget that services like Basecamp service a huge swath of people who wouldn't necessarily understand what's going on without that sort of copy massage.

Actually, I see the word 'attack' as an emotional trigger and maybe not ideal. I much preferred their analogy that assures non-technical users (on github???) and makes sense:

> This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way.

Some great language there

It is. Only 4 words into the DDoS announcement and I rolled my eyes. I think that's a record for DHH.

Sounds like your issue is with DHH and not necessarily the copy.

Totes true. His selection of words is in one of the bombastic veins that rubs me the wrong way. It goes beyond just one piece of writing, that does make my issue with the writer.

>"His selection of words is in one of the bombastic veins"

I am not completely sure what this even means, but I am sure there is irony in there!

I agree that DHH is such a great writer. He used the metaphor of a people blocking down your house so that non-technical users can easily understand what he's saying.


Explain please.

The problem he may have is its a bit to reassuring. They claim user data is safe while being under attack. This is conceptually very similar to teaching kids that if you duck and cover during an ICBM strike you'll be fine.

>This is like a bunch of people blocking the front door and not letting you into your house. The contents of your house are safe -- you just can’t get in until they get out of the way.

If this is truly 110% true, they couldn't even ssh into their servers (in before, "You don't ssh each into individual shards"). Which I'm betting they can, which means their are still attack vectors to exploit.

Not sure that your missile analogy holds. Most DDoS attacks do not attempt to crack logins to servers, but rather try to flood the servers with as much garbage as possible. Besides, even if they were trying to crack the SSH password, a properly secured server (long passwords/public key auth + fail2ban) should be fine.

Okay here is a better one. Just because people are blocking each other trying to run into your front door doesn't mean they (or somebody else) aren't cutting open your windows, picking the lock on your garage door, or trying to climb down your chimney.

Yes you do. Because DDoS, so like the service is down for users and attackers.

I'm surprised that on a technical forum there is still this complete misconception of what a DoS is.

There are different types of DDoS attacks.

Each level of the OSI model can be attacked in a DDoS, and its still of Denial of Service attack. I.E.: You can hold down a pre-2009 windows server with as little as 10-15 packets per second, totally less then a kilobyte per second.

Yes if your have a 10/100/1000 nic and your getting 20Gb/s then yeah nothing's getting in. Or if you have a 10Gb/s router, then 20Gb/s will freeze the whole data center. But if you have a 100Gb/s router with a 4x bounded 10Gb/s nics. And your service is down, then it isn't your hardware, but your software. Your kernel, sockets.h, are still processing packets, you can still be attacked.

yes, quite strokes neckbeard

I take it at one point people will start to believe that I work for OVH (I really don't) but... OVH has a mandatory DDoS protection on all its dedicated servers: fees have been slightly raised to take that mandatory protection into account.

There are a few gotchas, including if I understand it correctly the need to "retry twice" when you try to SSH in your server when a DDoS is going on but...

OVH doesn't even feel a 85 Gbps attack (let alone a 20 Gbps one like in the article). They can deal with attack much larger than that automatically.

They seem to have very good DDoS protection against the "flood" type of DDoS. And this is pretty much transparent to users.

I hope more and more hosting company start implementing similar anti-DDoS features: more competition would bring better protection against flood-type DDoS and cheaper price.

Here's the explanation as to how their system works (in french but there are several graphics):


Basically as soon as a DDoS trying to saturate your server(s) is detected the attacker faces the problem of needing to DDoS... OVH itself.

And the DDoS doesn't even make it to your server while the legitimate trafic still does.

I find it great that there are people actually looking for solutions to the DDoS issue.

Products by people like Arbor Networks (http://www.arbornetworks.com/) helps with this -- I think they essentially observe traffic patterns and siphon what they detect to be DDoS traffic to alternate routers at the edge of the network to study and blackhole.

OVH uses Peakflow as one component of its mitigation system.

I have a service on OVH myself.

Though a friend at another related service had been kicked from two VPS providers due to receiving a few DDoS attacks. These providers claimed it was against their Terms of Service and ejected him as a customer. That day he learned it is best to keep offsite-cross-company backups of everything, since he did not get a single byte from his machines.

Who are these providers that just delete client data? I run a small datacenter for our niche of clients and when the contract terminates or the project is finished I box up all their data and fire it off to S3.

Storage is so cheap these days there is no excuse not to keep client data for at least a month.

Claiming it was against the terms might be an easy out for them but is silly since being a target is outside of your control, for the most part. Hosts will usually null route customers without sympathy to protect other customers so it's the price of doing business.

It makes a DDoS an even better extortion. "Pay up or we'll get you kicked from your hosting provider."

"...and potentially lose all of your data, if you haven't been planning ahead"

Yes, I also have an OVH server, and I've gotten the email "You're getting DDOSed, we're handling it" (paraphrasing) about a half dozen times. Each time, it's a seamless transition.

I'm a big fan of OVH.

IMO if your business depends in your site being up, DDOS protection should be mandatory. You should budget for it and have it on or ready to go on short notice.

It just shouldn't be a surprise anymore that DDOS's happen.

What happens when a ddos is indistinguishable from regular traffic? Or is it the case that it almost always follows a particular pattern?

It's always a matter of capacity. If there is absolutely no attribute you can distinguish it by, you have no choice but to handle it like normal.

In practice, there is. If we're talking about an HTTP flood, the other endpoint address is always validated (due to the 3-way handshake) so it's plausible to rate limit and block individual addresses. (But without validated client addresses, the rule is to NEVER create state off those, because spoofing is too easy.)

We got hit by a DDoS about a year ago. Rackspace (who normally has amazing support) quietly null routed us and went about their day. No heads-up, trouble ticket, or any other form of notification. They didn't even put a note in our account so when we contacted their support to figure out why our servers were unresponsive outside their network the poor guy who answered the phone was just as confused as I was.

We've taken some steps since then to hopefully reduce our vulnerability. I'd be really interested in a DDoS protection best practices guide for small SaaS businesses.

It's worth adding that when we got hit, we were relying on "security through obscurity". I slept well at night because I thought nobody would be interested in DDoS'ing little ol' us when there are plenty of big fish out there to go after.

I'm sure a few of you out there are readying this thinking "too bad for Basecamp, but this will never happen to us because we aren't an interesting target." That's what we thought too...

I'm running a small SaaS business. I'm curious to hear what steps you took to reduce your vulnerability. Could you please share so others can take the same steps?

The biggest thing we did was remove our dependency to a single IP (this was a unique requirement of our business). We also improved our firewall and upped our managed service level. We're not 100% bullet proof now, but definitely better than we were. I'd be happy to go into more detail offline.

Thanks! I'm on managed service as well, so I may be able to request some of those things. I've never been hit but sounds like I should be proactive about this.

Yep Rackspace did little to nothing to help us but null routing.

I've had really negative experience with these type of criminals.

I was hired as a CEO at an <unnamed> company ($200m+ revenue) and we were hit by this type of attack.

Every second of being down cost us literally $10k, so we quickly negotiated with criminals for $5k one time payment and they stopped the attack.

Unfortunataly a few weeks later we were hit by 3 new attacks. Apparently the word had spread and these new attackers demanding $50k.

We were not going to pay $50k but I was also unable to stop the attacks. I was let go a few days later as we had a down time of 2 days and I wasn't able to fix this problem.


That's a good reason why it's never a good idea to pay for DDOS threats - in many other popular extortion scenarios such as kidnapping, blackmail w. secret info or mafia 'protection money' for storefronts, the deal generally doesn't allow other, new attackers to make the same demands, so you actually are getting some protection - but here it does simply mark you as vulnerable.

Same goes when bribing a cop here. If you bribe too much you're targeted as easy money among the other cops here. Say for example you're caught driving without your insurance, you bribe and then every other cop knows you don't have insurance and squeeze you for money left and right.

Source: 3rd world south america

Why did it fall on your shoulders and not the CTO / tech team? How did they let you go in a few days? They called a board meeting immediately after to fire you?

Although a smaller service, we were in a similar situation a couple of years ago. We assumed it was a competitor because there were not monetary requests, just a massive DDoS via torrents that lasted almost a week. Data center didn't help us in any way... it was crazy. Worst thing is that 90% of customers have no clue what a DDoS is and how hard it is to handle.

I used to know people who performed these types of DDoS attacks.

Usually it was because they were hired to do so by a competitor. Every time they would claim to demand a ransom, although they didn't expect for it to be paid. It just made people less suspicious.

A competitor using a DDOS against you seems like a very bad idea. A likely outcome, for a popular service, is that you get free press as a result. The news, combined with the way Bootcamp has handled this, will probably increase their business.


How is torrents protocol used to DDoS you? I never came across torrents being used as a DDoS. I would appreciate more details on what sort of torrent attack it was, and whether you found any ways of partially neglecting damage.

A malicious tracker, or a peer if using DHT, can claim an IP, the victim, is active in the swarm and has valuable bits of the torrent. Then torrent clients will try to connect to the victim.

The attack is pretty clever, being indirect it is hard to trace and because bittorrent allows arbitrary ports you can hit a specific ip & port pair.

The one downside is the victims can be sure it is a bittorrent DDOS by checking the attacking connection's requests. The attacker's packets will contain bittorrent's magic connection bits.

or a peer if using DHT

Please confirm my understanding: this would be by inserting yourself into the DHT with an address near/equal to a target high-volume torrent, so that you're frequently queried by clients looking for peers?

If so, I guess it could be possible in some cases to identify the peers who initiated the attack. The non-malicious peers attempting to make BitTorrent connections to your server will provide the infohash of the torrent they think you're downloading, which you might be able use to find the malicious DHT peer who's directing them.

At first I thought you were suggesting that it's possible to for malicious peers to insert invalid IP/port pairs into non-malicious DHT nodes, which I don't believe is possible. (The mainline DHT protocol [1] requires that peers provide a "token" value, previously sent to their IP address, to verify themselves when being listed for a torrent.)

[1]: http://www.bittorrent.org/beps/bep_0005.html

It sounds like you have a better understanding of DHT than me and tt sounds like DHT isn't vulnerable like traditional tracker. My knowledge of the attack method is served to what I read in a research paper 2 years back.

I'm always amazed at the clever ways people come up with to use non-aware clients for malicious purposes.

The attacker's packets will contain bittorrent's magic connection bits.

ISTM that once you've determined bittorrent is the attack vector, the hard part is done? Is dropping by "magic bits" harder than dropping by ip/port?

Yes. Very much harder. One can be done at line rate on any halfway decent router, and the other requires deep packet inspection which is considerably more expensive.

In theory yes, but it requires deep packet inspection to catch before it hits the server. Such equipment is expensive per GB/s and not something you'd have access to by accident.

unfortunately i don't have the technical details, we weren't 100% sure but it seems there's a way to exploit BitTorrent by misdirecting clients to send their traffic toward any host. We ended blocking out ranges of ip's but at a point you end up cutting a lot of legitimate traffic as well (but i really lack the technical expertise to go more into depth on this).



1. a strong and fast-moving stream of water or other liquid. "rain poured down in torrents"

2. a sudden, violent, and copious outpouring of (something, typically words or feelings). "she was subjected to a torrent of abuse"

Edit: Sorry, misread the post.

I think that is what's great about how 37signals is handling this. I am sure a large portion of their client base is not technically inclined and having DDoS explained in plain english like they did gives those customers an understanding of what is happening.

That's what i was trying to say, we did something very similar, but you still end up with a portion of your user base that will blame you for not being able to handle it. (and they are in part right, but due to the nature of the attack sometimes it's very complicated to handle it or costs a lot of money - something the same customers wouldn't want to pay extra for :)

"Worst thing is that 90% of customers have no clue what a DDoS is and how hard it is to handle."

Otoh that's where the opportunity is. The fact that "customers have no clue". People pay you for something that they can't do themselves or that you make easier for them to do.

There's an oportunity if you're Cloudfare or similar service, not a time tracking and pm app. Most users will end up blaming you because you're not prepared enough etc..

Or avoiding your service altogether when they realize they can't depend on the cloud being always accessible.

Pardon the off-topic reply, but I'd like to connect with you. In the breadbox article the other day, you mentioned there's an opportunity to compete with GrubHub on price. Check out forkable.com. You can reach me at joe at forkable dot com.

Is it just me or are these attacks becomming more and more common? I hope we can get some more details on the attack like the origination of it, type used, and what steps were take to mitigate it. I always use information like this as a learning opportunity :)

When even the governments use DDoS [1] as a method to 'turn-off' services they don't like, it will be a very long path to fight.

[1] - https://www.quakenet.org/articles/102-press-release-irc-netw...

Github provided some information on the attacks being performed.


Has anyone defended a DDoS attack on an application hosted on Amazon's AWS/EC2?

If so, how did that go?

Did Amazon help?

I was involved with a company that received several attacks on AWS. We were premium support customers, and were able to work with our AWS TAM to get a mitigation device in place and turned on. It was a bit shaky at that time, as this was not a common service offering. Things may be better now.

What law enforcement do you call in these situations. I imagine it would be a waste to call local police.

I don't know how you would get feds to pay attention?

I believe that the Federal Bureau of Intelligence investigates and prosecutes cyber crimes[0].

[0]: http://www.fbi.gov/sanfrancisco/press-releases/2011/charges-...

Investigation, not intelligence. CIA does intelligence.

Or you could've just typed FBI, like a normal person.

Assuming the ransom request wasn't fake. It's pretty likely that the attack came from outside the US. Law enforcement will probably not be able to help at all.

Why not? The US. Law enforcement obviously doesn't have jurisdiction, but as long as a DOS is illegal in the country that the attacker sits in, the US Law enforcement should investigate and hand off to a partner agency in that country, acting as liaison and serving a request for extradition.

It's a different matter if the attacker is based in a country where DOS are legal or that doesn't have any extradition treaty with the US, but that still needs to be established.

Kim Dotcom would like a word.

Would CloudFlare help here?

Probably. Mitigating a DDOS (from my understanding) has two important things that need to happen. (1) You need a larger incoming pipe than the data being sent to you. (2) you need to ignore invalid requests so you don't flood your outgoing pipe as well.

Properly ignoring invalid requests can be a challenge, the process of doing so will depend on the type of attack being used. SYN floods can difficult since the src IP is most likely invalid. The attacks we've seen with DNS and NTP amplification are difficult as the attack isn't trying to get your servers to respond, they are just flooding your incoming pipe with data. If they are trying to abuse some page within your application you can more easily mitigate it as you'll know the source IP of the request so it can be blacklisted.

EDIT: a few more details:

SYN flood: http://en.wikipedia.org/wiki/SYN_flood

DNS Amplification: http://blog.cloudflare.com/deep-inside-a-dns-amplification-d...

As for mitigation, while we hear about Cloudflare a lot, AT&T and other large providers can provide DDOS protection for leased lines[0]. Basically what happens, before the data gets to your leased lines, traffic headed to you will go through AT&T's DDOS detection/prevention systems that attempts to filter bad traffic. This type of service would apply more to companies like Linode or possibly the datacenter that they are housed in.

[0] http://www.business.att.com/enterprise/Service/network-secur...

It depends if this attack is on basecamp.com or the IPs that basecamp.com resolves to.

It appears Basecamp only has a /23, so even if they redirected traffic through Cloudflare, the attacker could still find their direct servers fairly easily and attack that IP. It's still possible to block, but not quite as easy as setting up Cloudflare.

> so even if they redirected traffic through Cloudflare, the attacker could still find their direct servers fairly easily and attack that IP.

Why would it be easier for the attacker to find their direct servers if they only have a /23 - doesn't Cloudflare obscure the identity/location/IP of the server on the other side?

It's only 512 addresses, so the attacker can just switch between different IPs until service degrades and keep on that address. Also, it's likely their rack/cage has a limited amount of bandwidth compared to the whole datacenter, so they can just send traffic to that range and overload the switch.

...the attacker could still find their direct servers fairly easily and attack that IP.

Can the upstream to the actual server restrict traffic to known Cloudflare blocks?

We've had issues with saturated upstreams and then been negotiating new ISP connections. All the ISPs I've asked (Level3, NLayer, Cogent) won't put an active restriction to only CDN blocks upstream.

The ISPs will help during a DDOS but response times are slow and we haven't tried getting them to put this type of block in place yet.

After taking a look at CloudFlare's knowledge base, it seems that their services would definitely help if you were under attack. According to CloudFlare, they offer basic DDoS Protection with their plans, and it seems like you can upgrade to a business account during attacks for improved protection/mitigation. They also claim that they don't have a cap on the size of attacks they can handle.

Relevant links: https://support.cloudflare.com/hc/en-us/articles/200172676-C... https://support.cloudflare.com/hc/en-us/articles/200170216-H... https://support.cloudflare.com/hc/en-us/articles/200170196-I...

"CloudFlare help"

I'll leave it to others to answer this (for this situation) but keep in mind also that adding cloudflare also adds an additional layer that can fail for different reasons.

That tradeoff may well be worth it for certain high visibility web properties but maybe not if you are a low value target.

There are pros and cons to any decision you make that depend on specific circumstances.

I've been wondering myself, if CloudFlare helps against DDoS attacks when the page is dynamically generated for each user. For static pages it should help.

If the attack is working by essentially flooding Basecamp's network links until they reach capacity, then yes, it could. CloudFlare could simply filter out malicious traffic and only pass on legit requests to Basecamp.

That's obviously very much dependent on the kind of attack and whether CloudFlare has more network capacity than Basecamp (which I would imagine is highly likely).

CloudFlare does more than just caching. Even on non-cached pages it can filter and otherwise mitigate traffic that it has identified as malicious.

Correct, many times I've had to fill out a captcha to load a CloudFlare-protected page.

The sensitivity/frequency by which the captcha challenge page gets displayed is dependent on the security settings selected by the website owner.

Depends on the scale/power of attack. The latest hits (happening in the last few months) have been very large and I doubt CloudFlare would be able to successfully defend any of those while maintaining all of the current clients online. I have a client that occasionally gets this kind of blackmailing followed by attacks and they told me they use a US based company specialized in DDoS defending - until now the defense was pretty efficient. I've never bothered to ask for a name, but I guess it's a known one in the "network industry".

You're mistaken. CloudFlare has mitigated many the largest DDoS attacks in history, including some that have exceeded 400Gbps. These recent extortion-based attacks are large, but they are typically 1/10th the volume of the largest attacks we see. For instance, Meetup has publicly stated that they used our network to stop a similar attack. Many of the other recent victims have used CloudFlare as well.

Because of the unique design of our network, I'm unaware of any other service that has as much capacity that can be utilized in aggregate to mitigate large-scale attacks.

Matthew Prince Co-founder & CEO, CloudFlare

Does anybody know how many companies, upon receiving a blackmail "give us $300 or you'll be DDoSed" email, pay it? For every meetup.com or Basecamp that resist, how many actually give in to the blackmailer's demands?

It isn't $300, it's "up to $50,000"[0]

I've seen articles before saying online gambling websites often do pay up as the downtime isn't just lost revenue but customers going elsewhere.

[0] http://www.prweb.com/releases/2012/4/prweb9455636.htm

I'm wondering what happens to botneted subscribers from which the attacks originate. Is any attempt made to locate them and contact their ISPs? I think there should be, and subscribers found to be participating in the attack (presumably unknowingly) should be disconnected immediately. After all it's the subscribers' responsibility to keep their computers botnet free. Launching a DOS attack, even unknowingly, is probably violating the contract they signed with their ISP.

Crime, crime, crime, criminal. While technically (and probably also morally) true, was I the only one to find the emphasize weird?

I thought it was weird until he mentioned the blackmail. DDoS-ing for the lulz is one thing, doing it and then blackmailing the victim to get it to stop is a whole other level.

Whoever is doing this thank you for reminding me how important Basecamp is to my business. I hope they hunt you down.

A speculative thought:

Apart from being distributed, the insidious power of DDoS appears to lie in "subscriber-calling-server". Why not go the other way around? At least only for specific subscription services, not general purpose web access.

The situation of a DDoS attack is first communicated by the web service provider texting a subscriber, who texts back their present IP address. The web service provider then "calls" the subscriber from a hitherto unknown IP address. Of course, that address could be leaked too, but at least it's not obvious public knowledge like a DNS entry.

Sounds like circuit switched telephony/modems rather than packet switching, but can it be implemented in software?

A great deal of consumers are behind NAT, and punching through that is a huge pain. UPnP is sketchy, STUN is difficult, and custom schemes like uTP are undocumented. You'll get the occasional consumer who is willing to forward a port just to connect to your service, but not very often.

How do larger companies (like Basecamp) prepare for these kinds of risks? Do they contract with DDoS mitigation firms beforehand, or do most tend to hire help only when they are actually attacked?

DDOS firms (prolexic etc) are really expensive, I would imagine they do it on an as-needed basis. From my experience working at a datacenter, the first line of defense are the techs in the datacenter, for most attacks, they can blackhole offending IPs etc, and mitigate it. When it gets to the point of being something huge though, like the meetup.com attack, I would imagine they call in an outside firm.

Surprisingly, they usually don't.

Something along the lines of CloudFlare could be an option here. However, if the attacker does indeed know the actual IP of the Bootcamp servers (and Bootcamp allows traffic from IPs other than CF) that point is moot.

Set up CF, only allow traffic from CF.

On another note, having CF monitor an attack like this could help them do more research into mitigating these attacks in general and allow them to try and hunt the attacker. They tend to make things like this public which would benefit everyone.

I personally wouldn't do any business with cloudflare, while they're still hosting the various booter sites where you can pay to run these attacks.

If you're going to make accusations like that, you should really back it up with extensive proof.


> As I noted in a talk I gave last summer with Lance James at the Black Hat security conference in Las Vegas, a funny thing happens when you decide to operate a DDoS-for-hire Web service: Your service becomes the target of attacks from competing DDoS-for-hire services. Hence, a majority of these services have chosen to avail themselves of Cloudflare’s free content distribution service, which generally does a pretty good job of negating this occupational hazard for the proprietors of DDoS services.

http://www.webhostingtalk.com/showthread.php?t=1235995 http://www.webhostingtalk.com/showthread.php?t=1285880 http://www.webhostingtalk.com/showthread.php?t=1182576

I could post more, but why bother?

CloudFlare's CEO, Matthew Prince, has made his stance on this matter very clear:

CloudFlare is firm in our belief that our role is not that of Internet censor. There are tens of thousands of websites currently using CloudFlare's network. Some of them contain information I find troubling. Such is the nature of a free and open network and, as an organization that aims to make the whole Internet faster and safer, such inherently will be our ongoing struggle. While we will respect the laws of the jurisdictions in which we operate, we do not believe it is our decision to determine what content may and may not be published. That is a slippery slope down which we will not tread.

Source: http://blog.cloudflare.com/thoughts-on-abuse

As a result, both the Israeli Defence Forces and Hamas are CloudFlare customers. Unless one of their customers is doing something that is unambiguously illegal (e.g. hosting child pornography), CloudFlare won't cut them off just because they're doing something that some people regard as "bad".

It's a very principled stance and one that I respect.

Well, do you believe that suppressing someone else's right to free speech is still free speech?

Information isn't really the question here. These aren't sites telling people how to conduct DDOS attacks, these are sites where you pay them, and they run a DDOS for you. This effectively silences someone until they either give up on their message, or sign up for expensive DDOS mitigation packages (or Cloudflare).

You may consider that to be free speech. I don't.

CloudFlare aren't the Free Speech Police. It's clearly not their job to guarantee everyone's right to free speech. However, it would appear that they have decided that they will not deny their customers their right to free speech unless they're breaking the law. I respect that approach.

You clearly don't and you're entitled to your opinion.

I could post more, but why bother?

The krebs story was interesting thanks, the forum posts less so. I understand why cloudflare are reluctant to start rejecting customers based on content, but surely it's illegal to sell DDOS services? Perhaps they should change their TOS to exclude any sites which sell attack tools/services, because it looks really bad for them to be protecting sites that promote DDOS, which then provides them with repeat business.

Are there still sites up protected by cloudflare which promote this sort of activity?

Sell service of running a DDoS for you? Probably illegal.

Selling attack tools, however, is explicitly legal in most places, it's just software just as a port-scanning tool, DeCSS or zero-day vulnerability data.

"Promoting this sort of activity" again is free speech issue, no matter what "that sort" is. For example, there are posts right here in HN that "promote this sort of activity", and it would be ridiculous if having such content is even close to allowing someone to take down a server.

In short, unless the actual site is performing illegal activities (implementing the DDoS or uploading childporn&stuff), I'd say that they're correct in explicitly ignoring whatever else the site is doing.

Sorry, promote was a poor choice of words, I meant offer illegal services, not just talking about it or promoting it. I believe DDOS is illegal in many jurisdictions, and offering it for money more so. The allegation in the krebs article is:

a great many of today’s DDoS attacks are being launched or coordinated by the same individuals who are running DDoS-for-hire services (a.k.a “booters”) which are hiding behind Cloudflare’s own free cloud protection services.

I don't see Matthew Prince's post quoted above as a satisfactory response to this. This is morally and legally shady because cloudflare directly profit from the continued existence of DDOS, so they should be very careful to offer not a shred of evidence that they currently support people who carry out DDOS IMO, it would just be good business and current customers are going to get restless if they find cloudflare protects DDOS sites knowingly.

They've obviously taken a different stance (based on not wanting to filter customers on content), which I'm sympathetic to, but if the content is illegal and directly benefits them by facilitating more DDOS attacks, that equation changes.

Yea. They don't really bother to take them down. Their logic is that the attack traffic isn't technically leaving via their network, so it's not their problem. Take a look at whois for the domains in that last forum link. Two of those domains are still pointed at cloudflare nameservers.

I'm sure there's tons more, but why bother compiling a list when nothing will change. If you're curious, a good place to look would be the hackforums 'DDOS as a service' section. I bet a lot of the active ones would go to cloudflare.

very interesting point. I agree that maybe an updated CF TOS could help quell this issue, but then again I don't think CF is going to dedicate too much time to vet out any potentially bad sites.

CloudFlare is hosting booter sites?

CloudFlare does not host any website or it's content actually. They are not a web hosting service.

I wonder if there will be a day where on-premise solutions will be touted as the solution to the DDoS vulnerability of cloud-based solutions, in much the same way that there seems to be an ebb and flow between fat and thin clients over the course of computing history.

Because on-premise solutions are even more vulnerable to DDoS. A large data centre will have large amounts of connectivity, giving you a lot of head room for most types of attacks. But in this case 20Gbps of extra traffic was too much too. What on-premise solution can handle 20Gbps of extra traffic?

And I don't think Basecamp is technically "cloud", but collocated. They appear to own most or all of their servers.

If you define on-premise as being accessed over a private network (which seems to be the idea here), then it is not directly vulnerable to DDoS at all, because it isn't reachable from the public internet.

Is there something like cloudfare but more aggressive?

Like something that tries to find exploits on the machines used in the attack and try to shut them down, close their internet connection or inject a self-targeting DNS or something of the sort?

IANAL, but I've seen this discussion come up multiple times, and the problem is that the counterattack would technically be illegal. The fact that somebody else has already broken the law in order to compromise an innocent bystander does not give anybody else the right to do the same thing. Vigilantism is as illegal on the internet as it is in the real world.

This is a huge constraint for the people (e.g. at Microsoft) who work to identify and take down botnets: they expose themselves to significant legal/PR risk if they do anything harmful to the bots.

But this could be considered self-defense which is granted by most law systems.

Again, IANAL, but my understanding is that the concept of self-defense is specific to the use of force, rather than broadly applicable. You'll find it difficult to prove an immediate thread of physical harm from a DDoS.

And even if it were legal, you'd still have to deal with all of the "$SELF_DEFENDER broke my web site" PR unpleasantness from the innocent bystanders.

Self-defense is granted only for a direct, immediate physical threat - for example, if someone is blackmailing you, defrauding you or extorting "fire insurance for your warehouse" then self-defense doesn't allow you to do anything to them; if you smash the computer of a blackmailer, it's just as any other computer-smashing.

This is like someone hitting you with someone else's arm while they're sleeping (attackers use compromised hosts/networks) and then you go back and burn the sleepy guy.

That doesn't sound like self-defense at all :)

That's probably the worst analogy I have ever heard; or is this killing with someone else hand something common... somewhere?

Well, that's not common anywhere as far as I know, but you didn't say why is it a bad analogy.

I any case let me clarify what was my purpose as it seems I'm not good at analogies. The point is that you're attacked using compromised computers so it is incredibly stupid to retaliate to the source of the attack.

Hope that clarifies!

Is incredibly stupid to assume you are not liable for what you own; that's the reason why the cardholders gets in trouble by lending his credit card to friends or not reporting it has been stolen. The same thing with cars; if someone else drives you car you are in big part responsible for what the car is being used for (i.e. a friend and a bank robbery)

Hope that destroys your absurd misconception!

A ddos usually have many clients committing the attack and they are often unknowingly part part of the attack (E.g. regular users being compromised by a virus or similar). It would be impractical as well as unethical to counter attack them.

I'd imagine any business providing that is likely to get shut down pretty sharpish...

It could be done by software (not an IAAS); because is likely that the criminals are not going to sue.

> When these attacks happen, the rest of the internet will sometimes put you in quarentine to prevent the fire from spreading.

I'm interested about what he means by quarantine.

Does it mean that ISP's will stop accepting traffic going to their servers?

Every business experiences fires that they have to put out, and their transparency on what exactly the issue is keeps us informed and on their side.

We need the kind of concerted attention paid to this stuff that we gave to horse thieves in the Old West.

This is another great example of why I wish there was support for disabling commenting on gists.

Forget baecamp. Setup a webserver throw Colalbtive on it. Now you are in control of your data (you are now also responsible for the uptime).

Colabtive: http://collabtive.o-dyn.de/

they did get a blackmail email so it does seem like they are being targeted by someone.

is it the first time they are facing this sorta attack ?

Yet another reason we should be utilizing P2P WAY more often

A perfect time for those affected to test drive BaseCamp's competitor https://www.teamwork.com/


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact