
Basecamp was under network attack - ibsathish
https://gist.github.com/dhh/9741477
======
swanson
Some great language there: framing it as an attack by criminals (gains
sympathy from users), explains in plain-terms what a DDOS is (front door
analogy), emphasizes (twice!) that user data is safe, apologizes for the
likely downtime, informs people where to get updates.

Probably worth bookmarking this for when you [hopefully never] have to deal
with this same situation.

~~~
pdeuchler
I'm going to play devil's advocate and completely disagree with you here :)

Customers, especially non-technical ones, don't give a crap. What they want to
know is when the service will be back up, and what steps you're taking to
prevent it happening in the future, although I'm sure a certain percentage
would be interested in why this is happening in the first place (not as in the
technical breakdown, but why you didn't have a contingency plan).

If I'm a customer of Basecamp it looks to me like 37Signals is couching this
as if _they_ are the victims here, when really _I_ am the victim. They're
business isn't being disrupted... mine is! I pay them to abstract me away from
the gory details... if I wanted to deal with that stuff I'd pay people to
build it in house. My job as a customer isn't to sympathize with an outage,
it's to move to a service that won't have one.

After turning in a term paper a day late a wise professor once told me "It
doesn't matter if your excuse is true, it's still an excuse." The basic facts
are the job didn't get done, and the person to blame is the person who didn't
get the job done. Any modern web service that doesn't take the simple effort
to sign up for cloudflare or their ilk to reduce attack surface doesn't
deserve my money. (Admittedly a harsh perspective to take, but one many do
take)

~~~
davidw
Reasonable people realize that unforeseen things happen, and might empathize
with someone being targeted by a criminal enterprise a bit more than someone
who just forgot to pay the electricity bill.

There is an entire movement in Sicily dedicated to highlighting and
frequenting businesses that refuse to pay protection money, because in the
past, paying was the norm.

[http://www.addiopizzo.org/](http://www.addiopizzo.org/)

Since that's not the kind of society I want to live in, I'd rather stand firm
behind a company that refuses to deal with criminals. If companies give in as
a matter of convenience to retain customers who turn a blind eye, that will
only make the criminals stronger.

Now, certainly, there are measures they can take to mitigate the problem, but
with all the things to do in a business, I suppose it's the kind of thing that
might not be on the front burner until it happens. There are all kinds of bad,
destructive things that _could_ happen in the world, but if you spend all your
time worrying about what _could_ happen, you won't have a viable business.
It's a tricky balancing act, and I'm willing to cut some slack to someone
being targeted by criminals.

~~~
pdeuchler
I more or less agree with you, but that's kind of a false dichotomy, isn't it?
Signing up for cloudflare or using a CDN isn't giving in, it's taking measures
to protect yourself (and that's ignoring the other benefits you get). The
unfortunate fact is DDOS attacks are becoming a daily occurrence, and if you
have something to lose you should probably take measures to counteract any
possible threats.

If 37Signals was a bitcoin exchange, aka a known target of DDOS attacks, the
mood here would be drastically different... yet we've hit a tipping point
where it seems everyone is equally at risk. DDOS attacks have become a sad
cost of doing business on the internet, and just because you acknowledge that
fact and try to prevent yourself from being a target doesn't mean you're
capitulating to the criminal enterprise.

In fact, I don't see a better way of sticking it to the thugs than responding
with "Hahaha, do your worst. We'd love to see if the money we're paying X
COMPANY is worth it." And then you get to write a totally different blog post,
one where you get to brag about your excellent foresight and how you have
proven to your customers that the money they pay you buys a top-notch service.

~~~
troels
That's a bit naive though. People can always find ways to hurt you - it's a
very asymmetric fight. With a complex application such as Basecamp, you can't
really put everything behind a cdn.

~~~
cft
That's why I actually think that their thrust on pursuing the legal/FBI route
is a good one, especially if they achieve any success there. This
extortion/racket is indeed criminal and not tolerable. It would be good to
catch the racketeers and make an example of them.

------
TacticalCoder
I take it at one point people will start to believe that I work for OVH (I
really don't) but... OVH has a mandatory DDoS protection on all its dedicated
servers: fees have been slightly raised to take that mandatory protection into
account.

There are a few gotchas, including if I understand it correctly the need to
"retry twice" when you try to SSH in your server when a DDoS is going on
but...

OVH doesn't even feel a 85 Gbps attack (let alone a 20 Gbps one like in the
article). They can deal with attack much larger than that automatically.

They seem to have very good DDoS protection against the "flood" type of DDoS.
And this is pretty much transparent to users.

I hope more and more hosting company start implementing similar anti-DDoS
features: more competition would bring better protection against flood-type
DDoS and cheaper price.

Here's the explanation as to how their system works (in french but there are
several graphics):

[http://www.ovh.com/fr/a1164.protection-anti-ddos-service-
sta...](http://www.ovh.com/fr/a1164.protection-anti-ddos-service-standard)

Basically as soon as a DDoS trying to saturate your server(s) is detected the
attacker faces the problem of needing to DDoS... OVH itself.

And the DDoS doesn't even make it to your server while the legitimate trafic
still does.

I find it great that there are people actually looking for solutions to the
DDoS issue.

~~~
cordite
I have a service on OVH myself.

Though a friend at another related service had been kicked from two VPS
providers due to receiving a few DDoS attacks. These providers claimed it was
against their Terms of Service and ejected him as a customer. That day he
learned it is best to keep offsite-cross-company backups of everything, since
he did not get a single byte from his machines.

~~~
yogo
Claiming it was against the terms might be an easy out for them but is silly
since being a target is outside of your control, for the most part. Hosts will
usually null route customers without sympathy to protect other customers so
it's the price of doing business.

~~~
kalleboo
It makes a DDoS an even better extortion. "Pay up or we'll get you kicked from
your hosting provider."

~~~
alxndr
"...and potentially lose all of your data, if you haven't been planning ahead"

------
akassover
We got hit by a DDoS about a year ago. Rackspace (who normally has amazing
support) quietly null routed us and went about their day. No heads-up, trouble
ticket, or any other form of notification. They didn't even put a note in our
account so when we contacted their support to figure out why our servers were
unresponsive outside their network the poor guy who answered the phone was
just as confused as I was.

We've taken some steps since then to hopefully reduce our vulnerability. I'd
be really interested in a DDoS protection best practices guide for small SaaS
businesses.

~~~
gk1
I'm running a small SaaS business. I'm curious to hear what steps you took to
reduce your vulnerability. Could you please share so others can take the same
steps?

~~~
akassover
The biggest thing we did was remove our dependency to a single IP (this was a
unique requirement of our business). We also improved our firewall and upped
our managed service level. We're not 100% bullet proof now, but definitely
better than we were. I'd be happy to go into more detail offline.

~~~
gk1
Thanks! I'm on managed service as well, so I may be able to request some of
those things. I've never been hit but sounds like I should be proactive about
this.

------
filet
I've had really negative experience with these type of criminals.

I was hired as a CEO at an <unnamed> company ($200m+ revenue) and we were hit
by this type of attack.

Every second of being down cost us literally $10k, so we quickly negotiated
with criminals for $5k one time payment and they stopped the attack.

Unfortunataly a few weeks later we were hit by 3 new attacks. Apparently the
word had spread and these new attackers demanding $50k.

We were not going to pay $50k but I was also unable to stop the attacks. I was
let go a few days later as we had a down time of 2 days and I wasn't able to
fix this problem.

Crap.

~~~
PeterisP
That's a good reason why it's never a good idea to pay for DDOS threats - in
many other popular extortion scenarios such as kidnapping, blackmail w. secret
info or mafia 'protection money' for storefronts, the deal generally doesn't
allow other, new attackers to make the same demands, so you actually are
getting some protection - but here it does simply mark you as vulnerable.

~~~
sergiotapia
Same goes when bribing a cop here. If you bribe too much you're targeted as
easy money among the other cops here. Say for example you're caught driving
without your insurance, you bribe and then every other cop knows you don't
have insurance and squeeze you for money left and right.

Source: 3rd world south america

------
janlukacs
Although a smaller service, we were in a similar situation a couple of years
ago. We assumed it was a competitor because there were not monetary requests,
just a massive DDoS via torrents that lasted almost a week. Data center didn't
help us in any way... it was crazy. Worst thing is that 90% of customers have
no clue what a DDoS is and how hard it is to handle.

~~~
alandarev
How is torrents protocol used to DDoS you? I never came across torrents being
used as a DDoS. I would appreciate more details on what sort of torrent attack
it was, and whether you found any ways of partially neglecting damage.

~~~
Danieru
A malicious tracker, or a peer if using DHT, can claim an IP, the victim, is
active in the swarm and has valuable bits of the torrent. Then torrent clients
will try to connect to the victim.

The attack is pretty clever, being indirect it is hard to trace and because
bittorrent allows arbitrary ports you can hit a specific ip & port pair.

The one downside is the victims can be sure it is a bittorrent DDOS by
checking the attacking connection's requests. The attacker's packets will
contain bittorrent's magic connection bits.

~~~
jessaustin
_The attacker 's packets will contain bittorrent's magic connection bits._

ISTM that once you've determined bittorrent is the attack vector, the hard
part is done? Is dropping by "magic bits" harder than dropping by ip/port?

~~~
phil21
Yes. Very much harder. One can be done at line rate on any halfway decent
router, and the other requires deep packet inspection which is considerably
more expensive.

------
rdudek
Is it just me or are these attacks becomming more and more common? I hope we
can get some more details on the attack like the origination of it, type used,
and what steps were take to mitigate it. I always use information like this as
a learning opportunity :)

~~~
alandarev
When even the governments use DDoS [1] as a method to 'turn-off' services they
don't like, it will be a very long path to fight.

[1] - [https://www.quakenet.org/articles/102-press-release-irc-
netw...](https://www.quakenet.org/articles/102-press-release-irc-networks-
under-systematic-attack-from-governments)

------
joevandyk
Has anyone defended a DDoS attack on an application hosted on Amazon's
AWS/EC2?

If so, how did that go?

Did Amazon help?

~~~
mgorsuch
I was involved with a company that received several attacks on AWS. We were
premium support customers, and were able to work with our AWS TAM to get a
mitigation device in place and turned on. It was a bit shaky at that time, as
this was not a common service offering. Things may be better now.

------
wehadfun
What law enforcement do you call in these situations. I imagine it would be a
waste to call local police.

I don't know how you would get feds to pay attention?

~~~
codazoda
Assuming the ransom request wasn't fake. It's pretty likely that the attack
came from outside the US. Law enforcement will probably not be able to help at
all.

~~~
Xylakant
Why not? The US. Law enforcement obviously doesn't have jurisdiction, but as
long as a DOS is illegal in the country that the attacker sits in, the US Law
enforcement should investigate and hand off to a partner agency in that
country, acting as liaison and serving a request for extradition.

It's a different matter if the attacker is based in a country where DOS are
legal or that doesn't have any extradition treaty with the US, but that still
needs to be established.

------
vidar
Would CloudFlare help here?

~~~
timdorr
It depends if this attack is on basecamp.com or the IPs that basecamp.com
resolves to.

It appears Basecamp only has a /23, so even if they redirected traffic through
Cloudflare, the attacker could still find their direct servers fairly easily
and attack that IP. It's still possible to block, but not quite as easy as
setting up Cloudflare.

~~~
chimeracoder
> so even if they redirected traffic through Cloudflare, the attacker could
> still find their direct servers fairly easily and attack that IP.

Why would it be easier for the attacker to find their direct servers if they
only have a /23 - doesn't Cloudflare obscure the identity/location/IP of the
server on the other side?

~~~
timdorr
It's only 512 addresses, so the attacker can just switch between different IPs
until service degrades and keep on that address. Also, it's likely their
rack/cage has a limited amount of bandwidth compared to the whole datacenter,
so they can just send traffic to that range and overload the switch.

------
CanSpice
Does anybody know how many companies, upon receiving a blackmail "give us $300
or you'll be DDoSed" email, pay it? For every meetup.com or Basecamp that
resist, how many actually give in to the blackmailer's demands?

~~~
cmdkeen
It isn't $300, it's "up to $50,000"[0]

I've seen articles before saying online gambling websites often do pay up as
the downtime isn't just lost revenue but customers going elsewhere.

[0]
[http://www.prweb.com/releases/2012/4/prweb9455636.htm](http://www.prweb.com/releases/2012/4/prweb9455636.htm)

------
ambrop7
I'm wondering what happens to botneted subscribers from which the attacks
originate. Is any attempt made to locate them and contact their ISPs? I think
there should be, and subscribers found to be participating in the attack
(presumably unknowingly) should be disconnected immediately. After all it's
the subscribers' responsibility to keep their computers botnet free. Launching
a DOS attack, even unknowingly, is probably violating the contract they signed
with their ISP.

------
norswap
Crime, crime, crime, criminal. While technically (and probably also morally)
true, was I the only one to find the emphasize weird?

~~~
Aqua_Geek
I thought it was weird until he mentioned the blackmail. DDoS-ing for the lulz
is one thing, doing it and then blackmailing the victim to get it to stop is a
whole other level.

------
codelittle
Whoever is doing this thank you for reminding me how important Basecamp is to
my business. I hope they hunt you down.

------
quarterwave
A speculative thought:

Apart from being distributed, the insidious power of DDoS appears to lie in
"subscriber-calling-server". Why not go the other way around? At least only
for specific subscription services, not general purpose web access.

The situation of a DDoS attack is first communicated by the web service
provider texting a subscriber, who texts back their present IP address. The
web service provider then "calls" the subscriber from a hitherto unknown IP
address. Of course, that address could be leaked too, but at least it's not
obvious public knowledge like a DNS entry.

Sounds like circuit switched telephony/modems rather than packet switching,
but can it be implemented in software?

~~~
sirsar
A great deal of consumers are behind NAT, and punching through that is a huge
pain. UPnP is sketchy, STUN is difficult, and custom schemes like uTP are
undocumented. You'll get the occasional consumer who is willing to forward a
port just to connect to your service, but not very often.

------
robgering
How do larger companies (like Basecamp) prepare for these kinds of risks? Do
they contract with DDoS mitigation firms beforehand, or do most tend to hire
help only when they are actually attacked?

~~~
lawncheer
DDOS firms (prolexic etc) are really expensive, I would imagine they do it on
an as-needed basis. From my experience working at a datacenter, the first line
of defense are the techs in the datacenter, for _most_ attacks, they can
blackhole offending IPs etc, and mitigate it. When it gets to the point of
being something huge though, like the meetup.com attack, I would imagine they
call in an outside firm.

------
coreymgilmore
Something along the lines of CloudFlare could be an option here. However, if
the attacker does indeed know the actual IP of the Bootcamp servers (and
Bootcamp allows traffic from IPs other than CF) that point is moot.

Set up CF, only allow traffic from CF.

On another note, having CF monitor an attack like this could help them do more
research into mitigating these attacks in general and allow them to try and
hunt the attacker. They tend to make things like this public which would
benefit everyone.

~~~
devicenull
I personally wouldn't do any business with cloudflare, while they're still
hosting the various booter sites where you can pay to run these attacks.

~~~
bybjorn
CloudFlare is hosting booter sites?

~~~
xxdesmus
CloudFlare does not host any website or it's content actually. They are not a
web hosting service.

------
olsonea
I wonder if there will be a day where on-premise solutions will be touted as
the solution to the DDoS vulnerability of cloud-based solutions, in much the
same way that there seems to be an ebb and flow between fat and thin clients
over the course of computing history.

~~~
samplonius
Because on-premise solutions are even more vulnerable to DDoS. A large data
centre will have large amounts of connectivity, giving you a lot of head room
for most types of attacks. But in this case 20Gbps of extra traffic was too
much too. What on-premise solution can handle 20Gbps of extra traffic?

And I don't think Basecamp is technically "cloud", but collocated. They appear
to own most or all of their servers.

~~~
Nacraile
If you define on-premise as being accessed over a private network (which seems
to be the idea here), then it is not directly vulnerable to DDoS at all,
because it isn't reachable from the public internet.

------
ivanca
Is there something like cloudfare but more aggressive?

Like something that tries to find exploits on the machines used in the attack
and try to shut them down, close their internet connection or inject a self-
targeting DNS or something of the sort?

~~~
Nacraile
IANAL, but I've seen this discussion come up multiple times, and the problem
is that the counterattack would technically be illegal. The fact that somebody
else has already broken the law in order to compromise an innocent bystander
does not give anybody else the right to do the same thing. Vigilantism is as
illegal on the internet as it is in the real world.

This is a huge constraint for the people (e.g. at Microsoft) who work to
identify and take down botnets: they expose themselves to significant legal/PR
risk if they do anything harmful to the bots.

~~~
ivanca
But this could be considered self-defense which is granted by most law
systems.

~~~
mobiplayer
This is like someone hitting you with someone else's arm while they're
sleeping (attackers use compromised hosts/networks) and then you go back and
burn the sleepy guy.

That doesn't sound like self-defense at all :)

~~~
ivanca
That's probably the worst analogy I have ever heard; or is this killing with
someone else hand something common... somewhere?

~~~
mobiplayer
Well, that's not common anywhere as far as I know, but you didn't say why is
it a bad analogy.

I any case let me clarify what was my purpose as it seems I'm not good at
analogies. The point is that you're attacked using compromised computers so it
is incredibly stupid to retaliate to the source of the attack.

Hope that clarifies!

~~~
ivanca
Is incredibly stupid to assume you are not liable for what you own; that's the
reason why the cardholders gets in trouble by lending his credit card to
friends or not reporting it has been stolen. The same thing with cars; if
someone else drives you car you are in big part responsible for what the car
is being used for (i.e. a friend and a bank robbery)

Hope that destroys your absurd misconception!

------
griffinheart
> When these attacks happen, the rest of the internet will sometimes put you
> in quarentine to prevent the fire from spreading.

I'm interested about what he means by quarantine.

Does it mean that ISP's will stop accepting traffic going to their servers?

------
reshambabble
Every business experiences fires that they have to put out, and their
transparency on what exactly the issue is keeps us informed and on their side.

------
stcredzero
We need the kind of concerted attention paid to this stuff that we gave to
horse thieves in the Old West.

------
stock_toaster
This is another great example of why I wish there was support for disabling
commenting on gists.

------
drewblay
Forget baecamp. Setup a webserver throw Colalbtive on it. Now you are in
control of your data (you are now also responsible for the uptime).

Colabtive: [http://collabtive.o-dyn.de/](http://collabtive.o-dyn.de/)

------
barkingcat
they did get a blackmail email so it does seem like they are being targeted by
someone.

------
ing33k
is it the first time they are facing this sorta attack ?

------
Allower
Yet another reason we should be utilizing P2P WAY more often

------
rootuid
A perfect time for those affected to test drive BaseCamp's competitor
[https://www.teamwork.com/](https://www.teamwork.com/)

~~~
xxdesmus
classy.

