
Pastie.org host pulls hosting after DDoS attack - yareally
http://pastie.org/
======
postfuturist
DDOS attacks are a fact of life, nice to know that Rails Machine will throw
you under the bus when one happens. Doesn't seem to fit their homepage
description: "You write Rails apps. We deploy, manage, support, monitor, and
scale them. Done."

~~~
jsprinkles
So if someone throws multiple tens of gigabits at your customer, and your
upstream threatens to turn off your entire hosting company, you would respond
"no way, we're going the extra mile for our customer"?

Rails Machine was, in all likelihood, compelled to act to either (a) preserve
its relationship with is upstream or (b) preserve its relationship with its
other paying customers that do not attract DoS attacks. Your idealism will
fall down quickly when one annoying customer threatens service for every last
one of your clients: it becomes a "do I continue to get paid or do I fight for
this customer" equation.

Price out mitigation equipment and the multiple high-level engineers you will
need to administer it before you respond telling me how wrong I am, by the
way. The gear alone is an engineer's salary just to get started.

Edit: Fine, I dropped the Amazon example.

~~~
postfuturist
Any given site might not experience a DDoS attack, but if you run a hosting
company, it will happen. The frequency depends on how many customers you have,
how popular they are, etc. If a DDoS attack catches you by surprise, then you
are very ill-prepared. One possible strategy is to throw the targeted customer
under the bus and call it a day. For hosts of a certain size, that may be
reasonable, as long as you clearly communicate that to customers.

I've seen these attacks mitigated single-handedly by an experienced fellow
with no access to fancy equipment. I believe it went something like this,
change the DNS to point at some EC2 instances to do front-end load balancing
with some scripts that detect and drop connections from attacking IP addresses
and do severe rate limiting. You can proxy legitimate requests back to the
original servers.

Granted, not everyone knows how to do that in a pinch, but it shouldn't be a
big deal for a hosting company. You only have to figure out how to do it once
and when you detect a DDoS attack, flip the switch for that customer.

~~~
jsprinkles
> I've seen these attacks mitigated single-handedly by an experienced fellow
> with no access to fancy equipment. I believe it went something like this,
> change the DNS to point at some EC2 instances to do front-end load balancing
> with some scripts that detect and drop connections from attacking IP
> addresses and do severe rate limiting. You can proxy legitimate requests
> back to the original servers.

That isn't how it works, but I appreciate you trying to explain it to me (I
have personal experience as a former employee of a hosting company, in this
exact position). A typical DoS attack is designed to flood the pipe, usually
with UDP. UDP spam attacks are _far_ simpler than layer 7 attacks that you
describe, and layer 7 attacks (if small enough) are easier to handle.

As a point of reference, I don't even have access to a botnet these days but I
could knock your home cable connection offline in about five minutes of my
time.

In your scenario, if you were to update DNS to point the victim at some EC2
instances, you'd accomplish DoS attacking EC2 instead as they only offer you
gigabit connectivity on most EC2 instances. If the DoS attack is multiple
gigabit, that isn't going to help you unless Amazon steps in and mitigates
about as best they can. A DoS mitigation strategy generally has nothing to do
with your application. If an engineer's first answer to a huge DoS attack --
as in, larger than the uplink -- is iptables or "scripts to detect and drop
connections", that engineer is uninformed. I'm sorry.

I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart
engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's
flowing across the wire, and a small TCP SYN flood directed at your server
won't gain any attention from your host or their upstream. A multiple-gigabit
attack that takes down the entire uplink will.

~~~
guelo
Would have appreciated if you did explain what an effective mitigation
strategy is.

~~~
jsprinkles
Depending on your position, there really isn't one. The hosting company can
implement one, which they might be able to insert in your path if you are at
the receiving end of an attack. There are Cisco products and a bunch of up-
and-comers that can do this.

If you're a "typical startup" with an Amazon footprint, you have no mitigation
strategy for flooding attacks aside from not attracting them. If someone
points multiple gigabit at you, there is just about nothing you can do except
hope Amazon can do something.

~~~
moe
That's a little pessimistic.

There's a range[2] of ISPs who will sell DDoS protection to you, either as an
addon when you host with them, or as an external service (re-routing your
traffic).

E.g. StormOnDemand just recently added it to their portfolio[1], which is
note-worthy because they actually list prices right on the website.

Either way, even without "explicit protection" any ISP beyond mom&pop-size
deals with these attacks every day and will sort them out for you for free the
first couple times. Only when they turn into a habit or become so huge that
they have to talk to _their_ upstream they will politely ask you to throw some
money their way.

Pulling the plug immediately is definitely not normal. However considering
Pastie was apparently a sponsored account it's at least somewhat
understandable (albeit a _terrible_ PR move).

[1] <https://www.stormondemand.com/ddos.html>

[2] When in doubt, and pockets deep enough, there's always Akamai. They're the
ones who can filter TBit/s-scale (yes, that was a T) attacks for you.

~~~
jsprinkles
> Pulling the plug immediately is definitely not normal.

Nulling immediately is. You're also assuming that this is Pastie's first DoS
attack, which we don't know based on the information presented to us.

~~~
moe
_Nulling immediately is._

For serious accounts (in the 6 digits/year) absolutely not, unless the attack
is large enough to affect other customers.

Admittedly RailsMachine looks very small, in all likelihood their pipe was
rather easily clogged and they simply didn't have the choices that larger ISPs
have.

~~~
jsprinkles
> For serious accounts (in the 6 digits/year) absolutely not, unless the
> attack is large enough to affect other customers.

If it doesn't affect other customers, a hosting company won't act or even be
aware, in most cases. They'll just send you a bill for the transfer. If
someone attacks you and it impacts other customers, you get nulled. I'm aware
of 7 digits/year and 8 digits/year accounts through industry anecdotes that
have had machines nulled. The engineer operating the null doesn't say, "oh,
that's X, maybe I shouldn't fix the network for my other customers".

I don't understand what you're disagreeing with.

~~~
moe
I'm disagreeing with your black/white record.

There's a bit of middle ground between "sending a bill" and nulling.

I've been hit by two larger attacks in the past (GBit/s range) and the
respective ISPs were both extremely supportive, switching our IPs while they
tightened their filters. Neither billed us a dime despite our ingress spike
making quite a bump in their charts and a lot of handholding over 2-3 days.

------
datums
This behavior of unplugging the destination of the DDoS is common with smaller
hosts. They don't have the capital to spend on expensive mitigation devices.
There are times when these attacks affect their entire network (bad design),
so their quick and fast solution is to null route you at their cores.

~~~
sha90
The issue isn't unplugging, I don't think anybody here thinks that it is
unreasonable if they were to do this. The issue is that they kept the site
unplugged for good because a single DDoS attack. My analysis based on the
official response is that they used this incident as an excuse to drop the
site because it was too much of a hassle to deal with the takedown notices.

------
pestaa
I can not understand these attacks. Why block a service that is free of
charge, useful and did no harm? Unless of course this DDoS was not targeted,
which makes even less sense to me.

Also, why did Rails Machine throw out the site so quickly? If I choose to
sponsor someone out of my free will, I'd do so without distinction from paying
customers.

~~~
seanp2k2
@everyone who doesn't work in hosting....

>"why did Rails Machine throw out the site so quickly?" If you run a
datacenter, you pay for an uplink. That uplink has limited capacity. 4gbit,
10gbit...whatever. A big attack can saturate that link completely, so even
with the biggest most expensive "mitigation device" on the market (some of
this gear can get into the hundreds-of-thousands-of-dollars for /one/ device,
mind you), if a DDoS is overloading your upstream bandwidth providers, you can
either have your entire DC brought to a crawl, or null route the site.

With that said, how did CloudFlare keep lulzsec up? Anycast, lots of iron,
lots of smart technicians, and probably tens to hundreds of thousands of
dollars in bandwidth fees. TL;DR it was a publicity stunt that they very
smartly played up.

DDoS is pretty misunderstood, and lots of clients think that there is some
magical box that can take all the traffic. Again, if your link is saturated, a
"mitigation" device can only filter the traffic; your upstream providers can
and will take you offline if you don't fix it. Failing that, you get a massive
overage bill and every other client at the facility is crawling. It's not
really a good solution (mitigation devices /can/ help with smaller attacks,
but for the big stuff, null routing is the best solution unless you have
something like CloudFlare -- and even they will pull the plug if the attack
gets too heavy, because it's simply not worth the expense to them to keep your
site online.)

~~~
saurik
As someone who does not work in hosting, I am somewhat surprised that
"upstream bandwidth providers" don't have mediation strategies for DDoS
attacks. It is my understanding that most of the asshats out there aren't
sitting on Stuxnet: they have well under a hundred machines that they are able
to flood traffic at you with (which is certainly more than enough). It seems
like there are network-level mechanisms you can use to just block such an
attack at the ingress connections. Is anyone willing to explain more about the
issues here? (I, at least, would find it fascinating.)

~~~
jsprinkles
Dropping UDP is a start, because most of these attacks randomize the UDP
source address. The real problem is AS operators who let forged UDP addresses
escape their network. If your outgoing edge ACL is not dropping source
addresses that do not belong to you, _you are doing it wrong_. Period. Full
stop.

If the packets are random UDP sources, then there's not really much you can do
on the receiving end except strategies that involve the destination address.
There's really only one effective one: nulling it.

~~~
saurik
Okay, but then why null route the entire machine, as opposed to dropping just
the UDP packets going towards it? If you already have routing infrastructure
that can disseminate "attempts to access this IP address will fail" it does
not seem a stretch to disseminate "attempts to access this IP address over UDP
will fail; over TCP there is no issue". Are "upstream bandwidth providers"
really that impotent against that kind of issue? :(

I, personally, have absolutely no need to have incoming UDP packets of any
kind at all entering my network, and can not come up with a reason why any web
hosting company would: it seems like it should almost be a question on your
bandwidth contract "will you need UDP (recommendation: no)". (Given the
simplicity of the UDP problem, I personally assumed that the complexity would
come from state-ful TCP filtering issues.)

~~~
prg318
The issue with dropping UDP is that DNS uses UDP in most implementations.
Unless you have no need for DNS on your network, you might want UDP packets to
be not dropped completely.

~~~
saurik
Ah, that a web host might want to host their own DNS in house (maybe for ease
or cost) is not something I considered (I outsource DNS as it is sufficiently
performance sensitive you really want to AnyCast it against numerous networks,
and there are people that specialize in that). As a client you can just use
TCP for DNS. (Again: I am not a host ;P. Thanks!)

------
wrecked
I'd like to apologize to those who have been negatively impacted by my
decision to pull support for Pastie (especially Josh). To understand why I
made the decision to pull our support after 9 hours of multiple DDOS attacks,
I'd like to share some background and our ops philosophy.

It is important to understand that I put our existing customers that pay us to
manage and scale their high growth revenue-generating web applications before
all else. This is the core of our business and what they trust us to do. As we
are seeing now, this means that I will protect them at the expense of making
some non-customers and "risky" customers upset. Let me explain further...

Rails Machine at this time is 6 people. Through a lot of tools like Moonshine,
experience, and process we manage 100+ web applications. Please note that I
did not say "host". Hosting is only part of the package. We commit to do
whatever it takes to keep our customers' applications available and growing
their business.

Everyone in the organization is a developer on a varying scale of dev to ops
including myself who started with Rails in 2005 and have been a professional
dev for 20 years.

Although not as quiet as we would like, in general the workload of responding
to outages, bugs, scaling problems, and traffic bursts are managed by the
team. We've been doing this for 6 years focused specifically on Rails and have
seen most problems with Rails applications in production. This makes us fairly
efficient in identifying and resolving issues.

We've been hosting Pastie pretty much since the beginning and free of charge
for several years. In the past two years, Pastie began to attract a lot of
users intending to use it to do illegal things. This includes sharing stolen
credit cards, stolen passwords, phishing schemes, copyrighted content,
virus/trojan horses, hacker scripts, confidential corporate info, etc, etc.

Please know that the overwhelming majority of Pastie's user base are well
meaning folks who kindly follow Josh's basic rule of "using Pastie for good".
A tiny minority however attract a lot of attention through their public pastes
that ruin the experience for everyone else.

Aside from the obvious problems, the public existence of this stuff makes a
lot of people upset who in turn threaten us. This includes but not limited to
criminals, giant corporations, angry individuals, trolls, and more importantly
our data center/upstream provider. These upset people then send us nastygrams
requiring us to take action or else. "Or else" includes suing us, arresting
us, DDoSing us, and more importantly terminating our service.

To avoid bad things happening to us and all of our other customers, we have to
take action immediately. Every now then other customers get a spam notice or a
DMCA notice but in general it happens once and not a huge deal. Pastie on the
other hand generates 100s of abuse complaints. Abuse complaints that we can
not ignore and require us to investigate and follow up on. While we are doing
this, we are not helping the 100+ other customers that generate zero abuse
complaints and most likely never will.

Enter the DDos. Over the past 6 years, we've handled multiple DDoS attacks on
different applications. Given that 95% of our customers are running revenue-
generating business applications, we deal with a DDoS about once or maybe
twice per year. They are really annoying and consume a lot of _time_ and a lot
of concurrent team members. Although we wish they never happened, as some have
pointed out DDoS attacks are a fact of life. Many high profile sites with much
larger teams and budgets have struggled for multiple days fighting waves of
attacks. We accept this as part of our job.

The problem in this particular instance is that these DDoS attacks on Pastie
were a continuation of the stream of operational disruptions already being
generated by the site. After handling the first attack with four of us
covering all of the angles around 10pm and with the help of Internap's network
team, we halted the attack.

Within a few hours, it began again in the wee hours of the morning. At the
same time, alerts for another customer _who had entrusted us with their
business_ came in. So a decision was made to halt the second attack as quickly
as possible and focus on doing our job as we promised to the rest of our
customers. We could have chosen to ride out multiple other attacks and engage
in a lot of time consuming and expensive behavior to preserve a site that was
already a source of ops disruption. Making that choice would have been
inconsistent with our values and commitments.

This decision was purely ops motivated to protect our team members ability to
serve our core customers.

Some of you are upset about this decision and I am sorry for that. I know 100+
customers that would approve. I put my customers and team first.

Please feel free to reach out to me directly (email or twitter) if you would
like to discuss this further.

@bradleyktaylor, Founder, Rails Machine

~~~
jsprinkles
You made the absolute right decision, and anybody that is saying otherwise
hasn't been in your position. The good of the many over the good of the one;
that's hosting.

The operational awareness in this comment makes me wish everybody would print
it out as an example of a tough call to make in defense of your brand.

~~~
rickmb
Throwing the victim under the bus is standard procedure for third rate
hosters. Doing it without notice is even worse.

Only if the brand is "we're el cheapo hosting cowboys" it's the right
decision.

~~~
_ikke_
You ignore the fact the this customer wasn't paying for the hosting. He only
costs them money, a lot of time (through dealing with DMCAs, DDoSs and more).

At this point they had to make a choice and they chose for their paying
customers and drop pastie.org

------
lotides
I know I'll get a lot of shit for this but good riddance. I hope it never
comes back. I've felt this way since I found my stolen accounts posted on
Pastie and they ignored requests to take it down.

~~~
jmah
Once they're on pastie, it's already too late.

------
veeti
Sounds like an opportunity for the other managed Rails hosts.

~~~
callmeed
Right, especially considering EngineYard and Blue Box are here at RailsConf

------
heliostatic
Perhaps an opportunity for Cloudflare to offer support?

~~~
dknecht
We would love to help, and have offered.

~~~
heliostatic
Awesome; I hope Josh is able to get things back online. I'm curious how much
support or assistance Rails Machine offered prior to pulling hosting.

~~~
seanp2k2
>"I'm curious how much support or assistance Rails Machine offered prior to
pulling hosting."

Me too, just out of curiosity, but if you have an attack that is large enough
to be affecting other clients, you can't usually wait around too long for the
targeted client to respond (and even if they did, there isn't typically much
you can do for a large attack.) How long? I'd say about 5 minutes. The typical
attack we see rises up over the course of a few minutes and lasts 6-48 hours.

------
mattmanser
I don't know about the scale of the DDoS attacks but have you tried
cloudflare?

------
nicksergeant
FWIW, apparently Rails Machine's CEO thinks some of Pastie's users are not-so-
great: <https://twitter.com/bradleyktaylor/status/194937146153508864>

~~~
avree
Uh, he's not talking about _all_ of Pastie's users, he's stating that Pastie
has some bad users who use it maliciously and threaten the provider.

~~~
nicksergeant
Sorry, worded that wrong. Fixed.

------
jackolas
I really enjoyed this, the user experience is very nice and its a well
designed site.

I tend to use Gist now because its improved and I have a client for it, but
this is a shame to see.

------
tristanoneil
If you're looking for more of a minimal snippet experience there's always
<http://marked.cc>

~~~
tvh2k
Unless you're pasting code. Check out my trivial example:
<http://www.marked.cc/4f9766f05e69190001000017>

~~~
tristanoneil
It does support code snippets you just have to use fenced code blocks ala
Github style.

``` x = [] ```

------
bryanl
This is a good chance to move over to <http://gist.github.com> for all your
pasting needs.

~~~
petercooper
I use both Gist and pastie.org. While I prefer Gist and wish it would just add
support for disposable gists (without making me log out!), pastie.org is
awesome for the disposable stuff on IRC, etc.

------
knewter
Clearly the gist.github engineers DDoSed them - it's all so clear!

