
Linode DDoS continues – Atlanta down for 16+ hours - gingerlime
http://status.linode.com/incidents/cbbcjnhhpkgm
======
mjrpes
Je suis Linode!

This is nothing less than cyber-terrorism, and I hope the FBI is involved.

A problem I see with people throwing Linode under the bus is that only the
mega hosting providers (AWS/Google) have the resources to mitigate such
attacks. I wound hope that the industry can find a solution out there that
allows smaller players (Linode, Digital Ocean, etc) to run a hosting business
without the threat of DDoS.

If everyone moves to the big players, it is a loss for us all: feature-wise,
quality-wise, security-wise, cost-wise.

~~~
larrymcp
"Je suis Linode" what an awesome sentiment...

I just signed up for a Linode server. Going to use it for part of our offsite
backup storage and to monitor some nodes on our primary network.

Huzzah...

------
dantiberian
I love Linode but they've done a poor job communicating with their customers.
Compare the Linode Twitter feed, their mentions, and the lack of replies to
how Slack handled it: [https://medium.com/swlh/slackdown-a-lesson-in-brand-
interact...](https://medium.com/swlh/slackdown-a-lesson-in-brand-
interaction-17f8c82b9fce#.4vsz21c0u).

The Main Twitter account still hasn't announced the Atlanta DoS.
[https://twitter.com/linode?lang=en](https://twitter.com/linode?lang=en)

I understand the severity of the issue is different to Slacks', but there
should be a bunch of people on the Twitter account replying to people and
saving customers. A lot of their customers are angry about lack of
communication as much as they are about the downtime.

~~~
empressplay
Although I'm not sure how important it is to announce outages on social media,
I'm a customer and I haven't seen an e-mail either warning of an outage or
explaining the situation. Generally I think linode is great but taking the
tack of "Oh, if they don't notice, we shouldn't point it out to them" is just
a bit cowardly.

~~~
dan1234
They have a status page[0] that you can subscribe to for email/rss/atom
notifications. You could use an IFTTT recipe to send the RSS feed to
twitter[1] or even a Slack channel[2], if you wanted.

Personally, I use Uptimerobot[3] to monitor my nodes and post any issues into
a Slack channel. This lets me know if it's a problem with my VPS or Linode (or
any of the other providers I use).

[0][http://status.linode.com/](http://status.linode.com/)

[1][https://ifttt.com/connect/feed/twitter](https://ifttt.com/connect/feed/twitter)

[2][https://ifttt.com/connect/feed/slack](https://ifttt.com/connect/feed/slack)

[3][https://uptimerobot.com](https://uptimerobot.com)

~~~
jonknee
Yes, or a company that isn't providing the service you're paying for could be
honest about it and let you know. Especially when it happens to fall on a
major holiday. I am moving my account ASAP and probably wouldn't if I had
heard from them before I found out myself.

------
gingerlime
I honestly love Linode and am sure they'll come out better as a result of
this. But our customers aren't as understanding. Currently we're playing a bit
of cat and mouse and with each data centre going down - we're switching our
recovery process into gear and restoring to a different VPS (outside Linode).
We have linodes on pretty much all locations, but if this continues at this
rate, we simply won't have any linodes left there.

It would be very hard to justify going back to Linode afterwards, even with
the best intentions to do so. _"... So you seriously want us to go back to
this hosting provider that caused us all this mess over Christmas / New
Year's??"_

~~~
jafingi
That's exactly what the reaction the attackers want.

However, what are the alternatives? Linode have been dead stable for me the
past many years, and delivers what they promise in a transparent way. No
overselling of servers. No sudden extra bills.

Linode will come out stronger after this, so it won't be able to happen on
this scale again.

The big question is: Who, with a lot of money, would want to hurt Linode's
business in this way? This isn't just a "script kiddie" having fun. It's a
very well planned and powerful attack requiring buying large botnet capacity
for an extensive amount of time.

~~~
agildehaus
Genuine question: What does anyone have to gain from Linode's misfortune? What
sort of group would do this?

~~~
mwill
A competitor seems like a pretty obvious answer.

But the vps market is fairly crowded, I can't imagine what attacking a single
competitor would actually accomplish.

~~~
rdtsc
> A competitor seems like a pretty obvious answer.

I don't know... unless Linode has a large presense in a country where this
level of play is expected. I have hard time imagining a large US cloud /
hosting company comparible in size or customer base with Linode trying it.
There is too high a risk, someone will talkt and reveal it.

------
cpqq
Work in IT. Server was stable & online for 189 days before the 25th, knew them
for stability.

No notification from them, just a handful of downtime alerts during time with
the family. They were completely gone from BGP tables in Newark.

Used backups and moved sites to OVH. Don't know who they pissed off, I suspect
another NJ competitor, who is known for taking cheap shots at other VPS
companies.

It's a pain in the ass, but at the same time, how is their network so fragile?
You would think at least some of the fragile systems being attacked would be
firewalled or at least ACL'd off from the public net.

This is what happens when you don't run your own network and rely on other
ASN's and uplinks to do the work for you. When it comes to other customers
being affected, they will simply null you. Unlike your network ops who would
be trying anything they could from OOB to rectify such.

~~~
StanAngeloff
What you are saying has been somewhat confirmed in Linode's latest update on
the Atlanta outages [1]. I can't help it but to wonder if Linode were prepared
or had a plan in place in case of a DDoS? It appears their upstream provider
cuts them off completely once an attack starts/resumes and gradually puts them
back on. The cycle then repeats.

We are also duplicating in OVH, read good things about their built-in DDoS
protection on HN.

[1]
[http://status.linode.com/incidents/cbbcjnhhpkgm](http://status.linode.com/incidents/cbbcjnhhpkgm)

~~~
gingerlime
What kind of VPSs are you guys using with OVH? I had a look at OVH, but to be
honest got really confused with too many options to choose from... (not to
mention I wasn't sure which site I should sign up to, the .com / .co.uk - is
this based on the VPS location in any way?).

Linode clearly wins on simplicity and clarity. I guess under the current
circumstances, I'd be willing to compromise simplicity for better availability
though.

~~~
StanAngeloff
(I work in IT as well. Not affiliated with Linode / OVH)

Linode wins on simplicity, agreed. We are in the same boat, OVH has too many
offerings. We are looking at their VPS SSD plans [1]. Last thing we want is to
be offline again. As such we are also looking for anti-DDoS which is included
in the plans. I intent on spinning up a few nodes to try them out first.

[1] [https://www.ovh.ie/vps/vps-ssd.xml](https://www.ovh.ie/vps/vps-ssd.xml)

~~~
cpqq
Whoever has the most transit wins the mitigation game. You need to take in
that traffic, then process it with a shitload of power.

OVH has 3 large datacenter PoPs to absorb attacks and do just that, then push
the traffic clean back to your server.

They may blow at support and response times, but once I have a dedicated
server from them, their Manager is intuitive enough to get going.

Add the fact I can get 64G server on a brand new E5 chassis with 255 free IPs
for VPS of my own, and I've been moving more and more sites there as hosts get
arbitrarily hit.

Piss off some competitor or skiddie and you get tested. It's ridiculous, but
sadly DDoS mitigation is becoming a must.

Good time to leave being a SysAdmin in cloud and go back to web design full
time as I watch a lack of best practices and SPOF take over.

Finally, I backup everything to 2 off-site locations and hope for the best.

------
gwright
Every business endeavor has associated risks, which can be mitigated in a
variety of ways for a variety of costs.

Offloading the responsibility for continuation of your business to Linode (or
any other data center provider) is unfair. A history of uptime, verbal
promises, or fancy SLA terms should never be interpreted to mean that
disasters won't happen. A ten day long DDOS is a disaster and in this case a
man-made disaster.

Using Linode (or another provider) instead of building your own data center is
more cost effective, but it means you are no longer in direct control of your
infrastructure (decreased costs, increased risk).

Designing your application to span multiple availability zones (data centers)
can mitigate single points of failure within a single vendor but is more
expensive than operating in a single zone.

Designing your application to span multiple vendors can mitigate single vendor
failures (or changes in offerings from a single vendor) but is even more
expensive.

And still there are ways to mitigate these costs, business interruption
insurance can help cover the costs for moving to a new data center or vendor
in case of a disaster (such as hiring staff, overtime, etc.). Lost profits can
be covered by business interruption insurance also.

Of course it is expensive to operate any business in a hostile environment. A
seaside restaurant better be prepared to weather a hurricane. I wonder how
much money has been spent on security cameras, guards, metal detectors, and so
on since 9/11? The increasing occurrence of targeted DDOS (and other types) of
attacks is the physical equivalent of an increasingly hostile environment and
is going to be associated with higher costs.

In the longer term, I think we need to find ways to get law enforcement better
suited to deal with these problems but ultimately I think we'll need to
radically change the way we handle network operations and the technical
foundations of the network such as content centric networking
([https://en.wikipedia.org/wiki/Content_centric_networking](https://en.wikipedia.org/wiki/Content_centric_networking))

------
empressplay
Let's hope the authorities can identify the bad actor(s) in this case -- if
they haven't made extortion demands it's hard not to imagine they're a
competitor, and it would be really frustrating if they were able to get away
with it.

~~~
zhte415
If a competitor, their funders would likely be non-amused at their actions if
publicised (even if already aware).

------
houseofmore
Linode has been great to us, but we can't risk further outages. We've switched
over 20 nodes to Google Cloud for the time being -- thankfully before that 16+
hour outage today in Atlanta. Happy check them out again once the dust
settles.

~~~
click170
From an armchair, it seems like it's a good idea to distribute a virtual
server farm on multiple providers (Linode, AWS, et al). There's even libraries
available to abstract away the provider layer, like libcloud. However, IME
it's typical to invest in just one provider.

Is anyone currently using libcloud or equiv. and able to share details?

------
dantiberian
The brutal thing about a DDoS on a web hosting company is that it affects
their business in very long lasting ways. If an ecommerce site is down they
may lose sales for that day, and a small amount of customers. If a hosting
provider is down then they can lose many of their customers for life.

~~~
jacquesm
If you look at how interconnected stuff is these days it is really not that
far away (if we haven't crossed that point already) were lives will be lost
due to crimes like these.

The analogy for me is one of roads. If you block a road on purpose then an
ambulance might not be able to reach an accident victim in time. The internet
is infrastructure, just like roads and purposefully obstructing it wholesale
is doing damage to a large number of parties.

What is sad is that these people get away with this stuff over and over again,
it is very rare for DDoS organizers to be caught, rarer still (if it even ever
happened) for them to be sentenced.

~~~
jops
Agreed. Attacks like this should be turned into a national issue and not left
to Linode to deal with alone. I hope the NSA or FBI or whoever are
investigating.

~~~
jacquesm
Likely you'll find they are an international issue.

~~~
SQLite
Isn't that why we have aircraft carriers?

~~~
jacquesm
Who is we?

------
meshko
I am a (tiny) Linode customer with just one node but for a good number of
years. Probably close to 10 now. This is the first outage I have with them.
All in all it's a good thing as it made me finally learn how to use EC2 and I
now have a backup there. I already shut down the EC2 backup instance and
switched back to Linode as they seem to be up now.

------
lambdud
I've used Linode increasingly since 2008, now consuming 10 times what I did at
the start. I'm preparing to move to AWS today but genuinely hope that Linode
comes through with a reasonable explanation of why I can expect this to not be
repeated.

------
leeforkenbrock
Well, in good news, it seems Linode's billing system was unharmed during this
outage. ;)

------
rrcap
In this moment I can access to my linode and my website is operational. News
about status of attack?

------
ugexe
their status page is update after update of the same message, apparently being
ad libbed by someone with a thesaurus

------
frik
This story fall from #2 on HN to page #102 in seconds.

Is it because of only 38 upvotes vs 43 comments? The story is just 3 hours
old.

@HN / dang: What's going on with HN sorting algorithm? The #3 story on HN is 1
hour old and has just 9 upvotes and 2 comments: "Churchill and His Money, or
Lack of It":
[https://news.ycombinator.com/item?id=10825575](https://news.ycombinator.com/item?id=10825575)

Screenshot:
[http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png](http://s3.postimg.org/6dn7h6w5v/hn_linode_fall.png)

~~~
cpqq
So I'm not the only one who saw this quickly disappear.

Couldn't find it on the 2nd or 3rd page, very odd.

------
DarthSith
I'm a Linode customer. I have been calling on a regular basis because I have a
hosted service for my customers (thousands of them) that has been all but dead
for 24 hours now. If I hear 'we are working on it' one more time, I may lose
my mind. My customers are suffering severely, and I'm losing thousands of
dollars as we speak. They keep telling me they are working with their upstream
provider and that it's out of their hands. I'm not paying their upstream
provider. I don't have a service agreement with their upstream provider. I'm
paying them based on the service agreement I have with them. This is either a
completely new level of DDoS or they are just completely incompetent in their
way of handling it. In any case, I believe that Linode is going to suffer
greatly for this in terms of lost customers. They're going to lose me by the
end of next week, that's for certain.

~~~
dwightgunning
>> This is either a completely new level of DDoS or they are just completely
incompetent in their way of handling it.

My take is that it's somewhere in the middle. This type of outage has in the
past, and will potentially in future, hit providers at all price points
regardless of their SLAs and guarantees.

In my experience (and other comments indicate) Linode are genearlly very
reliable. This isn't a mickey mouse, dirt-cheap VPS operation. The
communication hasn't been awesome but by keeping an eye on status.linode.com I
personally have felt reasonably well informed. If you trust they're working on
it and doing the best they can.

Other commenters suggest this is a fairly sophisticated, deliberate and
sustained attack. They're putting significant resources toward it. It also
seems whoever is behind it has potentially gained inside knowledge of their
network topology.

Based that understand, I'm not wasting my time calling them or sitting around
hitting F5. I'm working to improve my systems' architecture for resiliency in
this type of situation. That involves geo-distributed, multi-site redundancy
and fail-over.

My advice: be optimistic and proactive.

