
“We have been experiencing a catastrophic DDoS attack” - wowaname
https://status.linode.com/?
======
encoderer
I've done some googling before asking here: Can anybody explain why Linode is
so often targeted like this? We moved Cronitor off Linode in spring 2015.
During the christmas holiday when they suffered a 2 week DDOS I thought of the
family time I'd be missing that year as we did a crash migration to AWS had we
not migrated when we did. I have to imagine this has been _horrible_ for their
business.

I would use Linode if I needed to lease computational power, because it is
still a great value vs AWS, but I could not run a high availability service
there. It would feel like professional malpractice at this point.

~~~
funkyy
I am also surprised as this is not the first time I am reading about it on HN.
Linode seems to be highly vulnerable to certain attacks as we could see in the
past. I hope they will fix it and provide a permanent solution as I was hoping
to use them as a part of my network, but I see more and more signals they
can't handle serious traffic. Hopefully, they will redesign their
infrastructure to handle it.

I am with all of you guys that are affected by this. I am looking forward to
them to resolving this soon.

~~~
morecoffee
Isn't the problem money? They could certainly fix it but would have to raise
the prices on each VPS?

~~~
reefoctopus
I'd be ok with paying twice what I'm currently paying if they could solve this
problem. We're with Linode because it would cost us about 10x as much to run
on AWS, and we can't justify that.

~~~
hakanensari
They're price-matching DigitalOcean, so raising prices is probably out of the
question.

~~~
ksec
Linode is now half the price of DO per RAM

------
throwsep3
I wonder if this is a diversion to keep Linode's security team busy so they
won't notice someone compromising the Xen nodes with XSA-185/6/7/8?

~~~
technion
I've noted the AWS security bulletins[0] list nearly every Xen advisory with
"AWS customers' data and instances are not affected by these issues, and there
is no customer action required."

It would appear? you'd need to go back for quite a few months of being
unpatched to find a genuine issue. Unless something about Amazon's mitigations
don't apply universally.

[0] [https://aws.amazon.com/security/security-
bulletins/](https://aws.amazon.com/security/security-bulletins/)

~~~
lmz
AWS (and other large hosts: [https://www.xenproject.org/security-
policy.html](https://www.xenproject.org/security-policy.html) search for
predisclosure) get notified before the public.

------
matt_wulfeck
I disagree with people saying these types of attacks can't be prevented if you
switched hosts. I'm sure Google+cloudflare[0] would keep your website online.
AWS also if you had the cash.

The amount of distributed traffic happening right now against linode would
probably only represent a 5% increase in traffic to a popular Google product.
At least you know they have the expertise. Nothing against the very smart and
talented linode engineers, but the two companies are on _very_ different
levels of traffic engineering.

[0] [https://www.cloudflare.com/google/](https://www.cloudflare.com/google/)

~~~
the_duke
Google and AWS probably have sophisticated DDOS mitigation (can anyone comment
on this?) and you can scale up pretty quickly.

But if your service is the direct target of the attack (as opposed to the
whole provider) and your servers are getting hammered...

Even if you architecture allows quick horizontal scaling, you still face a
tough decision.

The attack could go on for days, and the hosting costs can go _really high
really fast_. Which can be catastrophic for a small company.

~~~
user5994461
Attacks are rarely targeted to the hosting providers. They usually target a
specific customer.

Google/AWS probably have 100 times the capacity (and redundancy and
architecture reliability and failover and awesomeness) of linode. That means
that, first, they can't be put down easily, second, a DDoS is limited to a
small subset of the infrastructure and doesn't bleed to every customers and
services.

As for traditional hosting companies (OVH and the likes) When you're being
DDoSed, they'll null-route your IP space. (i.e. they advertise your IPs as
dont-exist-on-the-internet-anymore). The traffic is dropped while in transit
on the internet because it can't go anywhere. It doesn't reach the hosting
company anymore.

Note: being null-routed means your site and all your services are off the
internet and thus effectively dead.

As for CloudFlare. They have many locations all around the world and they can
absorb a lot of traffic, to the point they themselves cannot be DDoS. They
have active monitoring and mitigation against common attacks and known
malicious sources, which may prevent the attack without even you knowing about
it.

When you're under attack, you can block subnet/AS/countries in cloudflare
settings, or request a challenge/capcha from every visitors. Cloudflare will
reject visitors (with or without challenging them) at their edge location
before any traffic can get to you. It is very effective from my experience.

Generally speaking. The only way to stop a DDoS is to do it before it reaches
your datacenters so you need help from your ISP/provider/CDN.

Edit: The attack that put down linode last christmas was against linode itself
and not a specific customer. Part of the mitigation included linode moving its
critical services behind cloudflare :D

~~~
iMerNibor
> As for traditional hosting companies (OVH and the likes) When you're being
> DDoSed, they'll null-route your IP space. (i.e. they advertise your IPs as
> dont-exist-on-the-internet-anymore). The traffic is dropped while in transit
> on the internet because it can't go anywhere. It doesn't reach the hosting
> company anymore.

OVH hasn't been doing this for a while, they got some beefy ddos protection
setup for this exact reason - it was way too easy to take down someone for
hours

Hetzner (another big european hosting provider) followed recently:
[https://news.ycombinator.com/item?id=12403783](https://news.ycombinator.com/item?id=12403783)

Online.net also has included protection (+ paid upgrades)

At least here in europe the big hosting providers are all switching to
providing included protection for all their customers, at least for traffic
intensive attacks which hurt everyone

------
morecoffee
It's not hard to compare this to brush fires. They happen periodically, and
only the big trees tend to survive them. Linode is getting pretty unlucky
here, but I would imagine that all the small time (and even the medium sized)
hosting provides are going to succumb eventually. Is the end game just going
to be Google vs. Amazon?

~~~
wowaname
I'd love to see a network infrastructure and transport protocol that's more
resistant to many (D)DoS attacks, because it seems like things will only
worsen if it never becomes more difficult for people to attack others' servers
online.

~~~
apapli
Good luck. A DDoS is basically lots of traffic. Perhaps run IPX? (Joking)

~~~
wowaname
If application- and transmission-protocol-level DoS vectors are fixed, then
you're left with just the raw "lots of traffic" volumetric attacks, which
means your attacker has to have a lot of compromised hosts (or the right
compromised hosts with lots of bandwidth). I'd say that's a reality that would
be easier to handle, because you raised the bar from anyone who can develop or
use a script and deploy to a few low-power systems, exploiting protocol
shortcomings, to only those who have a bunch of higher-powered systems.

The smaller hosting companies may still very well go out of the game if the
problem worsens, even if most DoS venues do end up being mitigated. I don't
know how I would respond to that as of this moment, but hopefully it doesn't
have to come to that. It's already tough to find a decent hosting company in
my experience.

------
simonmales
Play by Play of Linodes 'twelve day attack' over Christmas and New Year 2015

[https://blog.linode.com/2016/01/29/christmas-ddos-
retrospect...](https://blog.linode.com/2016/01/29/christmas-ddos-
retrospective/)

------
xmatos
I don't get the hate towards linode here, on hacker news. I've been their
client for a couple of years now and I find it an excellent vps provider.
Excellent uptime and performance at a pretty good price. AWS has a few outages
every year. Google just had one last week. Azure sucks balls. So, why the
hate? Is it because it competes with some ycombinator startups?

~~~
fizzbatter
I don't follow this too closely, so this is just wild speculation from me:

But could it simply be severity of the attacks? I keep seeing comments about a
2 week ddos attack last christmas - that's something that i would be shocked
to see Google/AWS succumb to. Not that Google/AWS attacks don't happen, i just
can't imagine them being down for ~2weeks

 _(I imagine it was just one datacenter from Linode, not the entire service,
fwiw)_

~~~
warbiscuit
They weren't down for the entire two weeks, but various datacenters went up
and down for hours, then quieted down for a few days, then was back again,
then another hit; stretching across two weeks.

One thing that took them so long was that their upstream ISPs at some of the
datacenters were _themselves_ unable to handle the DDOS, so they had to switch
ISPs, which took a while.

I don't see Google/AWS as easy to attack; but I'm not sure why similar tier
players like DigitalOcean aren't being hit -- or maybe they're just less
transparent about things, or are actually a smaller target (didn't _think_
they were?).

edit: here's a postmortem from linode of the christmas attack -
[https://blog.linode.com/2016/01/29/christmas-ddos-
retrospect...](https://blog.linode.com/2016/01/29/christmas-ddos-
retrospective/)

------
VonGuard
OK, who hosts at Linode and is very popular/pisses people off? 4chan? 9gag?
Reddit? Something Awful?

~~~
ryanlol
I think 9gag used to be on linode? I'm pretty sure none of the rest are
though.

~~~
discr3t3
Yeah Reddit is on AWS[1] and 4chan appears to be self-hosted (or at least it
was for most of its existence)[2]

[1] [http://highscalability.com/blog/2013/8/26/reddit-lessons-
lea...](http://highscalability.com/blog/2013/8/26/reddit-lessons-learned-from-
mistakes-made-scaling-to-1-billi.html) [2]
[https://www.4chan.org/news?all#106](https://www.4chan.org/news?all#106)

------
ablagoev
I've always wondered, while in similar cases GCE/AWS can handle the traffic,
is it not chargeable? So, while you will probably not get affected by the
DDoS, aren't the costs going to cut your head off?

~~~
kalleboo
Isn't incoming data on AWS free? Or are you thinking of some kind of attack
where they're using your infrastructure to amplify outgoing data?

Edit: I guess an attack that causes your infrastructure to auto-scale could
get expensive REAL quick...

~~~
technion
I'm running the maths right now.. and you could convince me to take down my
side project by just having a server outside their network put wget in a loop
targeted at my S3 resources.

------
ryanlol
>Update - We have been experiencing a catastrophic DDoS attack which is being
spread across hundreds of different IP addresses in rapid succession, making
mitigation extremely difficult. We are currently working with our upstreams to
implement more complete mitigation.

That's pretty harsh.

------
i_feel_great
Well, I had Linode shortlisted for an upcoming project. I hate to take them
off the list because it is not their fault, but I don't want this kind of
unreliability.

~~~
delroth
> it is not their fault

I don't understand this line of reasoning. It's not like DDoS attacks are some
kind of 0-day failure mode that nobody has seen before.

Would you also say "it is not their fault" if their uplink provider had a
fiber cut and they didn't have redundant uplinks? I'm guessing not: it's well
understood that has a service provider you need to plan for this kind of
unavailability and pay more money for redundant links. So it seems really
weird to have this double standard for a different kind of availability
failure mode.

Just like network availability or datacenter power availability, you need to
invest technical and financial resources into DDoS defenses if you want to be
resilient to incidents. If you don't do that as a hosting provider, I
definitely won't feel sad for you.

~~~
toomuchtodo
There are a handful of environments that can sustain a large, coordinated DDOS
attack. Can you sink 10-20Gb/s of traffic forever? Not cost effectively.

~~~
ryanlol
If you live in 2002, nah. In 2016? Yes. Bandwidth is cheap and linode has high
margins.

~~~
Tinyyy
Bandwidth is cheap but attacks are cheap too.

~~~
ryanlol
Which is exactly why you need to be prepared for them?

------
finid
This can't be good for (Linode's) business.

Vultr had problems a few weeks ago, but I don't think it was DDOS-related.

Somehow things have been quiet on the DigitalOcean's end.

------
diegorbaquero
3½ hours already. Distributed from and to many IP addresses. Seems like an
attack on them rather than on specific users. :/

~~~
tempestn
Or an attack on them as a vector to attack a specific user or users.

------
Dowwie
I have no evidence to support this theory, but I believe that Linode is not an
outlier with regards to frequent DDOS attacks. What makes this company special
seems to be with how it communicates to its customers when it's under attack.

This leads me to wonder: How much do other providers leave customers in the
dark?

------
jonahx
Can anyone recommend a good article that explains how attacks like these work,
and what is required to stop them?

Also, we're on Heroku and they advertise Ddos mitigation as a feature, but
"mitigation" sounds non-commital and I'm curious how they'd fare against a
similar attack?

~~~
viraptor
It's non-commital, because at some point, when you have enough zombie hosts
properly distributed all over the world attacking you, your only defence is -
have more bandwidth than the attackers. If your peers can't filter out the
traffic before it hits your network and it simply saturates your pipes,
there's nothing you can do inside the company anymore.

~~~
jonahx
Thanks for the reply. Can you elaborate on what you mean by "peers" in the
above? Eg, who (or what) would Heroku's peers be?

------
isarat
Linode is a cost effective solution comparing to AWS. But the security and
DDoS issues could raise eyebrows and bring confidence issues with customers.
This has happened even after they posted about enhanced DDoS mitigation
strategies like procuring more bandwidth etc.

If you look at the history of Linode discussions under HN, most of them were
related to DDoS attacks and service downtimes.
[https://news.ycombinator.com/from?site=linode.com](https://news.ycombinator.com/from?site=linode.com)

I hope they will recover soon and make the service stable.

------
bogomipz
I really feel bad for these folks. Does know if they have a DDOS mitigation
strategy other than RTBH with their transit providers? I would have thought
that after the 2015 attack they would have looked into traffic scrubbing with
something like Arbor Network or Prolexic. I understand that these are not
cheap and Linodes as well as many other hosting provider's margins are
probably thin but I would think that it would pay for itself in one or two
attacks by minimizing customer churn an event like that causes.

------
ekiara
I remember a few years ago when I moved my Linode from Fremont to Atlanta to
avoid the frequent outages. I've never had show-stopping issues with Linode
and the customer service has always been fast and responsive. Now though, I'm
thinking of moving to their Frankfurt datacenter.

But now I think I need to setup fail-over with another VPS provider. What's a
recommended alternative? Is Digital Ocean the next best choice after Linode?

------
pmalynin
This explains why Package Control, Ansible Docs, etc. were down. Sucks. I knew
this has happened before, just didn't know it was Linode.

------
chrischen
Does the US have a competent cyber-crime division that can handle stuff like
this?

~~~
SEJeff
Yes, the FBI. They're fantastic at it and part of their job is helping
businesses recover from compromise and going after the attackers. However,
they're overworked government employees with not enough resources.

~~~
matt_wulfeck
I think they very much want to, but what can you do if all of your IPs lead to
tor exit nodes?

From what I've heard, the FBI will collect a whole bunch of information and
then sit on their hands because of the above reason.

~~~
77pt77
How can tor nodes be used for that?

Tor exposes a SOCKS interface, you can't control TCP/IP with the level of
detail needed to perform these attacks.

~~~
gizmo686
Run the attack itself from a botnet of hacked computers, with the command and
control server proxied behind TOR.

------
jtl999
DigitalOcean has had a few DDoS attacks targeting their SFO1 datacenter over
the past few months, but fortunately each one seemed to disappear in under
half an hour.

~~~
hehheh
DigitalOcean is the source of many DDoS attacks. They've got a lot of
compromised servers on their network and ignore abuse complaints.

------
constit-protec
Godaddy has been getting attacked a lot recently as well. Who is likely behind
attacking servers, whether other server companies or governments?

~~~
mikebutash
When you are the dns root record and in many cases hosting too for some 60-70
million domains, someone in the internet is wanting to attack someone at
godaddy all the time. There isn't a time they _aren 't_ being attacked somehow
in all reality, and I'd presume it's much the same for Linode.

The means of really combating a ddos is costly and extensive, this is why most
use a service like prolexic or silverline, and typically with some massive
infrastructures that comes with it. Anything less than n-by 40gb disposable
internet pipes, preferably regional as you are, you can/will be smited at
will.

------
Scarbutt
What kind of DDOS attack is the most likely happening here, a simple L3
spoofed ICMP flood?

------
m0atz
Any details on the actual size of the attack and the attack vectors used?

------
ripken
You get what you pay for. TCO of running a highly available data center is
more than just the cost of metal, real estate, power, cooling, and cable

Going cheap with one of the non-big 3 public cloud providers will cost you.

------
thezach
I used to be on Linode... then one of their techs tripped on a power cord
causing a significant outage in a data center... and no I'm not joking.

~~~
jp_sc
You are confusing them with Dreamhost:
[http://www.manton.org/2011/03/brent_on_baked.html](http://www.manton.org/2011/03/brent_on_baked.html)

