
Linode Manager and API are under attack - mherrmann
https://status.linode.com/incidents/6mpxv406bhq9
======
endijs
I'm not really sure why "Linode" and "DDOS" makes the news. It happens
regularly. And is nothing to be surprised about. What is new is how they
managed to get their systems upgraded to the point where DDOS does not matter
anymore. After big blackout (when they were under attack for weeks and even
one of their DC's was completely offline for more than a day) their defenses
have become way better. Of course - maybe attacks are not so big. I do not
know that. It's my 8th year with them. During those years only once (I think)
I had downtime because of those DDOS attacks. Not bad at all.

~~~
matt_wulfeck
It's a huge deal for a linode customer. It's not like your site is DDoS'd and
you can't access. You can't access _anything_ on linode when their entire
service is DDoS'd.

Part of the reason I think it's news is to shame linode. The likelihood of a
DDoS attack bringing down the AWS management console is extremely low (has it
ever happened?) Yet attackers have managed to do it multiple times on Linode.

~~~
endijs
I'm one of those customers (for 8th year already) and for me it's not a big
deal. Unless you are one of those high profile customers who needs constant
API access, those few minutes of downtime is not a big deal. Not everyone is
spinning up and down new instances all the time. Plus - not all attacks result
in downtime for them. Sometimes you do not notice at all, sometimes
responsiveness will suffer a bit. And as I said - their defenses have become
way better than they were a year ago. And it's not really fair to compare
Linode to AWS. It would be more interesting to see how would DigitalOcean
handle the same amount of attacks. And I doubt DO would handle them any better
than Linode.

------
xarope
I feel bad for Linode. They've always been a good host (I use Linode, as well
as Digital Ocean and others), so I would hate to have to switch away.

~~~
prawn
Same. I've found their service to be great and even today they went out of
their way to be especially helpful to me with a support query.

Can only imagine how much money and time is spent covering for malicious acts
like this, malware, viruses, etc.

------
throw2016
It feels off that Linode is being targetted so intensely and one feels bad for
them but this can destroy their business.

It will be a pity if the barriers to entry are being constantly raised by the
need for more and more mitigation and this will reflect in the level of
expertise needed and prices for end consumers, and even then with the constant
threat of blackouts.

It also becomes easy to silence inconvenient voices.

It is difficult to run a business or website if you need to constantly fight
extortionists or make payoffs to address threats of downtime. Today Linode is
in the news, tomorrow it could be anyone else targetted.

There is no easy solution here, without putting constraints somewhere but
there must be a way to make the web more resilient and robust and not subject
to the whims of extortionists and other malicious agents.

------
hnarayanan
This sort of headline involving Linode is (unfortunately) beginning to feel
extremely frequent. I am happy that I've since moved away to GCP, but this is
still sad to hear.

~~~
cstrat
I feel bad for them and their customers.

Wonder what the root cause of all this is, they seem to be a specific target
for some reason...

~~~
boulos
High value (a decent number of high profile customers) and yet easier than say
us (Google), AWS or Azure. As the last discussion brought up, I'm curious
whether Linode is particularly singled out compared to say Digital Ocean, OVH,
Hetzner, et al. but maybe someone like jgrahamc knows?

~~~
Joe8Bit
Easier in the sense they have less scale/capacity/capability to mitigate the
attacks?

~~~
samhamilton
Yes for sure, think about what Google etc had as teams before they all
launched their cloud compute services, I would bet they all had abuse
mitigation teams. In this case I would figure there is an overlap of the team
and tooling that keeps google.com up and the one that keeps my VM being hosed
by a DDoS

------
rebootthesystem
Linode customer here. Going to take a tangent.

Way back when, Toyota instituted a policy that required every line worker to
effectively stop the factory if a defect was found. Engineers would come to
the floor. They were tasked with creating a permanent solution for the
problem, one that would have that never happen again.

For the first few months Toyota had a very hard time getting cars out the
door. As time went by things got better. Eventually they got to the point
where nearly every car coming off the line was perfect.

Around the same time companies like Mercedes Benz were devoting no less than
25% of their factory floor towards fixing manufacturing defects. During this
same time period Toyota was devoting less than 5% of their factory floorplan
to fixes. This, ironically, meant that the Japanese company was probably
delivering a higher quality and more reliable product than the high end car
manufacturer.

Source: Read a book on the subject many years ago: "The Machine that changed
the world" [https://goo.gl/VQw6HS](https://goo.gl/VQw6HS)

Back to Linode. The fact that they've seen so many attacks gives them the
opportunity to become far better when compared to someone who never sees
attacks. Whether they go there or not is entirely in their hands. If this is
what's happening they need to take the time to communicate it to the world.

There are at least two possible attitudes: The first is to live in fear of
attacks and hope they never happens. The second is to embrace them as an
opportunity to get better.

The latter is a formula for disaster, or mediocrity at best. The latter is
supported by millions of years of history on this planet showing that things
get better because something or someone had to find a solution to a problem.

You can't become a good sailor without facing a few storms.

I'm sticking with Linode.

~~~
sb8244
The former is a recipe for disaster.

~~~
rebootthesystem
Darn! Too late to edit.

Thanks.

------
alexforster
I appreciate the kind words in many of the comments here, and I'm sure my
colleagues do too.

For reference, our third party monitoring measured approximately 23 minutes of
downtime–

    
    
      1m 46s @ 2016-09-08 01:37:53 EDT
      1m 10s @ 2016-09-08 01:38:14 EDT
      17m 4s @ 2016-09-08 01:40:28 EDT
      2m 11s @ 2016-09-08 01:58:29 EDT
    

We are now capable of absorbing most volumetric attacks toward our web
infrastructure. This outage was a layer 7 attack that needed to be manually
mitigated.

------
jupake
I'd be interested to know which data centres are being attacked. And why just
the api and manager?

~~~
boulos
I don't know where, but as to the why: control planes usually have fairly low
traffic (create an instance isn't that frequent per customer) so they'd be
easier to swamp if you can get it to consider your traffic. Proper load
shedding and rate limiting is kind of hard (you have to decide what you want
to happen to well intentioned folks), but anyway that's probably why.

~~~
andruby
Maybe this attack is part of a plan to get access to the control panel of
customers. That would be a way to gain access to some relatively high profile
sites.

~~~
gcb0
more likely just DoSing as a way to prevent people already hacked from
bringing hosts down.

------
ksec
It cant be a customers of Linode that is being targetted, Linode has seen WAY
MORE DDOS attack then any other web host.

Someone must have hated them or a competitor is trying to destroy them.

These frequent attack just isn't normal.

------
vorotato
I wonder if it has to do with people using linode as a way to get around a
particular country's censorship.

------
alex_hitchins
Out of interest, is there anything to suggest this is an outside attack or an
inside attack?

~~~
viraptor
outside.

1\. "We are currently experiencing a DDoS attack targeting our
infrastructure." \- you don't call inside attack a DDoS.

2\. For internal attacks they can just firewall off / kill offending servers.
It would be resolved in minutes rather than "mitigated".

~~~
alex_hitchins
Thanks, I suppose I should have figure this out. I wondered if there was some
attack that could be kicked of within the network that could be more difficult
to trace. I guess you could just looks at port traffic at switch level.

------
alvil
Qui bono? :)

------
deftnerd
My theory is either a ruthless competitor has been trying to destroy linode
for years, this is a distraction to cover a different attack on them, or at
some point in the past someone attacked and blackmailed them and Linode paid
to make it stop. Once you set that president, it's hard to make it stop.

~~~
jacquesm
It's 'precedent' and without any evidence these theories are mostly worthless.

~~~
deftnerd
You're correct in both regards. As for the first point, that was just an auto
correct mistake. My apologies.

On the second point, you're correct that my theories are entirely
unsubstantiated. I'm just trying to guess why Linode seems to be the victim of
these kinds of attacks a lot more than other providers.

~~~
jacquesm
The first would be fairly unprecedented and would eventually come out,
besides, whichever 'ruthless competitor' would initiate it, it would help
_all_ the other competitors equally so I think you can simply forget about
that one.

A 'distraction to cover a different attack': The more obvious explanation is
simply to deny control to people who wish to take down or re-image hosts that
have already been hacked, that's much more direct, distraction alone of course
could be a factor but is less likely if there is a more plausible explanation.

Finally, linode is on the record as _not paying attackers_ on principle in
spite of being DDOS's within an inch of their lives, so that's a pretty bad
thing to say about them unless you have evidence that they did in fact pay
attackers in the past.

What it _could_ be is a high profile example to scare smaller companies into
paying saying 'you don't want to happen to you what happened to Linode'.

~~~
ryanlol
>Finally, linode is on the record as not paying attackers on principle in
spite of being DDOS's within an inch of their lives, so that's a pretty bad
thing to say about them unless you have evidence that they did in fact pay
attackers in the past.

Not DDoS but they've got a history of doing _somewhat_ similar things re:
hacks in the past.

