
Linode is suffering on-going DDoS attacks - empressplay
http://status.linode.com
======
gtrubetskoy
I used to run a hosting company similar to Linode back in the day, and DDOS's
were the most annoying thing ever.

The main reason DDOS attacks exist is poor security and lack of cooperation
between ISPs. Lack of adequate security on desktops (usually Windows) makes it
possible to build large bot networks.

Lack of cooperation between ISPs makes it very hard to track down the source
of the DOS. Very often the DDOS isn't as distributed as it may seem - it can
just be a couple of machines on a very well connected network (e.g. a
university). But getting a hold of someone in the middle to filter that
traffic can take a very long time or be outright impossible. First-responder
network engineers (typically referred to as "security") are overworked and
underqualified, and the people who really know their stuff typically can't be
bothered with silly DOS attacks.

We've also observed that (D)DOS's happen because of content. Anything
political, religious, or whatever other shade of the many things someone out
there disapproves of is a potential target for a DOS. Contrary to what you may
read in the press, extortion is only a small minority of all DOS attacks out
there. We've actually told customers to go away because their content was too
DOS-prone.

And because these things usually happen across countries, even though they are
very real crimes that cause serious damage and cost money, they are hardly
ever prosecuted. As the target of a DOS all you want is for it to stop, nobody
ever bothers reporting it to the authorities afterwards (because how would you
even know who the "authority" is).

~~~
devicenull
> The main reason DDOS attacks exist is poor security and lack of cooperation
> between ISPs. Lack of adequate security on desktops (usually Windows) makes
> it possible to build large bot networks.

These days it's the hundreds of thousands of misconfigured NTP servers,
recursive DNS servers, and various other protocols being abused for reflection
attacks.

Granted, it still requires that the attacker have the ability to spoof
packets, but preventing that requires even more time investment and has very
little benefit to the ISP.

~~~
jsmthrowaway
BCP 38 is like herd immunity with immunization, and much like anti-vaccine
folks, networks that don't follow it are knowingly choosing to infect people
for any number of reasons. Despite your claim, it is _extremely easy_ to
implement and has been a known best practice, with accompanying educational
Web sites devoted to the topic[0], for many years. There are nearly zero
reasons for your AS to transmit forged packets, and if your configuration
allows it, you are _knowingly_ making the Internet a worse place with your
laziness and transferring your laziness to other people like me that carry
pagers. This isn't a surprise to any network administrator unless they've
spent their entire career not reading RFCs.

I'm of the opinion that networks that allow customers to emit forged source
addresses should be depeered until they take the literal hour to fix it. "But
we have to update equipment when we get new blocks!" Boo hoo. Automate it or
get off the Internet so I can stop spending my life dealing with your
customer's amplified traffic.

If you run a non-transit/eyeball AS, you are in the absolute best position to
stop these types of attacks from ever happening. The rest of the Internet,
particularly your transit peers, can't really clean up after you on this one.
Do us all a favor.

[0]: [http://bcp38.info](http://bcp38.info)

~~~
secstate
Given my ignorance of much of these issues, I probably shouldn't be commenting
(take my comment with a huge grain of salt). But the idea of depeering
networks on the Internet for misapplication of a voluntary protocol seems like
the beginning of the end of a free Internet (if ever such a thing existed).

If BCP38 is critical to the success of the Internet, I think rather than
ranting about those not implementing it, energy would be better spent
petitioning to have it made a requirement of running a peer on the Internet to
begin with.

Perhaps I'm off base or starting some sort of network guru flamewar. If that's
the case, down vote me and I'll go away ;)

~~~
jsmthrowaway
Your heart's in the right place, but the Internet is built on policies of
individual networks because there is nobody to enforce. Your suggestion back
to me is simply mine in different clothing, because you think someone can
enforce such a global requirement. Enforcing policy like "filter or get
depeered" is the only way to achieve a global requirement like you want with
the way the Internet is structured. As akerl points out you need consensus,
too, because such a policy could drive customers to other networks upon
enforcement, which is a business disincentive to do it.

It's kind of a surprising moment when you realize what the Internet is and how
little structure it has aside from the protocols themselves. We are one global
Internet (semi) outage away from rethinking some of this structure, and I
expect one in my lifetime.

~~~
secstate
Thanks for the kid-gloves reply :)

I hadn't considered that there isn't really a central authority for
controlling who runs a peer, aside from ICANN, but they have pretty loose
reins.

Funny that everyone waxes poetic about bitcoin being a revolution in anonymous
and tacit network management. Meanwhile our little Internet experiment
continues to be a HUGE tacit agreement to adhere to a handful of network
protocols.

~~~
jsmthrowaway
If you talk to ICANN, IANA, and friends, they're very clear (and careful to
reiterate, even in minor threads) that all they do is run databases that
contain interesting information. All of their policy revolves around admission
to and management of said databases. It's comforting to think of authorities
that govern the Internet, particularly when one of those organizations even
has "Authority" in the name, but the truth is that the Internet is a miracle
of decentralization as you say; we as network operators on the Internet give
the databases meaning and force, not ICANN or any of its subordinates. ICANN
has power because we have chosen their databases as the root of said power.

Even the fact that 'news.ycombinator.com' leads here is a de facto consensus
since the Internet could, theoretically, switch roots tomorrow and completely
invalidate every domain name. It won't happen, much to the chagrin of alt-root
operators from the 90s and 2000s and contemporary attempts like NameCoin[0],
but it _can_ , and there is absolutely nothing ICANN could do about it. They
charge admission to a well-maintained database that underpins this whole show,
and that's pretty much the entirety of what they do.

It's on _all of us_ to tend to the best interests of the Internet, and way too
many people with access forget that. However, with responsibility like that
also comes opportunity: once you realize that it's basically all of our good
faith and de facto consensus holding this thing together, the barrier to entry
for you to get involved with the Internet is suddenly far lower.

[0]: Oh yes, it's been tried, a _lot_ :
[https://en.wikipedia.org/wiki/Alternative_DNS_root](https://en.wikipedia.org/wiki/Alternative_DNS_root)

------
USNetizen
I find this ironic because about 2 years ago I had a couple VMs with them that
suffered CHRONIC DDoS attacks, all the time. I had to move my clients to a
whole other platform. Linode, on the other hand, simply blamed us for
supposedly causing the repeat DDoS attacks - one after another for months on
end. They even got rather flippant with me exclaiming how "dumb" I was to not
understand that is was MY problem apparently, not theirs, even though the
target of the DDoS was several addresses in their IP block and not tied to a
single client domain, server or anything else. They continued to blame us
personally, and even tried to get us to foot the $3,000+ traffic bill.

Well, the tables have turned. Looks like it IS a Linode problem after all. And
not a one-time issue, either. This has been happening to them for YEARS.

~~~
workitout
Every point you make I can counter, I've been with them for years, have
multiple VMs with them, find their support team fast, competent and courteous.
DDOS is a problem for every ISP and every ISP customer.

~~~
USNetizen
Not so fast. We migrated all of our former Linode clients to AWS and Azure and
have not suffered a single DDoS in the roughly 2 years since the move to other
providers. At Linode, on the other hand, we suffered more than a dozen in less
than a year - even with different VMs, different IPs, different OSes and
different configurations. We were also plagued with overwhelming attempts to
brute-force SSH and other services from IPs in China while we hosted with
Linode. Nowhere else did we see so much malicious activity targeted against
our clients' VMs as we did at Linode.

During that time, Linode support tried ruthlessly to pass the buck and blame
us and our clients, denying they had any major problems. Well, here we are and
Linode DDoS attacks are STILL occurring on a regular basis, still making news,
and they seem to do little about it in terms of a long-term solution. Whereas
I have yet to have a single client suffer a Linode-scale DDoS on AWS or Azure
at all after almost a half decade of using them.

~~~
mordocai
To be fair, i've had a single linode VPS for years (i'd have to check to
figure out how many) and this ddos is the first time i've ever had any issues
whatsoever with my linode.

~~~
USNetizen
Luck of the draw. Certain IP blocks in the Linode range are attacked all the
time, as evidenced here. We had a couple VMs that were never attacked, but far
too many of them were on a regular basis for us to even consider staying with
Linode.

------
mmaunder
So weird that Linode hasn't been able to mitigate this. I'd love to learn more
about what's happening there. Particularly since we host our production
documentation site on a linode vm. I want to move it off their server and into
our DC but can't access the server to do it. Bummer. I've been such a big fan
of theirs.

~~~
kyrra
If the DoS is large enough there is little they can do if their downstream is
100% saturated. They would either need more capacity or for their upstream
providers to filter the attack for them.

~~~
ryanlol
Heard of nullroutes?

(Edit: how is it that perfectly valid technical solutions get downvoted?)

~~~
scurvy
I didn't downvote, but there's one thing to keep in mind. You must request
RTBH functionality with every peer I've ever worked with. It's free, but they
don't set it up automatically. You need to request it during provisioning or
wait 3-5 days for someone to activate it.

If you don't already have it enabled....well, good luck mate.

~~~
jsmthrowaway
As of a few years ago Linode got transit from the facilities they are in and
almost all of them had RTBH set up with a capacity of 5 or 10 routes. It would
be incredibly foolish to operate a hosting provider without it.

~~~
scurvy
Wait, you're saying that Linode uses facility transit? Like, they buy
bandwidth from Savvis and TelX? Well that would be the problem right there.

From what I can tell, Linode doesn't even have their own AS for customer
traffic? It appears that they have an AS for some internal use, but not for
customers?

~~~
jsmthrowaway
I don't know if I'd outright call it a problem, per se, since in a couple
facilities (say, Fremont) it's a good thing given how good the provider (say,
Hurricane Electric) is at providing transit. So in the datacenters where the
provider is better at transit than facilities (say, Fremont), the equation
breaks in their favor.

When I left they also didn't really have the staff to run a proper NOC for a
full-on AS; one extremely talented network engineer who has since left mostly
owned all the "running a grown-up network" stuff, like running down an ASN,
working RTBH with the facilities, well-designed uplink strategy at Linode's
edge, and lobbying for anycast infrastructure for DNS and so on. They had just
hired another purebred networking wonk as I moved on who probably owns it now,
and likely has help. So punting a lot of the typical NOC work upstream to a
facility with whom you're already working anyway made a bit of resource-
related sense at the time, since staffing a NOC for a network of that scale is
a significant challenge and they like to operate lean.

That's changing now (I didn't know this and learned it from agwa's comment),
which is probably reflective of growth on the resource front.

~~~
scurvy
Real talk: it's not that hard to run your own network. It sounds like Linode
was skirting by on not making the infrastructure investment both in hardware
and people. It's finally come back to bite them. No excuses and no pity for
them.

Also, I wouldn't run around saying that having HE in house is a great thing.

------
rdl
If anyone from Linode admin team would like some help analyzing the
attack/friendly advice on mitigations, please contact me (or anyone else at
CloudFlare); we see a lot of these.

~~~
imoff
How do you guys trace the real packet sender of a packet with a spoofed IP
address?

~~~
rdl
We don't, generally. It would be challenging.

If it were a volumetric attack, you could walk back links to find the source.
But for anything but a huge attack which lasted for weeks/months, that would
be inefficient use of resources.

Paul Vixie is really at the forefront of pushing for providers to solve this
problem. Until that happens (and they've tried for a decade), it will remain
technically difficult/impossible, so you need to use other solutions to
mitigate attacks.

------
at-fates-hands
Looks like they have a history of suffering these kinds of attacks:

(2012) Upcoming DDOS Attack - FINAL Warning -
[https://forum.linode.com/viewtopic.php?t=8530](https://forum.linode.com/viewtopic.php?t=8530)

(2013) Linode Mitigates DDoS Attack on Linode Manager -
[http://www.thewhir.com/web-hosting-news/linode-mitigates-
ddo...](http://www.thewhir.com/web-hosting-news/linode-mitigates-ddos-attack-
on-linode-manager)

(July 2015) Incident Report for Linode -
[http://status.linode.com/incidents/vnslh3rmm9gq](http://status.linode.com/incidents/vnslh3rmm9gq)

So what makes them such an attractive target for these types of attacks?

~~~
xena
The fact that so many people on their IRC channel will react to the attacks.

------
technikempire
Whenever I see top level comments where people are saying "this is what you
get for not going with X", I am more and more convinced that a competitor is
doing this. I for one am not leaving Linode. I JUST recommended to several new
clients to setup at Linode and I've already had to explain that server setup
is halted because of this attack. Did I recommend moving away from Linode?
Nope. Did I hang my head in shame or stutter when I delivered the news,
feeling it would reflect poorly on me or Linode? Nope. I told my clients this
is the best place for them and I stick by it. Just because there are scumbags
specifically attacking your service right now, doesn't mean another service is
better, or that your service is poor. It just means that hey, news flash,
there are scumbags in the world.

------
mwcampbell
Yep, this started on Friday (Christmas day). I assume Linode itself is the
target of the attacks, since they have spanned multiple regions.

~~~
timdorr
Dallas is their oldest DC, so they have the most customers there and therefore
the most attack surface area at that location. It could just as easily be one
of their customers.

~~~
dangrossman
The DDOS has hit their data centers in Dallas, Fremont, London, Newark,
Atlanta, and Singapore since Christmas.

------
thrownaway2424
I would like to correlate the comments in this thread with past comments on
every single article about AWS or GCE of the form "this is so expensive /
complicated I run my boxes on Linode for half the price". DDoS protection is
one of the things you pay for on the big clouds.

~~~
scurvy
What DDoS protection does AWS provide? The only thing mentions on their
webpage is autoscaling, more nodes, etc. In other words, AWS' DDoS protection
strategy is to open up your wallet.

About 6 months ago they did hire Jeff from BlackLotus. Given that timeline,
I'd expect them to announce some sort of DDoS protection offering in the next
few quarters.

Edit to be more specific: AWS gets hit with a lot of DDoS attacks. While all
of AWS isn't unreachable during an attack, parts of it are. It's so large that
you might not notice, but parts are unreachable. AWS/GCE size only makes it
less noticeable, but they have no customer facing DDoS protection offerings.
Their only offering is to buy more of their services. These providers don't
have magical 1000000gbps links. They're regular 100gbps links (or 100gbps LACP
channels) that can get overrun in large enough attacks.

~~~
thrownaway2424
I don't know, but all traffic to GCE is routed through Google's frontend,
which provides in-built DDoS protections.

~~~
scurvy
I'd imagine they use VRF's to quickly segment the traffic after ingress.
Google.com might have DDoS protection, but I'm wary that it extends to GCE.
I've read about Google Andromeda, but there's no real meat in any article
about DDoS mitigation.

~~~
thrownaway2424
This document specifically claims that "All traffic is routed through custom
GFE (Google Front End) servers to detect and stop malicious requests and
Distributed Denial of Service (DDoS) attacks."

[https://cloud.google.com/security/whitepaper](https://cloud.google.com/security/whitepaper)

~~~
scurvy
How though? There's remarkably little information in there for being a
whitepaper. If all they do is drop Christmas-tree packets and bogus UDP/ICMP
traffic, that's not much in the way of protection. I'd like to see exactly
what/how they're doing. How do they know what traffic is malicious? Do you get
a control panel to block certain L7 traffic? What L7 inspectors do they
support?

Sorry, but I'm not buying it.

------
silverlight
Been happening daily for about 5 days now. Pretty frustrating but I'm sure
it's frustrating for them too.

Thinking about moving to Google Compute Engine instead.

~~~
empressplay
I worry that moving to a larger company will just hasten the consolidation of
hosting to a few players.

~~~
swiley
The best thing for reliability is to use multiple companies. AWS has proved
that multiple times.

~~~
yeukhon
Disagree. It is hard to maintain codebase and consistent infrastructure setup
for multiple providers. Amazon's m3.medium != some vendor's m3.medium. Network
setup and configuration are also nightmare. Speaking from experience dealing
with four cloud vendors at once. It sounds great from a textbook perspective,
but unless you are ready to spend millions every year to fight fire, please
don't do that. If you were to use AWS, please build on multiple regions.

~~~
mark-ruwt
I think you're looking through the wrong end of the telescope, here.

We currently have ~50 servers in 8 cities, across Linode, Digital Ocean, and
Vultr. It took me two weeks to craft a ~400 line script that abstracted the
server creation APIs for each. Once spun up, they're each bootstrapped with a
script that builds each server from scratch identically regardless of the
provider (with a couple one-offs for Vultr), because they're all running the
same distro.

A whole data center can go down, and there's no reason for me to get out of
bed.

~~~
yeukhon
Maybe, maybe not. I still believe in single-vendor approach, perphas because
in my view I am either going for AWS or GCE.

There is so much to with than just be able to spin up an VM and then run
Ansible/Chef/Puppet on it. Heck I can write all of that in Fabric. There is no
direct connect on Digital Ocean. I am not sure how you set up VPN with Digital
Ocean or Linode. We use cloudformation on AWS, and I am pretty sure there is
no such thing on Linode or Digital Ocean. Exception and response codes
different across providers. Able to reproduce an environment from scratch is
important to us, and of course, we try to do that in stages. I own a DO box
myself, and that box turns out to be really slow in the NY region (where I
live), maybe I am just an lucky bastard.

But to be honest, did you really build your entire infrastructure in three
vendors to begin with? What are your reasons to really build on Linode,
Digital Ocean and Vultr? How do you copy your data across environments? Are
you splitting dev/qa/ci/sandbox/stage/prod?

~~~
mark-ruwt
No, not at all! [http://areyouwatchingthis.com](http://areyouwatchingthis.com)
is almost 10 years old at this point, and the architecture in its current form
wasn't solidified until last year. As a one-man shop, redundancy and failover
are my best friends.

------
mikegioia
This may be a dumb question as I haven't read the bill, but isn't this what
CISA is supposed to help guard against in the future? If Linode has an easier
way of sharing info with the USG, can that help pinpoint and mitigate this
attack from happening in the future?

Edit: how about a response instead of downvotes?

~~~
pjc50
Dealing with DDOS has always been way below anti-piracy on in the priority of
the relevant US authorities. It's very easy to do and hard to identify those
ultimately responsible. Although every now and again a big botnet command-and-
control network gets shut down.

Nobody's interested in the _defensive_ side of ""cyberwarfare"".

~~~
fanf2
Botnet takedowns seem to be mostly due to industry cooperation without
significant help from the cops.

------
peterhadlaw
It's funny that it took an event like this for me to realize this but now I
see that it's really a bunch of companies (like Linode) all leasing out space
in some huge data-center. I use ASO vps hosting and they too have been
experiencing a large network outage. I'm sure you can find at least a handful
of hosting companies that are SOL right now too. I guess I understand the
whole premise of putting all your eggs into one basket, but then again you
really shouldn't.

------
jafingi
My four London servers were also unreachable last week. Looks like Linode have
a pretty rough time.

------
Keverw
Oh wow. I remember a couple months ago the ATL datacenter had network issues
too. Really annoying, but I guess it's not their fault 100%. I wish they
offered more DDoS protection solutions. I know some VPS companies specialize
in that offering for things like game servers. It'd be nice if some sort of
solution could just be included. I don't know if it's more of a technical
issue or legal problem. As far as I know the only way is to null route(which
is an inconvenience to the website, but no traffic going to the router other
client's are on) or just adding a bunch of servers to "soak" up the extra
bandwidth.

~~~
X-Istence
OVH does VPS's and they have their own Anti-DDoS network setup that is pretty
amazing:

[https://www.ovh.com/us/anti-ddos/](https://www.ovh.com/us/anti-ddos/)

~~~
Keverw
Interesting. What is "Multi-point Mitigation"? I know it mentions a few
locations. I googled it and it just brings me back to that page.

I wonder if any solution would shutdown a VM and then restart it on another
host but that'd be really sucky in some situations like an app might not
shutdown cleanly, or the app is in the middle of something like charging a
credit card.

~~~
X-Istence
Multi-point just means that the traffic is going to flow to the nearest
datacenter that hosts the VAC, gets filtered and checked and then traverses on
to your system.

The goal is to spread the DDoS out over as many different bandwidth heavy
locations as possible.

See the images at the bottom of this page:

[https://www.ovh.com/us/anti-ddos/hoovering-
up.xml](https://www.ovh.com/us/anti-ddos/hoovering-up.xml)

------
latenightcoding
Strangely this whole thread convinced me to try Linode. Why? everyone keeps
saying it's so cheap and I do A LOT of web crawling so I don't need 99.999%
uptime

~~~
blisterpeanuts
I use it for basic stuff, mainly for testing node apps, no high throughput.
It's got a nice, simple management console. I don't like the fact that apt-get
operations to set up your VM can blow your monthly quota pretty fast if you're
not careful, but other than that, it's been fine. Periodically, they double
everyone's bandwidth and storage for free; it's something to look forward to
in another year or two... I hope.... if these DDOS idiots don't ruin it for
everyone.

~~~
stefantalpalaru
> I don't like the fact that apt-get operations to set up your VM can blow
> your monthly quota pretty fast if you're not careful

What are you talking about? From
[https://www.linode.com/pricing](https://www.linode.com/pricing) :

"Please note that all inbound traffic is free and will not count against your
quota."

~~~
jonathanoliver
Any the distros are configured to use local Linode mirrors for apt-get anyway.

~~~
stefantalpalaru
All the inbound traffic, not just traffic inside the Linode network. "Inbound"
means from the Internet to the Linode-hosted server.

------
tshtf
Just 2 days ago, a Linode employee was badmouthing AWS here on HN for being
too expensive:

[https://news.ycombinator.com/item?id=10796094](https://news.ycombinator.com/item?id=10796094)

A DDoS will be much more expensive to customers than choosing AWS over Linode
(or an equivalent low-priced service).

EC2 has been around since 2006, and never has had any issues resembling this.

~~~
scrollaway
So you're comparing DDoS of a provider with everyday spending?

I was contracted a few months ago to save a website crumbling under its bills.
They had $11k / month in AWS bills. Brought it down to $600 / month by
switching them to Linode with a more reasonable stack.

Not everything is a nail. Sometimes the mistake is choosing AWS.

~~~
cynicalkane
If you reduced their bill from $11k to $600, I'm betting AWS was not the main
problem.

~~~
workitout
Given that the price calculators for Google Cloud, HP Cloud, Rackspace, etc
are online, you can check for yourself. The key one is bandwidth cost for
their services.

------
jafingi
Linode is _still_ hit by DoS. They currently have a "Major Outage" in the
London DC. Seems like there are issues in almost all DC's.

Unbelievable they can't get that DoS stopped! My server have had 19 outages
the past 7 days, and over 5 hours of downtime!

Still no e-mail from Linode whatsoever..

~~~
pbowyer
I know. I didn't mind when the first outage happened in London (24th?) - these
things happen. But now... this is getting beyond a joke. I'm going to have to
consider moving, which as I host for friends is not as straightforward as it
should be (since they control the DNS)

------
TheWoodsy
I wonder if this is correlated to the extortion thread on NANOG I saw the
other day.

------
altern8
For a change... :-/

------
Hello71
Link says DoS, not DDOS (which isn't even a proper acronym).

~~~
robandrews
DoS == Denial of Service; DDoS == Distributed Denial of Service

