
Linode Security Advisory - scub
https://blog.linode.com/2016/02/19/security-investigation-retrospective/
======
tptacek
_Update: also, read this comment_ right away.
[https://news.ycombinator.com/item?id=11136948](https://news.ycombinator.com/item?id=11136948)

I find this update very hard to follow. Can someone tell me if I'm misreading
it? I'm going to quote it twice, and then attempt to summarize:

 _After examining the image from our July investigation, we discovered
software capable of generating TOTP codes if provided a TOTP key. We found
software implementing the decryption method we use to secure TOTP keys, along
with the secret key we use to encrypt them. We also found commands in the bash
history that successfully generated a one-time code. Though the credentials
found were unrelated to any of the unauthorized Linode Manager logins made in
December, the discovery of this information significantly changed the
seriousness of our investigation._

and then:

 _The findings of our security partner’s investigation concluded there was no
evidence of abuse or misuse of Linode’s infrastructure that would have
resulted in the disclosure of customer credentials. Furthermore, the security
partner’s assessment of our infrastructure and applications did not yield a
vector that would have provided this level of access._

 _Linode’s security team did discover a vulnerability in Lish’s SSH gateway
that potentially could have been used to obtain information discovered on
December 17, although we have no evidence to support this supposition. We
immediately fixed the vulnerability._

Here is my read of what this says; I'd like to know if I'm wrong.

"One of our customers got owned up in July, and gave us an attacker source
address within Linode. We pickled up the attacker's host. In December, we
examined the pickled host, and found secrets related to the way we store 2FA
credentials, indicating that our credentials database may have been
compromised. In conclusion: we have no idea how that could have happened."

Am I missing something else?

~~~
aqtrans
Yeah, they really gloss over the fact they have no idea how the TOTP secret
key was compromised, which worries me the most.

~~~
SomeCallMeTim
They changed the 2FA to use a microservice, so _whatever_ the vulnerability
was before, if the 2FA is now on an isolated server, that vulnerability
shouldn't have access to the new 2FA key.

~~~
KaleidoscopeFan
I think it's fairly important to note that they're NOT currently using the
microservice for the 2FA, and they're NOT using bcrypt right now.

The blog post states they're "working towards" these changes, they're not
currently in place. It's fairly unlikely that they're using the same secret
key as the one they found on the server, but it's fair to assume that they are
still using salted SHA-2 for your passwords and the same 2FA setup right now.

They likely won't roll out the major changes until they roll out the "new and
improved" Linode dashboard they're coming up with.

~~~
monster2control
The article didn't state that. The article stated they are rolling out soon.
The new dashboard will be an open source project. So you'll know when that
gets released. There is no link to the project yet so assume that part isn't
started yet. So the microservices should be released in a timely manner. Let's
hope with the new focus on transparency if there are any delays they will keep
us posted.

~~~
KaleidoscopeFan
Isn't that exactly what I said? o.O I said they will "likely" get rolled out
with the new dashboard, not that the article said they would lol. But, they
never stated when it would happen anyways, so "delays" aren't really a thing
when there's no deadlines.

------
AlbertoGP
The page's title "Security Investigation Retrospective" fits the content
better than the current HN's "Linode Security Advisory": it is about last
July's breach that caused January's password reset; what happened and what
they've done about it.

------
noir_lord
Not sure what to think about Linode anymore, on the one hand from a pure
reliability point of view they have been bullet proof, had a few issues during
the DDoS in December and I've always found their support to be good (the few
times I've used them in 7 years).

On the other hand they've had security issues fairly regularly and their
response to the DDoS was pretty poor.

That said if I was a cynic I'd say they probably have one of the best setups
for dealing with future attacks (old joke about never firing an employee who
made an expensive mistake because he'll never make that mistake again) and
_finally_ seem to be taking security seriously, for me the list of changes all
sound good (particulary open sourcing Linode Manager and tokenising CC's had
to reset a card because of them once before).

We are slowly moving a lot of stuff back to a DC up the road but we still have
some stuff with them.

~~~
odonnellryan
I feel DO's level of service is on-bar. I've gotten multiple discounts from DO
for "annoyances" I wasn't even annoyed by.

~~~
developer2
This is because DO is desperate to retain customers. I don't know if it's
still the case, but a year or so ago the CEO was personally handling customer
service, responding via email and adding credits to people's accounts. If the
CEO has that much time on his hands, to personally handle every customer
dissatisfaction, then the customer base must be quite small.

I think for me the largest red flag was DO not even having bandwidth
monitoring in place. The only reason there were no bandwidth limitations was
because they were not even physically tracking bandwidth usage. How do you
launch a hosting platform without something as basic as being able to monitor
bandwidth?

~~~
AdamGibbins
DO aren't small: [http://news.netcraft.com/archives/2015/05/01/digitalocean-
be...](http://news.netcraft.com/archives/2015/05/01/digitalocean-becomes-the-
second-largest-hosting-company-in-the-world.html)

------
ryanlol
>Linode’s security team did discover a vulnerability in Lish’s SSH gateway
that potentially could have been used to obtain information discovered on
December 17, although we have no evidence to support this supposition. We
immediately fixed the vulnerability.

Yeah, LiSH was exploited like 3 years ago...

Unless it's been completely redesigned it's probably still got heaps of
vulnerabilities, screen wasn't designed for untrusted input.

>CC Tokenization: Although our investigation yielded no evidence of credit
card information being accessed, we are taking advantage of our payment
processor’s tokenization feature to remove the risk associated with storing
credit card information.

Nobody thought about doing this when every customers card plaintext card info
was taken years ago?

Anyway, the entire blog post is bullshit. It completely fails to address their
previous security track record. A single hack isn't that big of a deal, but
this has happened to Linode countless of times.

------
thesimon
It seems like the post creates more questions than it answers, but it's great
that they are sort of transparent. I guess it's due to ongoing investigation.

But it is quite surprising that someone was able to acquire the key for the
token generation and they seem to have no explaination for it. And wow, they
only started tokenizing credit cards now? And SHA-2 for password hashes?

THB, after reading this post my confidence in them hasn't really increased. It
feel like: "We fucked up, but are not exactly sure why but will fix some
issues nevertheless".

~~~
msbarnett
> SHA-2 for password hashes

They're moving on from them, but they're going to leave SHA-2s sitting there
and wait _until_ everyone logs in to upgrade to bcrypt hashes at rest.

Not getting a super competent vibe off of these folks.

~~~
elbee
Waiting until login until you upgrade to bcrypt is a requirement is compotent
password storage. At this point in time all Linode should know is
SHA-2(password) and they can't use that to derive bcrypt(password).

The way upgrade should work is that the user provides their password, which is
verified with SHA-2 and then hashed with bcrypt and stored again.

In order to do this without people logging in Linode would have to bcrypt hash
the SHA-2 hashed passwords and then keep doing that for all password
validations.

~~~
msbarnett
> Waiting until login until you upgrade to bcrypt is a requirement is
> compotent password storage

It's not even remotely competent. This blog makes it clear they're not even
sure how their secret key was stolen. These hashes could be walking out their
backdoor as I type this. Keeping vulnerable hashes at rest is insane.

It would be far more competent to bcrypt the SHA-2s, so that at least when the
hashes wander out the backdoor they haven't really found, peoples passwords
aren't trivially attackable.

> In order to do this without people logging in Linode would have to bcrypt
> hash the SHA-2 hashed passwords and then keep doing that for all password
> validations.

No, they'd just have to replace Bcrypt(SHA-2(password)) with Bcrypt(password)
once the customer finally logs in.

It's an immediate net upgrade to the resistance of at-rest Hashes to brute
force attacks with zero downside.

~~~
elbee
That will work as a way to strengthen the hashes (a few other people pointed
that out as well).

My point was that if you have a system which can go straight from
SHA2(password) to bcrypt(password) then the system must be storing the
plaintext of the password, which would be very bad.

~~~
msbarnett
> My point was that if you have a system which can go straight from
> SHA2(password) to bcrypt(password) then the system must be storing the
> plaintext of the password, which would be very bad.

Yes, I understand that. It's just completely irrelevant to the question of
whether or not it's competent practice to store vulnerable hashes
indefinitely, awaiting customer log in.

Again, it is not a competent practice. Wrap vulnerable hashes in strong ones
immediately; they're a huge liability to leave sitting in your storage even
when you don't have evidence that there's a backdoor in your systems that you
cannot seem to find.

------
colinbartlett
Two things in their "What We’re Doing About it" section really surprised me
because they are things I would have expected a company of their size and
sophistication to have done long ago:

> we are taking advantage of our payment processor’s tokenization feature to
> remove the risk associated with storing credit card information

> we are hiring a senior-level security expert

------
TheSwordsman
Hey There,

I'm a PagerDuty employee and am the same individual who made this post on the
last HN thread:

* [https://news.ycombinator.com/item?id=10845985](https://news.ycombinator.com/item?id=10845985)

Unfortunately, there are some facts in Linode's post that are not correct.

>On July 9 a customer notified us of unauthorized access into their Linode
account. The customer learned that an intruder had obtained access to their
account after receiving an email notification confirming a root password reset
for one of their Linodes. Our initial investigation showed the unauthorized
login was successful on the first attempt and resembled normal activity.

This is almost correct. Someone got in to our account on the first try. They
knew the password and a valid TOTP token. Although, Linode's email isn't what
notified is, it was our intrusion detection system.

>On July 12, in anticipation of law enforcement’s involvement, the customer
followed up with a preservation request for a Linode corresponding to an IP
address believed to be involved in the unauthorized access. We honored the
request and asked the customer to provide us with any additional evidence
(e.g., log files) that would support the Linode being the source of malicious
activity. Neither the customer nor law enforcement followed up and, because we
do not examine customer data without probable cause, we did not analyze the
preserved image.

This is partially correct. We informed Linode that we saw suspicious activity
within their network, and reached out to them to inform them. We provided any
and all logs we had. We also informed them that we passed the info on to law
enforcement, in case they wanted to proactively preserve the data. The knew we
had no further information, and as such didn't ask for anything additional.

>On the same day, the customer reported that the user whose account was
accessed had lost a mobile device several weeks earlier containing the 2FA
credentials required to access the account, and explained that the owner
attempted to remotely wipe the device some time later. In addition, this user
employed a weak password. In light of this information, and with no evidence
to support that the credentials were obtained from Linode, we did not
investigate further.

The story behind the mobile device is totally incorrect. The user did not lose
their device, the device had been restored (intentionally wiped) 9 months
prior to the compromise. The user got a new device, and never set up MFA on
their new phone after wiping the old one. The device was, and still is, in the
user's possession. The device has not been powered on in a long while.

The user who was compromised was no longer in possession of their MFA secret.
They deleted it, intentionally, with no backups existing.

If anyone here is going to be at Velocity 2016 in Santa Clara, or at
Monitorama PDX 2016, I'll be giving talks on how PagerDuty was compromised
back in July. This includes _full_ details of how this happened, including the
details of the mobile device referenced above. There are some details in my
talk that don't line up with the blog post provided by Linode. :)

~~~
ethomson
This is very helpful information. Can you say if you've moved to a different
provider or if you're now racking your own machines? And if you do have a new
provider, can you say who you are and how you evaluated them?

I have been doing some research in my (limited) spare time to try to find a
new provider, but I still have not made the switch from Linode.

~~~
tptacek
If you follow the link he posted, you'll see they switched away from Linode
almost immediately after the July breach.

~~~
ethomson
Indeed I did read it - my question wasn't "did you move" but "to whom did you
move".

------
forgotAgain
I've been a happy Linode customer for a long, long time. With that I have to
say the fact that they released this late on a Friday afternoon leaves a bad
taste in my mouth. That in turn leaves me with doubts about the veracity of
their statements. I'll probably be looking at alternatives now.

~~~
nsgf
Me too (for about 5 years), but this was the last straw. Seriously. Recently
moved to a dedicated kimsufi (worst support but great VFM hardware-wise).

~~~
ploxiln
I thought this kimsufi thing sounded interesting so I looked it up, and ...
wow the processors in the systems they offer are old. Where do they even get
these things? Must be leftover from OVH?

    
    
      Xeon E5504 - 45nm process, launched 2009
      Core i5-2300 - 32nm process, launched 2011, discontinued 2012
    

And the Core i5 does not support ecc memory, of course, so what kind of
servers has it been sitting in for 4 years?

~~~
snuxoll
kimsufi === OVH.

------
matt_wulfeck
> We have been working with federal authorities on these matters and their
> criminal investigations are ongoing.

I cringe when I see companies say this. As if we're supposed to feel like the
"hack" was somehow more sophisticated than spearfishing or social engineering
because there's feds on the case.

There's a hole in your security. Diligently look for that hole. If it's a
mistake own up to it fully and apologize. Make your system robust. If you
don't have the talent in your organization to do this, then hire more talented
engineers. Compete with other companies for good people.

~~~
Sir_Cmpwn
Disclaimer: Linode employee

Regardless of the severity of the means, the fact is that this sort of attack
is entirely illegal and involving law enforcement is a clear requirement.

~~~
tptacek
That's fine, but I think the parent is implying (probably correctly) that
involving law enforcement isn't really doing anything for the customers of the
service. Sure, what happened was a crime, and if the attackers are really
unlikely they could end up getting arrested in a couple years. "And?"

------
anaphor
I've been using Linode for years and I haven't had too many problems with the
service itself. That being said, this makes me think twice about staying with
them. If I wanted to switch, is the only real competitor Digital Ocean or are
there any other good choices?

~~~
ergo14
[http://oktawave.com](http://oktawave.com) and
[http://vultr.com](http://vultr.com) are said to be good. I haven't used them
though, I personally use baremetal servers at Hetzner.

~~~
eatonphil
I use vultr to host freebsd servers. Have no problems with it but it is young
and it definitely shows. Good host provider to keep an eye on.

~~~
stock_toaster
Same. I have been using it for a cheap freebsd website instance, for just over
a year now. So far, I have needed very little interaction with support, so I
can't speak to how good it is.

------
ivank
I learned to not to trust Linode after incident around 2012-01-23: I emailed
them asking if they would compile a new kernel without the /proc/pid/mem local
root exploit; they manually patched and compiled a new 3.2.1-linode40 kernel.
I booted into it, but it repeatedly locked up my VPS after ~10-12 hours of
runtime. No monitoring or automatic reboot on their side, and both lockups
happened right before I went to sleep. Did they bother notifying anybody about
their buggy kernel, which many people probably booted into? Nope. Nothing.

------
joejoebob
"Linode Security Advisory" is a very misleading title.

------
elchief
Should Linode have not stored their key in a smartcard/hardware security
module and calculated the TOTP using that?

------
ryanlol
I'm just going to leave this glassdoor review here:

[https://i.imgur.com/sJd56AT.png](https://i.imgur.com/sJd56AT.png)

~~~
jsmthrowaway
I left a Glassdoor review about Linode that was removed because I mentioned an
employee (anonymously) who rubbed his genitals on coworkers' keyboards as a
joke. This was reported to and covered up by management because the employee
was essential. Anyway, Glassdoor responded to ostensibly a Linode complaint by
removing my review several weeks after I left it. So they do watch it.

There's a lot more to the story, for sure. It's the worst of the bro culture,
or at least it was when I left. At my next employer they looked over Linode
employees as recruiting opportunities after hiring me, and came to ask me
about potential candidates, and the only one they were interested in was
genital rubber. I laughed and said I'd quit.

~~~
jabberjaws
Employee Disclaimer: It's not 2011 and Mike is not working at Linode anymore.
You really don't get what it's like working at Linode TODAY. I'm sure
everything your stating was terrible for you but it's not an accurate
representation of what the company has become.

~~~
jsmthrowaway
The same people are running the company who did then, who were responsible for
setting the culture of the company, covering for employees who did unspeakable
things (worse than what I've said here), and pretty much instructing employees
to lie to customers.

Also, all the people who quit Linode and ran to this coast after I left have
kept me very apprised of what working at Linode is like TODAY. I still
communicate with your colleagues quite regularly, too.

It is important to reiterate here, as I have before, that I wish Linode no ill
will. I'm becoming more comfortable with calling spades spades, but I do not
wish Linode to fail. I actually hope a lot of this stuff can be fixed,
whatever that entails, but I have a serious gripe with some events that have
transpired since 2011, from security to personal. If Linode would start being
a little more honest about their gaping security troubles, and not rely upon
people like me and Tim who actually know the truth to just shut up about it
(I'm getting braver as Tim does, and I appreciate how willing he is to not let
PagerDuty be tossed around by Linode's deceit), I'd be a bit happier. We're
crossing into knowingly compromising the safety of the Internet, PII, and a
number of production infrastructures that still run on Linode in some cases,
and I _do_ care about that.

And again, that tone of deceit is set from the top.

~~~
dang
> _You know that. Don 't be disingenuous._

> _And you know it._

That's unduly personal. Please remain civil.

~~~
jsmthrowaway
I mean, I was responding to a personal comment that directly and personally
told me I don't "get" things (and which included what can be reasonably
interpreted as a veiled threat by mentioning my departure year from a
throwaway account, to make clear that I'm a known quantity in the equation).
I'm really trying, here, Dan. We've talked about this over e-mail, but it's
getting really tough to contribute here with arbitrary boundaries that are
inconsistently enforced, and that a penalty remains on my account for some
comment I made in the past that doesn't even matter any more.

I bit my tongue on you detaching this subthread because I've learned that
moderation is opaque and largely not welcome to outside opinion, but I agree
with Ryan up there and suspect you detached the thread to hide where I went
with it. I'm fine with that (honest). Just wish you'd say that.

I've edited, regardless. Is that better?

~~~
dang
Yes, that's much better. You took out the personal attack, which was all that
was needed.

I don't have the least opinion about Linode or "where [you] went with it". My
concern is with civility on Hacker News. That's not an arbitrary line, though
I'd never claim we make every call correctly.

The GP seemed to me merely to be saying that the company had changed since you
left. That doesn't seem personal, nor a threat, but perhaps there are
subtleties I'm missing.

