
Linode Manager Security Incident - ErneX
http://status.linode.com/2012/03/manager-security-incident.html
======
VonLipwig
The message seems to be...

Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done.

It seems to me the truth is the attacker looked for bitcoin wallets and
emptied them. The fact he could identify 8 accounts and access them suggests
the attacker could have accessed far more accounts if they wished. I think
this is the most worrying thing about the breach.

I don't really understand how bitcoin works but it seems that people with
wallets need to set up multiple wallets on multiple providers and limit the
amount of bit coins in each wallet to limit any losses from breaches like
this.

If I was a linode customer I would be thinking about moving. This message,
while fairly open, doesn't give me much confidence there aren't other security
issues with the platform.

~~~
drostie
Just imagine that BitCoin is like having cash in your wallet, because that's
more or less its intended model. There are a lot of 'anti-counterfeiting'
measures because computers are very good at copying, and you don't want people
to be able to copy BitCoins the way they can copy music -- and when you ask
"what is BitCoin?" people basically start to tell you about the
anticounterfeiting technology, and the limits on printing uncontrollable
amounts of money. But it's essentially stamped paper in your wallet in any
other sense, worth whatever people using it on the Internet will pay for it,
not backed by anything in particular but its usefulness.

Basically a lot of people were renting storage rooms in an apartment complex
run by Linode, you get your own key to enter the door and retrieve and store
things -- whatever. Some people left their wallets inside these buildings,
with cash therein. Someone else used some unidentified systematic security
flaw, but we don't yet know what it was. Maybe there is a ventilation system
which is easily navigable once you know how to get in; or maybe all of the
rooms have unlocked windows for no good reason; we haven't been told yet.
(There are some suggestions that they stole a key from one of the janitors who
cleans these rooms up.)

What we have been told is that some burglar stole eight wallets, and that "All
activity by the intruder was limited to a total of eight customers, all of
which had references to 'bitcoin'." That suggests that the burglar did indeed
peek in the windows beforehand somehow, to find out that these 8 rooms had
wallets inside. Otherwise, presumably they would say something like, "The
intruder broke into many of our customers' accounts but didn't actually do
anything in 99% of cases." In that sense I think the scary bit isn't that he
accessed the 8 accounts, it is the fact that he identified them in the first
place.

Amortizing the loss across many points of failure may be a good idea, but it
wouldn't seem to solve the central problem. Suppose I put $20 in two accounts
with 5% chance of compromise, rather than $40 in one account with 5% chance of
compromise -- either way, I should expect to lose $2. What I've changed is
that I am more likely to lose some of my money (9.75%), but I am less likely
to lose all of my money (0.25%). This may appeal more to risk-averse people
but it is not fundamentally changing the situation.

Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's
pretty simple to do in day-to-day life. This is something that you can't do
with your wallet -- you cannot turn your wallet into a steel vault with two-
foot-thick walls.

~~~
lambda
> In that sense I think the scary bit isn't that he accessed the 8 accounts,
> it is the fact that he identified them in the first place.

This isn't all that surprising. There are basically two reasons why you would
have a Bitcoin wallet on a server: if you are mining using the CPU power of
that machine, or if you need to send Bitcoins from an online application. For
example, one of the people who mentioned having coins stolen was from a mining
pool; you need some automated system to pay out the earnings to the people who
have been doing mining, and so the wallet for that automated system was on the
server, and was stolen. I suppose one further reason might be as a backup, but
in that case, I dearly hope that it's an encrypted backup without the
encryption keys in the sever.

Given these reasons for having your wallet on the server, it's not surprising
that people found them. These require network-facing services, that are easy
to trace back to the server in question. The mining pool is a public service;
anyone can join, and find the address of the server. Furthermore, when you
make payments, you announce them to the full Bitcoin network. Someone sniffing
transactions can watch where transactions originate, and target that. If they
already had a compromised customer service account on Linode, they probably
watched the Bitcoin network for a while, made note of transactions originating
from IP addresses in the Linode range, and then targeted those accounts.

One way to protect yourself from this would be to proxy your Bitcoin
transactions through a host other than the one that has the wallet,
obfuscating where the transactions are actually coming from. You could even go
so far as to make all of your transactions via Tor, which would probably make
it fairly difficult to find where your Bitcoin wallet actually lives.

> Perhaps a better approach is to keep a BitCoin wallet encrypted, since
> that's pretty simple to do in day-to-day life. This is something that you
> can't do with your wallet -- you cannot turn your wallet into a steel vault
> with two-foot-thick walls.

The problem is, if you need to make payouts from your wallet, then the machine
that does that needs to be able to decrypt the wallet. That machine can then
be compromised to be able to steal your keys. Encryption doesn't buy you all
that much, unless you are just doing a backup and don't need the machine to be
able to do online transactions at all.

Perhaps another solution would be to encrypt each key in your wallet
separately using a k-out-of-n encryption scheme (where produce n keys, any k
of which can decrypt the wallet). You can then distribute those keys to
independent hosts, which hopefully should not all be subject to the same
vulnerabilities. Then any time you do a transaction, k of those hosts will
need to produce their key to decrypt the key in your wallet and perform the
transaction. That way you would have to compromise several different,
independent hosts in order to steal the wallet.

Of course, this would drastically increase the cost and complexity of the
system; and you would need to ensure that whatever system that authorized
payments was likewise distributed, which if you had, say, a web-facing service
would be difficult.

The easiest thing to do to reduce the risk is to only leave enough value in
the wallets that are on the servers for a couple days worth of transactions.
Then you transfer Bitcoins from a more secure location once a day to keep the
coffers full. This is not much different than a physical store; yes, you are
at risk of being robbed, but if you only have one days worth of cash there,
with the rest somewhere more secure, you reduce how much risk you have.

~~~
j_s
If they had used 'unlock-at-boot'/true-crypt style disk encryption and kept
the password/key off the machine they would have been safe from this attack.
(They would also have to provide the password/key every restart.) It is only
in hindsight that something like this seems worth implementing!

Your more general solutions would protect from an attack that rooted a live
system rather than just resetting the root password while the machine was
offline.

------
orofino
I'd like to understand the what actions will be taken to prevent similar
attacks in the future. Also, what can I as a linode customer to prevent my
host from being compromised in a similar fashion.

Implementation of two factor authentication for your customers and requiring
it for a root password reset would go a ways to preventing similar attacks.

~~~
eli
Two-factor authentication for their support personnel you mean?

~~~
brown9-2
Also, is this customer service portal available via a public URL? Shouldn't
some sort of VPN access be required to even get on the network hosting these
things?

~~~
ErneX
It is public yes, this is it: <http://manager.linode.com/>

~~~
eli
I think there is some confusion here. That's the link to the customer portal,
which needs to be pretty public.

I have no idea how Linode's support team access accounts, but I would hope it
is less public.

~~~
ErneX
The title of their blog post is: "Linode Manager Security Incident" and that's
exactly the name of the customer website where you can manage your instance,
billing, etc.

I think someone found a way to gain access to any Linode customer account
through the customer website and from there shut down the instance, changed
root password and rebooted (you can do that from there).

~~~
RossM
I think it's just poorly worded - from the OP and what I've read elsewhere
someone accessed the portal used for customer service employees that has
access to options/all hosts.

~~~
davidw
In any event, it's not very clear, which adds to my confusion/worries as a
Linode customer.

------
videoappeal
All this talk about banks being safe yada yada and cloud hosting not safe for
US50k. Real banking companies (with billions of dollars on hand) do use
commodity cloud hosting including Linode, for even sensitive parts. Take for
example Natwest online banking login. On initial login page they load a cookie
via an image from www.advanced-web-analytics.com and then once you enter a
customer number the next page loads a ...drum roll... javascript file from
www.omni-traffic.com. Now who can tell me what one can do when you have
control over the Javascript on a banking login page?

Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were
hosted on conventional Linode VPSs. Points still stands though.

~~~
tomg
In my experience working on a US financial website, a bank would never
consider using a VPS like Linode to store actual banking and customer data.
It's not even close to Level 1 PCI compliant.

~~~
videoappeal
Not sure about this PCI complaint stuff, but perhaps this is why major banking
companies jumped from Linode to EC2? Much improvement? Although I must say I
have friends working on banking websites in the UK that dont know the whole
picture, its not unreasonable to assume that these things are fucked up.

~~~
tomg
From what I know, PCI compliance is firstly, just a guideline. It's not like
OSHA, but IANAL.

I also know there are "levels" of PCI compliance. The highest one, which
reputable banks should be following, is very strict AFAIK, and includes
provisions for controlling who has access to the physical hardware, encryption
levels, etc. The fact that a Linode VPS can be 'rooted' via their management
software by a sysadmin working for Linode would, from what I can tell, make
them unqualified to be used to store banking transaction & customer data,
though perhaps I am wrong.

------
glfomfn
I am failing to understand what exactly happened.

The user who was affected by the incident quoted an email from linode that
stated "Our investigation has revealed a customer support interface was used
to access your account.", based on that and all the information of that post
you get the impression that through the 'interface' the attacker was able to
change the vps root password.

Now a reply from linode comes and says "The portal does not have access to
credit card information or Linode Manager user passwords". So if the portal
doesn't have access to Linode Manager how the attacker gained ability to
change the root passwords ?

Thy should give more details on the incident, i do have a certain trust in the
ability of linode to have a secure environment & i can understand that things
like that will happen at some point to everyone. However its one thing for
someone to get access in your system because you had your roots password to
'password' and another if there was a bug that got exploited.(yea this is an
extreme example)

~~~
citricsquid
> So if the portal doesn't have access to Linode Manager

They didn't say that, they said it doesn't have access to the _passwords_.
They have an interface to change details, they just can't read them. So they
can reset your password to "hunter2" but they can't see if it's "hunter2".

~~~
glfomfn
You are right, my bad on that. Still this looks like a Public relations post
by them than giving out facts. They should be explaining what the attacker
could do by gaining access on that interface, the ability of the attacker to
change the password has the same consequences.

The point is that exploited interface had a backdoor access to the virtual
machines (to be able to change passwords or w/e)

~~~
nknight
I understand how this might be confusing to a third party, but Linode's
response thus far makes perfect sense to those of us who have been customers
for a while. We're pretty aware of the general parameters of Linode's internal
systems.

------
look_lookatme
I'm surprised that a tool with this level of access isn't walled off behind a
VPN.

------
nwmcsween
Can someone with a legal background comment on the viability of the victims
taking legal action?

~~~
keypusher
I've got to wonder too. If you rent out an apartment and someone breaks in
with a key they stole from the landlord and takes your TV, can you sue the
landlord? On the other hand, if you rent a security deposit box from a bank
and it gets broken into, is the bank liable?

~~~
mbreese
I think the threshold would be, did Linode take "reasonable" precautions in
protecting the servers in question. Just like the landlord or bank. So long as
they take reasonable precautions, they can't be held liable.

You can expect for your landlord to keep their copy of the key in some sort of
lockbox, but you aren't going to expect them to keep it in a pressure-
sensitive safe, guarded by movie-style lasers and a German Shepard.

You expect a little more security out of your bank.

The real question is: why were services that act like bitcoin banks storing
their coins on Linode in the first place.

~~~
mauriciob
They probably used Linode to _generate_ bitcoins. And they kept them there.

~~~
RKohr
No, that is extremely not likely.

The only (realistic) way to generate bitcoins today is with GPU or other
specialized hardware that doesn't exist on webservers.

These servers that were compromised were used to manage generated bitcoins.
One was used by a pooled mining service (mined coins were sent to the server
then payed out to miners) and the other was a faucet service which would give
a little bit of bitcoins to new users. The other 6 servers that were
compromised are unknown to me.

------
amalag
No mention of liability. So they are fixing it, but someone is out of $12k of
bitcoins. Linode says, tough luck for trusting us with your valuables.

~~~
chrissnell
These bitcoin servers trusted their currency exchange to cloud servers--VPSes,
really--and they want Linode to compensate them for the money lost? Insane. I
don't claim to know anything about the bitcoin infrastructure hosted here but
I'm going to go out on a limb here and say that there was no dedicated
hardware firewall in front of it, no IDS, no WAF, nothing but some Linux
instances running iptables.

The payment card industry wouldn't certify this hardware to process credit
cards for a mom-and-pop online business, yet these guys use it like their
bank? Come again?

~~~
nwmcsween
PCI-DSS is actually quite easy to get certified in. I would say the issue is
Linode not protecting their customers but as you say it's completely optional
and would incur higher costs.

------
desade
In Linode's FAQ, they mention that their virtualization management software
was developed in-house. They seem quite proud of this, quipping: "The Linode
Manager is custom software, written in house, and is not for sale (although
others have tried to mimic it)."

It seems to me that this is a classic example of security failures following
inevitably from a lack of peer review. Maybe Linode didn't consider its LM
software to be peer-reviewable, but I bet the victims of the bitcoin thefts
wish that someone else had tested the code (and human systems surrounding it)
for vulnerabilities.

Is this not exactly what Bruce Schneier frequently points out? Anything that
must withstand attacks to protect the valuables within should be tested by
attacking it. A lot. My hunch is that the vulnerability exploited by this
attacker would have been found and fixed already if the LM software were more
open.

~~~
jarito
I think you may be a little off here. The statements in the thread seem to
indicate that the compromise was not based on a vulnerability in custom
software, but compromised credentials. You can certainly argue that the
management console should be protected by two-factor (and it should be), but
their software doesn't seem to be at fault here.

I would be willing to bet that they have had the system tested by external
security contractors and scanned with automated scanning tools. This seems to
be a people problem and features problem not a vulnerability problem.

I guess we just don't know at this point. You may very well be correct. I
guess if you want to use an open source provider, just make sure they are
running OpenStack (openstack.org).

------
rdl
I really hope the various bitcoin related incidents don't poison the well for
online currencies in general. Yes, you need a greater level of security when
dealing with transferrable, relatively anonymous or pseudonymous (and rapidly
extractable) online assets that you don't need for book-entry accounting with
an audit trail and reversible transactions (credit cards, ACH, etc.). Yes,
this is beyond what even most banks currently use. No, it's not beyond current
technological state of the art.

Gaming (i.e. casinos), at least some of them, do a reasonable job with some
very similar security problems.

~~~
wmf
Wasn't the well pretty thoroughly poisoned by e-Gold and such? (We realize
they're not the same as Bitcoin, but most don't.)

~~~
rdl
Yeah, kinda. Although E-Gold's problem was force majeure, not technical (did
they ever get hacked? people stole passwords all the time, but since it was
double entry, it was pretty easy to reverse).

osGold, etc. were probably better parallels. But yes, this whole field has had
a long series of technical, business model, governance, and government related
failures.

~~~
StavrosK
Can you explain what happened with eGold, osGold, etc? I haven't heard
anything about it.

~~~
rdl
e-gold was shut down by the government for running an unlicensed money service
business. They were officially Nevis based, but actually run out of Melbourne,
FL (where they started originally); criminal charges were involved. The main
reason was a particularly motivated federal prosecutor and the specific uses
e-Gold users found -- child porn, paying for stolen cc's, etc. It wasn't a
general thing about PATRIOT/MSB laws.

osGold was just an internal fraud; digital currency, give me your (gold) and
issue tokens -- then one day it disappeared, taking the original value with
it.

There were hack attacks by third parties against some other gold currency
systems at the time, but I don't remember the details.

~~~
StavrosK
I see, thanks. Apparently the space is plagued with problems.

------
strags
While the incident is unfortunate, I do give them credit for being up-front,
honest, and relatively speedy in their response. Sad to say, I'm not sure a
lot of other hosting providers would be as quick to admit culpability.

That said, there's no way that resetting the root password should be something
a customer service rep can do. Particularly when Linode are very explicit
about not being a service for newbs - and are in general unwilling to provide
help setting up a Linux system.

------
meow
May be its time for some cloud insurance (covering outages/security etc) ? I
don't mind paying a little extra if it mitigates risk.

~~~
tptacek
Near unlimited liability and virtually zero ability to model the likelihood of
events. A hard product to design.

~~~
meow
Liability might be unlimited, but most of the insurance products for other
categories cap it based on premium. So that may not be much of a problem. I
agree with the difficulty of modelling the events though... they need more
data for any kind of modelling.. I'm guessing amazon already has some kind of
insurance to cover its risks under AWS service agreement..

------
jostmey
Hey, I'm one of those guys! But I wasn't stupid enough to leave a stockpile of
bitcoins sitting around to steal.

~~~
olefoo
I wonder about that. It seems to me that the bitcoin wallet shouldn't have
been accessible after reboot, at least until someone came by to give the agent
managing it the passphrase that would allow it to decrypt it's state. From the
sound of things the wallet was just laying around unencrypted?

~~~
ceol
[https://bitcointalk.org/index.php?topic=66979.msg778666#msg7...](https://bitcointalk.org/index.php?topic=66979.msg778666#msg778666)

According to the admin of bitcoinica, he didn't have the wallet encrypted
because the ruby gem he was using to process withdrawals didn't support it.

~~~
ashconnor
Doesn't even matter if the wallet was encrypted, he'd still have to have the
password somewhere on the server.

~~~
lmm
Not if he entered it manually on every reboot. There'd still be attacks around
snooping the contents of the wallet from memory, but the ability to do a root
password reset that involves rebooting the box wouldn't be enough (and it
seems like that may have been all the access that the attackers here had).

------
Lazare
There's already quite a lot of discussion of this incident here:
<http://news.ycombinator.com/item?id=3654110>

------
frsandstone
Well handled.

~~~
mrschwabe
Agreed. This is in sharp contrast to the Media Temple / Plesk debacle. Which
affected how many thousands of sites? How long did it take them to admit it?
Meanwhile, Linode reports full details of a hack affecting no more than 8
accounts the SAME day.

~~~
ErneX
It's good the fast acknowledgement, but I find it lacks in detail. I'd love to
know exactly what kind of measures are they going to take so the chances of
this happening again are truly reduced.

I share a Linode with a friend and at work we deal with 4 so this is a
concern.

~~~
Legion
Further comment is indeed warranted, but that part can come in a few days,
when there's more clarity on the scope of the incident and a fully-hashed plan
of attack for closing the vulnerability.

~~~
blakeperdue
I hope so. They said they will be "reviewing our policies and procedures to
prevent this from ever recurring."

That doesn't imply they will be posting a public update. As a Linode customer,
I would like to know details of how this breach occurred and what actions will
be taken to prevent it in the future.

------
shazow
Perhaps people affected by the many Bitcoin Exchange password leaks were
reusing their passwords on Linode?

~~~
frankydp
The breach originated on the Lionode customer service system. Which the
attacker used to reset user passwords. Unlikely the incidents are directly
related.

