

Cloud providers found to be leaking data cross-customer via dirty disks - marshray
http://www.contextis.co.uk/research/blog/dirtydisks/

======
ben1040
Slicehost (now under Rackspace) sent a very cryptic email a few weeks ago
saying users needed to "migrate" their servers, but gave no reason why. Then
this mail went out on Friday:

 _We are writing to update you on the migration you recently completed, and to
provide more information about the reasons it was necessary. We know that
migrations can be inconvenient, and we thank you for your patience. Now that
the migrations are complete, there is nothing more that you need to do
regarding this issue.

When we announced the recent migrations, we explained that such measures are
periodically required to promote the stability, performance, security, and
feature-richness of our Slicehost platform. We were not able to share more
information at the time, without putting you and other customers at risk. Now
that the migrations have been completed, however, we want to provide you with
the transparency that you expect from Slicehost. We now can tell you the
timing of the migrations was driven by the need to fix a potential security
issue.

We discovered the issue in collaboration with an independent I.T. security
consulting firm that tested our products. The security consultants used
forensic techniques to examine the underlying physical disk in newly created
instances. They discovered that, in certain use cases, random fragments of
temporarily stored data could be left behind on the physical disk.

In repairing this vulnerability, we have ensured that all data is wiped
effectively whenever a customer vacates hard-drive space on a host machine.
And through the migration that you and other customers have completed, we have
cleaned up all fragments of remnant data. The security consulting firm that
discovered this issue has performed follow-up testing and has found no remnant
data on our environment.

We know of no case of customer data being seen or exploited in any way by any
unauthorized party.

One reason is that the remnant data could not have been seen through normal
use of slices, but would have had to be sought, using forensic techniques. It
was not possible for anyone to specifically target a particular customer
through this vulnerability, given the random and fragmented nature of the
remnant data. Customers who encrypted sensitive data on slices would have
faced no risk of exposure.

If we had made this issue public earlier, we could have opened the door for a
malicious user to exploit the vulnerability. For that reason, we decided to
keep information about the vulnerability within our company — until now, when
the issue has been fully resolved.

Dealing with security issues is a constant in any type of computing or cloud-
hosting provider. At Slicehost, we work to provide you with the safest, most-
stable environment possible. We regularly consult with independent security
consultants. We employ a large and growing staff of security specialists and
IT engineers. We are proud of their work in repairing this vulnerability, and
grateful for your patience.

If you have questions, please reach out to your support team. We are here to
serve you.

Slicehost Support Support@Slicehost.com_

------
ars
Summary:

The problem: When provisioning a disk, the provider reuses an old disk that
has old customer data on it. Part of the disk is overwritten with the OS image
but the rest is not.

The fix: Zero out the disk. Either when de-provisioning it, or when
provisioning it.

The author's ability to turn 49 words into 2,543 words is quite impressive!

~~~
politician
Also: we couldn't actually tell you this until now because we don't trust some
of you.

------
tptacek
One thing to consider here is keeping sensitive information on virtualized
hosting setups block-level (full-disk) encrypted, for instance with Truecrypt.

~~~
drivebyacct2
Obviously it might seem silly to make this comment here, but would this really
protect in many cases besides this one? It seems like almost every time the
intrusion would occur somewhere where the intruder will have access to the
mounted encrypted disk, no?

~~~
tptacek
Full disk encryption is unlikely to protect you from anything other than
custody issues at your hosting provider.

~~~
lunixbochs
The idea behind this exploit:

The area on the SAN occupied by your disk contained a simple disk image. When
your image is deleted and another is created overlapping the same spot on the
host storage, they can look at their free space to see the contents of your
disk.

If your disk happened to be encrypted, their free space would appear to
contain garbage data.

Encrypting your disk would do you little good versus your provider if your VM
can boot without your intervention... someone could just boot your VM to look
inside.

~~~
tptacek
Nothing you can reasonably do protects against a malicious (or compromised)
provider; these are all controls that protect against provider mistakes.

------
chuhnk
A very interesting article about data leakage in cloud resource allocation.
I'm impressed by the steps taken by all parties involved. Security
vulnerabilities will continue to creep up over time as they do in most
systems/services and the seriousness with which providers treat them is ever
more important. What we have to remember is it's not just individuals hosting
their own content that are at risk. More and more startups are utilizing cloud
infrastructure which means we as consumers must be aware of where our personal
data is going. I would love to see some sort of auditory process in place that
verifies the security of cloud providers regularly and we as consumers having
access to this information.

------
jamieb
Yay LUKS/dm-crypt kernel level disc encryption.

We had need for short-term (several hours) EC2 compute nodes with sensitive
data from customers. They'd boot up and encrypt all the drives using a random
key without persisting it anywhere. Even so much as rebooting them made any
data unusable. YMMV =) I'm thinking one could set the key using a start-up
file so reboot would be possible.

------
rachelbythebay
This is not just limited to "cloud" services. Hosting companies recycle hard
drives all the time. If you're lucky, they'll wipe it after they pull it out
of your machine. If you're not, they won't.

It all comes down to what kind of recycling policies they have for old
hardware. Ideally, all drives should be wiped and then stress-tested to avoid
the problem where you eventually wind up with nothing but bad disks in the
recycle loop. After all, _good_ disks stay in production and thus exit the
loop, whereas bad disks fail and come right back.

------
spullara
Why did this article so carefully not mention EC2 in the positive or negative?

------
jsprinkles
This isn't "new" information, and many providers already wipe before
reprovisioning space. Off the top of my head, I'm aware of three. The paper's
sample size is frightfully small (VPS.NET, in particular, is not a shining
star of cloud hosting). All you have to do to find out if this is the case is
run this on new storage that you've requested from your provider:

    
    
        strings /dev/<whatever>
    

This has nothing to do with "cloud", aside from rapidly reusing disks in that
environment. This is the same problem with reusing disks in a datacenter,
reusing disks at a dedicated host, reusing storage even inside a corporation,
and so on. There is nothing "cloud-specific" about this in the slightest.

Of course, they wait until near the bottom of the paper to tell you that, so
the title of the paper seems like a needless cloud scare which is really a
shared-environment problem. Clueful people in hosting figured this out a while
ago (publicly-used hosting has been around a long, long time). Long enough ago
that you can measure how clueful your provider is by the usefulness of this
trick.

~~~
SoftwareMaven
The cloud angle is important because, as a cloud user, I have no control over
the situation. More so, most people just expect the provider to take care of
them as a result.

Regardless, it's why I'm willing to pay a little more for a company that will
be proactive.

------
drivebyacct2
At first I thought this was going to be the issue where people were sharing
their EC2 instances and were leaving their keys or un-wiped copies of their
private keys or tokens on images that others than stumble upon.

