
DigitalOcean leaks customer data between VMs - sneak
https://github.com/fog/fog/issues/2525
======
xSwag
TL;DR: In the DigitalOcean web panel you can check the "scrub data" checkbox
when destroying a VM. When using the API this option is not ticked. This can
lead to other customers being able to retrieve your data.

The author thinks that this is a security issue because this option should be
enabled by default. However, (I assume) it's not in Digital Oceans interest to
do full disk scrub because it reduces the lifespan of their SSD.

If a user forgets to log out of Facebook on a public computer, is it
Facebook's responsibility? Similarly, if a user does not correctly delete data
on a budget host, is it the hosts fault?

~~~
rgbrenner
Wait a second. Why does this option exist at all? Are you saying there are
people who DO NOT WANT their data destroyed when they destroy a VM? Who are
these people?

~~~
jd007
It's probably there for DO's sake. SSDs have limited rewrite cycles, and each
secure wipe will definitely shorten the lifespan of the drive much more than a
simple quick wipe. This translates into cost savings.

~~~
switch007
Yes. This is a shitty default peddled as a benefit for the customer, when in
fact it benefits DO.

~~~
sneak
No other provider charges you extra to erase your data when tearing down an
instance.

It's called a dark pattern.

~~~
bluefinity
Okay, the misinformation is getting a little out of hand here. DigitalOcean
certainly aren't charging you to erase your data, all you have to do is check
the "Scrub Data" checkbox. Yes, you have to pay for the time your VPS is on
while erasing, but that's perfectly reasonable.

~~~
revelation
Why would that be perfectly reasonable? They have an obligation to not make my
data available to other people, I don't give a rats ass about the technical
details of doing that.

------
neom
There are a couple of things I wanted to say, and I can speak with some
authority on the subject as I speak on behalf of DigitalOcean.

This was mentioned to me on twitter hours ago, prior to this post. The first
thing I said is that most people these days understand the importance of a
responsible disclosure, and that we take all security issues very seriously.
Not following responsible disclosure with a company such as DigitalOcean is
extremely irresponsible and I would be amiss to point that if anyone did ever
find a software vulnerability filing it and waiting 24 hours for the
appropriate response is preferred. -
[https://www.digitalocean.com/security](https://www.digitalocean.com/security)

As far as I can tell here, there is no unexpected behavior that isn't
documented or stressed. In both our API documentation, and our control panel,
we note that you must either pass a flag or check the box to security delete
the data. As far as I can tell, the flag is currently functionally correctly.
so..

Is the complaint that customer data is being leaked from VMs? That the flag
being passed via our API/Dash isn't actually working? Or, that our policy on
not doing a secure delete by default isn't something you agree with?

j.

~~~
innoying
It doesn't seem like a security issue by DO in any way. It seems that the fog
api (the project this 'issue' is filled for) doesn't allow a user to access
the required flag to scrub the drive. Not a DO problem in any way I can tell
assuming the scrub parameter is working correctly.

~~~
nixgeek
This is a well-documented situation that almost every provider of 'consumable
infrastructure' before DigitalOcean came along has faced and solved.

It is disheartening to see the same mistakes being made.

Whilst I absolutely see their USP was always solid-state storage, and that has
pitfalls in terms of how you can scrub data to avoid it being leaked, the
platform should take every precaution to protect customers data.

There shouldn't be an option to 'scrub data' and it shouldn't be defaulted to
off so they can save some hassle, and avoid spending a few dollars. It
shouldn't be an option because it should be on all the time, anything else is
surprising the customer mightily.

"What do you mean, my data leaked? Oh that's fine" -Nobody

"Why'd you pick a provider who doesn't take our companies information security
seriously?" -Every boss anywhere

------
tachion
This reminds me my own story: few weeks ago I was trying out their service and
on newly created droplet I've noticed a... shell history of downloading and
executing a shell script:

    
    
        1  clear
        2  ls
        3  clear
        4  wget https://kmlnsr.me/cleanimage.sh
        5  rm cleanimage.sh
        6  cd /tmp/
        7  wget https://kmlnsr.me/cleanimage.sh
        8  chmod +x cleanimage.sh
        9  ./cleanimage.sh
    

This looked very disturbing, so I went and check what that script is, and it
is available to read for everyone, and seems to be a part of their
provisioning procedure for the vm's, written by some guy who works for
DigitalOcean as 'Community Organizer' (however, at that point I thought the
website might be created by an attacker and misleading).

Not only it looks bad and alarming to customers, but also poses a security
threat, where an attacker could target his website and/or server and replace
the script with something nasty inside. How long before they'd notice such
fact? No idea, but I've opened a ticket about it right on, giving them some
advice on why its bad (availability, scaling, performance, security and PR
reasons) but also how to better handle it, and it seems nothing has been done
about it so far.

That rings a bell in my head not to use Digital Ocean service as things they
do are looking pretty amateur.

~~~
coolj
To be honest, this looks like an artifact of the base image creation process.

"This file is used to clean up traces from DigitalOcean images before being
published."

I don't think someone actually logged in and ran those commands on your
instance. I could be wrong, but I'd bet this is just from sloppy creation of
the base image leaving weird stuff in history after the image was published.

~~~
tachion
I have not said that someone does it manually, I only said, that on first
look, that looks really suspicious. And, at the end, the support told me that:

"This is Kamal Nasser's script that has been set up to run on the images. The
cleanimage.sh script sometimes doesn't clear the history. Thank you for
bringing this to our attention. I've brought this to his attention.

There is nothing to worry about with this."

So in fact, it seems like it is being used, instead of being a leftover in
shell history. In addition to that, I was later answered to the same ticket,
by different support member that this script is not being used and just sits
on that web page, but it all looks really bad in terms of professionalism.

~~~
coolj
Agreed. I wasn't defending DO's professionalism there, just saying that from
my experience creating base images for a (different) cloud provider, it looks
like an artifact of image creation rather than a real cause for alarm (I would
certainly be alarmed if random people were logging in to my instance and
running random scripts from the web!).

------
sneak
Oh, hey guys, they've responded. It's no big deal, they just disabled the
security because because _users were complaining_.

Turns out it "add[s] a very large time to delete events" when you actually
delete things when a user makes an api call to DESTROY. Who knew?

[http://i.imgur.com/MFW8ng6.png](http://i.imgur.com/MFW8ng6.png)

~~~
jlawer
Damn straight.

If I have something important on a VM it is purged before issuing a destroy. I
will overwrite the block device myself so I know it is empty.

Why should I have to pay for DO to do the same again just because someone
can't be bothered to read the manual.

~~~
AaronFriel
As I replied to you elsewhere, I feel your comment here may lead people to
believe that it would cost DigitalOcean money to correct this behavior.

This is wrong, it is costing DigitalOcean money not to fix it - in terms of
SSD lifetime, not TRIMing those blocks increases fragmentation of the internal
physical layout of the pages of flash memory. The behavior of DigitalOcean's
virtual machines has surprisingly managed to achieved the worst possible
outcome. Their hardware is being misused and their customer's data is being
mishandled.

------
nbpoole
Interesting: this sounds like a recurrence of the same issue which was
described a number of months back:

[https://www.digitalocean.com/blog_posts/resolved-lvm-data-
is...](https://www.digitalocean.com/blog_posts/resolved-lvm-data-issue)

At the time, the blog post claimed that the issue was resolved and that data
was now being wiped by default. I wonder why that would have changed.

~~~
sneak
I'm the one who reported it. I was able to recover someone else's web logs
from 29 December 2013 on a VM an hour old.

------
jlawer
Talk about a link bait title. Its a bit hard to call it a leak,Its a
configuration option that is well presented in the web UI. It is optional as
it adds ~ 10 minutes of billing to the small 512mb VMs and as such it is
optional if you do it.

If your using an overlay or API on top of a cloud or service, its the
overlay's responsibility to ensure a consistency with your expectations. The
API is consistent with the UI.

While other cloud providers accept the time that this takes as non-billable,
DO don't. By getting higher utilization is how they are able to offer their
prices and still have some modicum of service.

~~~
sneak
Let's be clear about what this is: they are charging their customers after
their customers have deactivated a service (destroyed a VM) to not create the
situation wherein they give that data to another customer later.

What sort of mental gymnastics are required to make that a reasonable choice?

~~~
Dylan16807
>they are charging their customers after their customers have deactivated a
service (destroyed a VM)

They are charging their customers for the number of minutes it takes to safely
destroy the VM. This is not a charge for something coming 'after'. It's
fundamentally a charge for their actual server use, not a bonus fee.

>What sort of mental gymnastics are required to make that a reasonable choice?

They aren't charging for security, they are giving you the option to buy less
server time if you don't need security, or handle it yourself by wiping only
the sensitive files. There are no mental gymnastics here.

~~~
krallin
I think you hit the nail on the head here: offering the option to buy less
server time if you don't need to wipe data is probably reasonable.

Now, the problem here is that DO turned that choice around, and are therefore
not providing security by default, but offering you the option to pay more to
get it.

Additionally, this is poorly advertised (the API docs do not clearly state
"Your data may be accessible by other users!"), and that explains why many
customers are (reasonably) a bit pissed at DO.

~~~
Dylan16807
Yeah, they screwed up the default via the API, but the choice is a reasonable
one to have.

------
PhantomGremlin
Anyone who's tempted to use "the cloud" for anything sensitive should first be
forced to write out, at least 5000 times, in longhand:

"The cloud. Somebody else's computer".

I think cloud computing is great for the right applications, as long as people
understand the risks.

But there will always be problems like this. Always. This is part of the
hidden cost of "simple cloud hosting".

~~~
orblivion
What about complicated cloud hosting? Rackspace is still "Somebody else's
computer".

------
comice
Since day one, Amazon EC2 used a copy on write system with their LVM volumes
to protect against this problem (without them having to do expensive zeroing
operations).

This has been an identified and solved problem for YEARS. No excuse for a
modern VPS/IaaS provider to be leaking customer data in this way, except
incompetence.

~~~
zimbatm
Amazon shifts the costs of zeroing on the user of the new VM. Because the
blocks are zeroes on first access writes are twice as slow on the first pass.
You can try that out easily by running `dd if=/dev/zero of=/mnt/test` twice.

~~~
rpedela
Yeah, but that isn't a big deal especially when you consider the alternative:
leaked data.

------
AaronFriel
This is a huge problem and there seems to be a good deal of misinformation
about this issue that has confused things. I'm going to debunk two things:
first, that DigitalOcean is not violating user expectations (they are), and
second, that doing this correctly is difficult (it isn't). The tl;dr is that
if DigitalOcean is doing this, they are not using their hardware correctly.

First, it's not uncommon for virtual disk formats to be logically zeroed even
when they are physically not. For example, when you create a sparse virtual
disk and it appears to be XGB all zeroed and ready to use. Of course, it's
not. And this doesn't just apply to virtual disks, such techniques are also
used by operating systems when freeing pages of memory - when a page of memory
is no longer being used, why zero it right away? Delaying activities until
necessary is common and typically built in. Linux does this, Windows does it
[[http://stackoverflow.com/questions/18385556/does-windows-
cle...](http://stackoverflow.com/questions/18385556/does-windows-clear-memory-
pages)], and even SSDs do it under the hood. For virtual hard disk technology,
Hyper-V VHDs do it, VMWare VMDKs do it, sparse KVM disk image files do it.
Zeroed data is the default, the expectation for most platforms. Protected,
virtual memory based operating systems will never serve your process data from
other processes even if they wait until the last possible moment. AWS will
never serve you other customer's data, Azure won't, and none of the major
hypervisors will default to it. The exception to this is when a whole disk or
logical device is assigned to a VM, in which case it's usually used verbatim.

This brings me to the second issue. Because using a logical device may be what
DigitalOcean is doing, it's been asked if it's hard for them to fix it. To
answer that in a word: No. In a slightly longer word: BLKDISCARD. Or for
Windows and Mac OS X users, TRIM. It takes seconds to execute TRIM commands on
hundreds of gigabytes of data because, at a low level, the operating system is
telling the SSD "everything between LBA X and LBA X+Y is garbage." Trimming
even an SSD with a heavily fragmented filesystem takes only a matter of
seconds because the commands to send to the firmware of the SSD are very
simple, very low bandwidth. The SSD firmware then marks those pages as "free"
and will typically defer zeroing them until use. Not only should DigitalOcean
be doing this to protect customer data, but they should be doing it to ensure
the longevity of their SSDs. Zeroing an SSD is a costly behavior that, if not
detected by the firmware, will harm the longevity of the SSD by dirtying its
internal pages and its page cache. Not to mention the performance impact for
any other VMs that could be resident on the same hardware as the host has to
send 10s of gigabytes of zeroes to the physical device.

Not only is DigitalOcean sacrificing the safety of user's data, but they're
harming the longevity of their SSDs by failing to properly run TRIM commands
to clean up after their users. It hurts their reputation to have blog posts
like this go up, and it hurts their bottom line when they misuse their
hardware.

Edit: As __RWG __points out, not all SSDs will read zeroes after a TRIM
command, so other techniques may be necessary to ensure the safety of customer
data.

~~~
rwg
In your second paragraph, you're conflating two different things. File-based
disk images don't leak data when they're deleted because the filesystems those
images live on ensure that (non-privileged) users can't get at data from
deleted files. Sparse images can be smaller than the data they contain
because...well, they're sparse images. They're files with holes in them, and
the filesystem automagically turns those holes into zeroes on read.

Now, about Trim... Trim is only an advisory command. You tell the disk, "I'm
not using these LBAs anymore, so feel free to do whatever with them." The disk
has the option to completely ignore your Trim command, and even if it does
mark those LBAs as unused in whatever LBA->NAND mapping table it uses
internally, the disk can also continue returning the old data on reads of
those LBAs if it wants to. There _are_ disks that make the guarantee that
Trim'd LBAs will always read back zeroes until written again (an ATA feature
called Read Zero After Trim), but I'm guessing DigitalOcean isn't using SSDs
that support RZAT since that's generally only found on more expensive SSDs,
like Intel's DC S3700.

What I'm getting at is that Trim isn't guaranteed to do what you think it
does. Unless the disk supports RZAT, the only way you can guarantee that the
disk won't return old data in response to a read command is to write zeroes
over that block.

If you're a VM provider and can't count on Trim doing what you want it to
(reading back zeroes on Trim'd LBAs) because your drives don't support RZAT
and you don't want zero out partitions at creation or destruction time, the
right thing to do is encrypt every partition with its own randomly generated
key at creation, then destroy the key when the partition is destroyed. Users
will see random data soup on their shiny new block devices, which isn't as
nice as seeing a zeroed out block device but is still nicer than seeing some
other user's raw data. (Also note that doing this doesn't stop you from also
issuing a Trim for a partition when destroying it so the SSD gains some
breathing room.)

~~~
AaronFriel
You're absolutely right and thank you for the clarification. I didn't intend
to conflate sparse files with file-based disk images, but I was trying to
convey that there can be a difference between the logical data of a disk image
and the physical data, and that deferred zeroing is the default and the
expectation of developers and sysadmins. Images can be sparse and/or file-
based, as the features are orthogonal, if cross-cutting.

More importantly, you clarify that RZAT is a necessary feature for what I'm
mentioning to work properly. You're right. They should both be ensuring the
blocks served to customer VMs are zeroed on use _and_ ensure that they are
appropriately running TRIM commands to ensure maximum performance from their
hardware. Not all SSDs perform RZAT, and it wouldn't be a bad idea for the
host to ensure the device is logically zeroed for the VM anyway.

DigitalOcean could easily switch to doing both, or at least guaranteeing the
former by creating new logical disks for customers as every other vendor does.
If, as they have blogged about in the past, they are directly mapping
virtualized disks to the host's LVM volumes, they are unnecessarily
complicating their hosting set up and making their host configuration more
brittle. With thin-provisioned/sparsely-allocated or with file-based virtual
disk images, they can more flexibly deploy VMs with different disk sizes with
minimal changes in host configuration.

Alternatively they could trivially ensure that even forensic tools would have
a very difficult time erasing volumes by enabling dm-crypt on top of LVM, and
resetting the key every time a virtual machine is deleted. This could reduce
performance on some SSDs (particularly SandForce based models) but would allow
minimal changes to their configuration to ensure deleted data is
unrecoverable.

~~~
rwg
Using 1:1 mappings of LVM logical volumes to guest VM block devices is the
most straightforward and performant method of doing it on Linux, short of
doing 1:1 mappings of entire disks or disk partitions to guest VM block
devices. While using file-based disk images would prevent data leaks between
customers without any further effort required on the VM provider's part
(assuming they don't reuse disk images between customers!), there are tons of
downsides to file-based disk images, mostly related to performance and write
amplification.

I don't agree that file-based disk images are more flexible than LVM's logical
volumes — it's ridiculously easy to create, destroy, resize, and snapshot LVs.

~~~
regularfry
Until very recently there were serious problems with putting LVM under any
sort of concurrent load. Making more than a few snapshots at the same time,
for instance, was asking for trouble. I say "was" \- I've got no idea if these
problems were fixed. You just don't have those problems with file-based
images.

~~~
justincormack
Yeah but you can't snapshot a file based image, so lvm without snapshots is
just as good (and much faster).

~~~
regularfry
ZFS and btrfs both let you do this, as do qemu COW images.

------
sillysaurus2
There is a simple solution to this: don't trust providers to do what they say
they'll do with your data. You should scrub any drive that's ever contained
sensitive info before you throw it away, and terminating a VM instance is
precisely equivalent to handing the VM's harddrive to your provider.

It's pretty easy nowadays to scrub a drive. Writing zeroes would suffice.

Personally, I'd worry more about what data is being leaked when your VM is
paged to disk on your provider's servers. Parts of each of your VMs will
probably reside in the pagefile at some point, so therefore writing zeroes
won't save you if the provider has bad disposal practices (like not scrubbing
before disposal). So it seems impossible not to have to trust a cloud
computing provider whatsoever; some basic trust seems to be a requirement.

But that minimum level of trust should be the extent to which you trust them.
Not scrubbing your drive before handing it over is placing faith where faith
doesn't belong.

~~~
sdfjkl
You may not always have the chance to scrub it yourself, for example when your
VM has a hardware issue.

------
peterwwillis
FWIW, this kind of problem is minor in comparison to the potential exploits
that have and will continue to crop up in shared computing environments.

[https://www.cs.cornell.edu/courses/cs6460/2011sp/papers/clou...](https://www.cs.cornell.edu/courses/cs6460/2011sp/papers/cloudsec-
ccs09.pdf)

[https://www.informationweek.com/security/risk-
management/new...](https://www.informationweek.com/security/risk-
management/new-virtualization-vulnerability-allows-escape-to-hypervisor-
attacks/d/d-id/1104823)?

[http://www.insinuator.net/2013/05/analysis-of-hypervisor-
bre...](http://www.insinuator.net/2013/05/analysis-of-hypervisor-breakouts/)

[http://slashdot.org/story/12/03/02/0059202/linode-exploit-
ca...](http://slashdot.org/story/12/03/02/0059202/linode-exploit-caused-theft-
of-thousands-of-bitcoins)

[http://thoughtsoncloud.com/index.php/2013/07/how-your-
data-l...](http://thoughtsoncloud.com/index.php/2013/07/how-your-data-leaks-
from-a-virtual-machine/)

I was trying to find any cases of a public cloud provider's customer data
being leaked or easily visible on the internal customer network, but didn't
come up with anything. Somebody's got to do a study on the major cloud
providers and see if the good old methods to subvert network routes still
works, or if you can easily mitm vm neighbors. (My guess is you can...)

------
cordite
Just a random idea from an ignorant individual..

What if DO actually encrypted the SSD space with a key that they only have,
and a new key is created for each droplet?

Then any droplets that are created later in a deleted space will just see
effectively random data, no?

~~~
thirsteh
You can put the key on the disk itself too--you just need to zero out the
space containing the key when you're done. That's how phones with encryption
and remote wipe features usually do it.

In fact, a lot of SSDs, e.g. Samsung's, already work this way, transparently
AES-encrypting data before it is written. (With AES in hardware, the overhead
is negligible, even for high-performance drives.) The "Encryption" feature
they advertise actually just lets you set your own key for the encryption key
container--it happens either way.

~~~
serf
Intel FDE, as far as I know, also works this way.

The key is protected. In the event that the drive must be re-provisioned (like
in the case of a lost password), the decryption key is simply overwritten by
the new key, rendering the original data unreadable.

------
nwh
It's almost public knowledge as there's an option in the GUI called "scrub
data", the absence of a tick there implies that it's not going to be erased
before the VMs partition is reassigned. I had a chat to the support months
back about "erase data" not being in the API at the time, and the solution I
came to was just to ditch their API and go back to scraping their forms for
the option.

That said, this would probably go down better for the company and the
community if you tried a private disclosure rather than posting about it on
Github.

[https://www.digitalocean.com/security](https://www.digitalocean.com/security)

~~~
1945
I've been a DO customer for about a year and some, this is news to me. So no,
I wouldn't say it's public knowledge.

That said, I haven't really destroyed many of my VMs.

~~~
nwh
[http://i.imgur.com/Aliyewf.png](http://i.imgur.com/Aliyewf.png)

I suppose when I made that comment I was expecting everybody to be like me;
eager to flip switches to see what they do.

~~~
djur
I would expect that option to do something along the lines of scrubbing the
data on the hardware level so that someone who physically examined the disk
wouldn't be able to access my data. That's because I'd expect my data to be at
least logically wiped in the first place.

~~~
bsaul
Same here

------
threeseed
If the "securely scrub" parameter has ANY kind of performance impact which I
suspect it does then isn't it a good thing that DO gave their users the choice
? There are plenty of use cases where I would trade privacy/security for
performance.

Anyway disingenuous title to say the least.

~~~
mikeash
It is good to give users the choice. It is bad to default to the dangerous
choice. It is even worse to not describe the bad consequences of the default
anywhere.

Nowhere in the DO UI for destroying a droplet does it indicate that they will
leak all of your data to the next customer if you don't check the box.

------
kolev
This is bad for Digital Ocean as the checkbox is nothing but a legal excuse to
let beginners shot themselves in the foot and for a new company, this isn't a
situation you want to be into. Look at Amazon - how do they solve the same
problem? Is it harder for them? Is there a checkbox? When you compete with
Amazon you have to be that and better, not worse, and treat beginners better
than experts!

~~~
cones688
Amazon doesn't do SSD VMs for $5 a month... If you are beginner you are using
the Web GUI which clearly states you need to click the box to securely wipe
all the data (which has a cost associated with it) if you are using the API
then you should RTFM (Read the F Manual) if you build a service on top of the
API then it is the service's fault for not including (which is what the gist
is about)

~~~
owenmarshall
From above: [http://i.imgur.com/Aliyewf.png](http://i.imgur.com/Aliyewf.png)

What that poorly worded checkbox says to me: "tick this box if you want to
prevent DigitalOcean from reading your stuff."

Nothing in that web GUI "clearly states" that not ticking the box will allow
the next random user to read my files.

------
bachback
New response [https://digitalocean.com/blog_posts/transparency-
regarding-d...](https://digitalocean.com/blog_posts/transparency-regarding-
data-security)

I like DO as a service, but this is kind of strange. Humans act always the
same. When catastrophe hits they want to sit it out, underestimating the
impact

------
bsaul
Please DO, do something brave and make the default behavior sane ( there is
absolutely no way i would except another customer to get my data with a one
line shell script, ever). I LOVE your service, so dont screw it PLEASE !!

------
mendelk
DO released a blog post on the issue[0]

[0] [https://www.digitalocean.com/blog_posts/transparency-
regardi...](https://www.digitalocean.com/blog_posts/transparency-regarding-
data-security)

------
0x0
It would be interesting to see a survey of the various cloud VM services; if
any of them will return non-zero (data) blocks for uninitialized storage.

Sounds like a major risk if SSH, SSL, passwords etc can leak this easily.

~~~
coolj
Also, even if they zero the primary block device, what about swap if it's
provided as a separate block device?

~~~
jlawer
DO provide a single block device (so swap is on the same block device), that
gets zeroed if you set the scrub data option.

------
mark_lee
I just moved one of my apps from linode to DO several days ago, some
explanations here just blowed my head off.

------
nphase
I have created a uservoice issue to scrub by default, please vote:
[http://digitalocean.uservoice.com/forums/136585-digitalocean...](http://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/5280685-scrub-
vms-on-delete-by-default)

------
k3oni
Onapp has the same issue by default, it's not set by default to zero fill the
old disks are being destroyed. This need to specifically be enabled in the
config file by the administrators.

In case you are running a VM on top of their platform you may want to check to
make sure this is enabled.

------
patrickg_zill
Is it the same issue, or a different one, from this one back in April of 2013?

[http://www.wired.com/wiredenterprise/2013/04/digitalocean/?c...](http://www.wired.com/wiredenterprise/2013/04/digitalocean/?cid=co6834744)

------
bachback
for those wondering, how that compares to AWS. AWS sec policies
[http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper...](http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper.pdf)

------
yeukhon
Hmm have you done this with private disclosure? If not, please do that next
time. It is unethical to disclose security-related issue of some PaaS publicly
without first going through the responsible party first. Even if a security
issue you want to report is actually something only the owner can see... I
don't know what this do because I don't really use DO, but just judging the
way it is reported, it looks like you haven't...

~~~
nknighthb
Even if one bought the ridiculous notion that private disclosure of
vulnerabilities is a good thing, why would you think it would be necessary to
privately "disclose" to anyone what Digital Ocean already publicly documents
in their API?

~~~
yeukhon
[edit]: okay I misunderstood the issue here.

Some day I would forget to close my home door with a bang. No home invasion
yet because nobody notice that the door wasn't fully closed. Now I am telling
people I will forget to close my door, and I will get an invasion.

Some people's repo have password committed. It only takes someone to google
that to find out. If someone posts that on HN, yeah, it's on the web anyway,
it only takes one person to make my password known instantly.

~~~
nknighthb
You are extremely confused. Digital Ocean does this deliberately and already
publicly documents it. It is intentional.

This is not a bug report to Digital Ocean or any other PaaS. It is a request
for a third-party library to support an option Digital Ocean's API provides.

~~~
yeukhon
[edit]: okay I misunderstood the issue here.

------
norlowski
Linkbait. Title should be "Digital Ocean API is not told to scrub (securely
delete) VM on destroy"

~~~
mikeash
"Don't scrub" does not imply "give the data to the next guy".

If I delete a file on my computer without doing a secure delete, I know that
the data is still on the disk and can still be recovered. However, I _also_
know that in _normal_ operation of the computer, that data will never show up
again. There are no circumstances where I can create a new file and have it
get filled out with the contents of the deleted file. There are _certainly_ no
circumstances where another user on my computer can do that and get my data.

I would have every expectation that this scrub option works the same way. That
it defends against specialized recovery efforts, not random people making new
VMs. DO's documentation says nothing to indicate otherwise.

------
jnankin
what i dont understand is, why would any one want to START with a VM that has
some one elses data on it? Forget people wanting their data scrubbed on
delete.

------
dpacmittal
This makes me wonder if AWS secure on this front?

~~~
tszming
Customer instances have no access to raw disk devices, but instead are
presented with virtualized disks. The AWS proprietary disk virtualization
layer automatically resets every block of storage used by the customer, so
that one customer’s data are never unintentionally exposed to another. AWS
recommends customers further protect their data using appropriate means. One
common solution is to run an encrypted file system on top of the virtualized
disk device.

[1]
[http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper...](http://awsmedia.s3.amazonaws.com/pdf/AWS_Security_Whitepaper.pdf)

------
kylek
have you gone to DO directly with this first?

~~~
sneak
According to them, it's either a bug they already fixed in April, or leaking
data today by design?

[https://www.digitalocean.com/blog_posts/resolved-lvm-data-
is...](https://www.digitalocean.com/blog_posts/resolved-lvm-data-issue)

[https://twitter.com/jedgar/status/417515181418479616](https://twitter.com/jedgar/status/417515181418479616)

~~~
coolj
I don't really see how this is leaking data. I can use the dropbox API to make
a bunch of files with private data shared with the world. But dropbox isn't
"leaking" my data, I'm using the API in such a way that makes my data
accessible. Not an exact analogy, and I would agree that the option should be
on by default so people who know what they're doing can opt-out and everyone
else gets a safer default, but this isn't a "data leak."

~~~
sneak
It absolutely is a data leak.

I spun up a vm and ran "strings" on the blockdev and got this:

[http://i.imgur.com/fJOxRN9.png](http://i.imgur.com/fJOxRN9.png)

Some poor iPhone users in Portugal have no idea that the app they're using is
backed by a webservice on a VM that gives its block storage contents to anyone
who gives Digital Ocean a $5 PayPal payment.

If that isn't a data leak, I don't know what is.

~~~
coolj
If you make an API call that asks for your data NOT to be scrubbed, then it's
not a leak that your data isn't scrubbed--you asked for it. If you haven't
read the docs, you might not be aware that you're asking for it. That's a Bad
Thing. No question. It should be enabled by default, to prevent unknowing
users from leaking their own data. If you ask for a scrub and you can still
find data on the scrubbed block device, then you have a leak from DO.

~~~
sneak
I'd agree with you, except that in this case the API call is called "destroy".
Were it called "deallocate", this would be a different story.

------
harkness0310
lol of course.

------
bliss_kiss
these problems cannot be avoided

------
godyo
leaks leaks leaks. no good!

