
I lost all my data stored in DigitalOcean Spaces - nodegin
https://medium.com/@nodegin/i-lose-all-my-data-by-using-digitalocean-spaces-247888cc05ae
======
danols
They were chasing me around last week with Google Adwords for a $100 free
trial so I thought why not. Set up a droplet easy enough and invested 2-3 days
part work in a test project for fun (didn't bother with external backups..).

Then without even bothering sending me an email they disabled my account to
the likes of "We need to investigate that you are a real person, please send
in docs A, B etc.". Normally you do that at the signup stage and when you
approve someone for a trial you honour it. Now I cant even access my files.

Now been waiting 5 days to hear back from their security team after giving
them the required docs. The regular support tells me the security team are
"very busy at the moment". I guess someone decided to start offering $100
trials without thinking through the effect on the support team. Or maybe the
campaign was too successful and this is their way of cost control?

If this is how they treat a big portion of their new trial users customers I
doubt this will be a very long-term profitable campaign. I know that I will
stick with my Google Cloud account for now.

------
nodegin
Hi guys, thanks for your comments. Actually I do have backup in elsewhere so
my website is still able to run, but surely this incident taught me never use
DO services again.

~~~
betageek
Glad to hear it, I've been a big DO cheerleader and this is definitely making
me re-think things, thanks for the write-up.

------
csdreamer7
What I take greater issue with is that the author had to explain how it was
impossible for him to accidentally delete the bucket (the files in it had to
be deleted first) before DO admitted they had an issue.

I had a similar experience with moving my Ubuntu 16.04 image over to Debian.
My Debian reimaging, where it warns me all data will be destroyed for an
untouched Debian installation, had it's repos configured to use Ubuntu's, on
my Debian reimage, which of course failed. At first they blamed me, suggesting
I did something I can not remember, and then I changed the repos as they
suggested and demanded an answer when it did not fix it. They admitted their
repos have an issue that I can not solve and I have to destroy the machine
completely to be able update or install to my Debian machine and lose my IP
address that my DNS is set to.

They said this issue only happens when you reimage from Ubuntu 16.04 to
Debian. Which I believed at the time since I was tired from this. I do not
know if I believe them after the above post.

~~~
Starz0r
I remember having this same issue trying to do the same thing! Although my
setup is different, the base problem was the same. Tried to reimage from
Ubuntu to Debian only to realize their hypervisors are not setup to handle
this as it came with all the same issues you mentioned.

------
colechristensen
Lesson learned? Backups are important.

Unreliability like this should be an expected irritation, not a disaster.

~~~
rkangel
Both can be true. He should have had backups, but the service from DO was
still very bad.

------
zaarn
They key mistake here was lack of backups and backup procedures.

I wouldn't use DO Spaces or AWS S3 or others without backup, which means it's
at minimum a separated account with separated credentials if it must be on the
same provider otherwise good procedures to create the backup are sufficient.

Oh and sure, DO and AWS will promise you high and holy that the data will
never be lost (99.99999999999999% permanence or whatever they write down). But
no promise can prevent the eventual disaster, be it a software error, error by
the provider or a meteor strike. At some point you will want your backups to
restore.

~~~
technologia
I concur, the author of the post doesn't even talk about their own backups but
assumes DO does it for them. I personally love DO but I'm not about to skip
over my own security & resiliency measures.

~~~
betageek
Whether OP has backups or not, this is a very poor response from DO - they
screwed up, not OP, and the compensation they've offered is nowhere near
commensurate with the size of the screw up. Deleting customers data in an
unrecoverable way is breaking the number 1 rule of hosting - it's not like a
drive went out, DO specifically sell Spaces as a Backup solution!!

------
Buge
Wow, that seems bad. Does Digital Ocean provide any guarantees about data
reliability? A 2 month refund seems way too low, maybe a refund of all money
paid in the past related to storage would be closer to fair.

------
codegeek
DigitalOcean spaces certainly has a few bugs and glitches that I have noticed
while playing around with it. I don't have the confidence to do any production
work with it yet. But it is also a relatively new product even though no
excuse for losing data like that.

Btw, offsite backups folks, offsite backups. Always run offsite backups on any
system

------
rootw0rm
Since this post brought up the issue of backups, what do people use these
days? I don't admin any servers at the moment, but when I did have a few side
projects running I was a fan of HashBackup.

~~~
gmiller123456
I use MySQLdump, tar, and sftp to store the files I need on another server. I
know everyone's needs are different, but I would avoid any type of 3rd
party/complex backup system. If you have to install some tool to restore your
backup, you run the possibility of that tool becoming unavailable, or
incompatible with your backups. Sticking to tools that have been around for a
long time, and have multiple implementations mitigates that as a concern.

------
stareatgoats
To see data being lost in the cloud is truly terrifying. I don't have the
insight into DO or cloud management in general to assess whether this is a
one-off. My guess is that most providers would go to great lengths to keep
incidents such as these under the radar. DO didn't seem to be willing to go
the extra mile (buck) and will no doubt pay a much bigger price now. Would be
interesting to hear if the HN community has any insights into similar cases,
regardless of provider.

------
ksec
I have been saying this for quite some time. Apart from two 'targeted'
secruity incidents, Linode has been better at bandwidth, CPU perf, Storage
Speed, and has had most of the features such as bandwidth pooling, private
network for much longer then DO.

I don't understand why mainstream has all moved to DO instead.

~~~
Grue3
They are also not VC-funded unlike DO, which makes them much less likely to
follow the fate of GitHub.

------
jdietrich
Two is one and one is none. Drives fail, servers fail, keys get lost, accounts
get compromised, planned deletions get fat-fingered.

Modern cloud APIs make it trivial to build redundancy into your system. Usage-
based pricing and nearline storage makes redundancy relatively cheap in most
cases.

------
c12
It sucks that the DO data was lost; it's an awful situation to be in and
should never happen given the way that cloud options are sold.

The moral of the story being that you should always diversify your cloud
options and keep regular verified backups.

------
oceanghost
Nothing about this surprises me. DO rebooted one of my VPSes and it lost all
network connectivity. I can't get any files off of it and can only access it
from the console. Its frustrating as hell.

------
creshal
No, you lost all your data because you didn't make any backups.

~~~
CSDude
Do you backup your S3 buckets? Digital Ocean Spaces is equivalent of that. So
one would hope that would be stable enough to treat as a backup. Of course you
have too important files, you can duplicate them, we backup our S3 buckets to
another region, but another provider would be an option as well if we relied
too much on them.

~~~
misnome
> So one would hope that would be stable enough to treat as a backup

Ah, ‘Hope’ - the most used, and least effective, backup strategy in the world.

~~~
v_lisivka
You need to use paid prayers for better efficiency of this backup strategy. J

------
Mononokay
"What are backups?"

