
Why the “Digital Ocean killed my company” incident scares the hell out of me - tnolet
https://blog.checklyhq.com/why-the-recent-digital-ocean-killed-my-company-incident-scares-the-hell-out-of-me/
======
freehunter
I've had a DO mistake take down my site before, and when it was brought back
up it had been reverted to several months prior. DO support was at a loss as
to why this would have happened. I tried to restore from DO's backup service,
but their backups had apparently stopped running several months prior as well.
This was a major issue and could have easily been the death of my company, all
because of a DO glitch.

But it wasn't, because every night I run a PG backup and copy it to AWS S3. I
just had to download the backup from the other cloud vendor and restore it on
my DO server.

Did DO fuck up? Yes. Did it cost me downtime? Yes. Was I mad? Yes, and I still
am. But I still do business with DO because it costs ~half the price of a
comparable EC2 instance, and writing a 10 line bash script to move my database
backups to another cloud vendor isn't that hard. Storing that backup on S3
costs pennies per month.

I don't see "I've never run a business before" as a valid excuse, nor do I see
"the cloud vendor is better equipped to handle backups" as a valid excuse.
When it comes down to it, you are solely responsible to your customers.
They're not going to care whose fault it was, because it was your fault.

Don't trust _any_ of your vendors.

~~~
jwr
> But I still do business with DO because it costs ~half the price of a
> comparable EC2 instance

I'm curious why people who don't have massive scaling and variability issues
choose DO or AWS for their hosting.

For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM,
2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run
most sites out there. And if you need more oomph and better reliability, rent
three.

I looked at this many times and still check the pricing regularly, and it just
doesn't make sense for me to switch from these physical servers to virtualized
offerings, not in any foreseeable future.

~~~
Someone1234
Because hypothetically your instance should be hardware agnostic. If the
physical hardware dies, it should automatically migrate to another physical
server at their data center without your intervention. It will only look like
an unexpected restart from your perspective.

That's something worth paying for. It would be more comparable to two servers
at Hetzner with rapid failover. But even that is more involved since you have
to set up the logic of when to failover.

Auto-scaling is a benefit of "cloud" services. But it isn't the only selling
point. Hardware abstraction is perhaps bigger.

~~~
olau
That's wishful thinking.

If there's state in that virtual machine, it's probably either stored on the
physical host or in a SAN. If it's in the physical host, it has to be fished
out of that machine or restored from a backup. If it's a SAN, you can lose
your virtual machine if the SAN goes down.

I've seen both happen.

Actually, a single machine with RAID is surprisingly stable. A provider like
Hetzner can switch out a faulty disk in less than five minutes, or switch a
faulty motherboard/power supply in less than half an hour.

Virtualization on top of this does not increase stability, in my experience.

Now, some cloud providers do have more sophisticated distributed systems that
do not have single points of failure, and that's a completely different story.

Of course, the software itself is a source of correlated failures, so even
there you should never rely on a single cloud vendor.

There's a poster here on this site who commented some months ago that he had
for years rented three servers, each on a different continent, each from a
different provider, and never had downtime. That's engineering.

~~~
treis
>If there's state in that virtual machine, it's probably either stored on the
physical host or in a SAN

I think the point is to write your application in a way that there's not state
in the VM. My VMs are disposable and in fact the way I do deployments is to
spin up a new VM in DO and then assign it the floating IP for production. If
things go haywire I can easily swap back the IP address to the known good VM.

~~~
dijit
Are you using managed databases then?

~~~
treis
I am. Mostly for the backup and other DB administration that comes with it.
Also use Spaces (DO's S3 equivalent) for any assets that aren't packaged with
the code like user uploads.

~~~
aitchnyu
Future wish: all VPSes have managed databases, object hosting and centralized
logs as standard so as not to rely on local disk at all. There is a streaming
backup protocol, which allows any database and queue vendors to stream to any
location, and acts as a peer for fully synchronous replication. If your DO
Postgres, Kafka and Mongo instances are killed, you have a Bakstream cluster
in Linode which lost no data and allows you to restore in Vultr.

------
terryf
I am old enough to have lived in soviet times for a short while. One thing I
remember from back then - in order to get things done, you had to know
someone. Perhaps your aunt's friend worked in the politbureau or your mom's
classmate was a friend of the director. Through connections like that you
could get what you needed.

It was such a relief when times changed and regardless of who you were, you
could start exchanging money for things and services.

This story and many others like it remind me of those times - it's back to who
you know to get your account unlocked. I've never gotten a story on the front
page of HN and probably have 5 twitter followers. What's my avenue of getting
my data back?

This isn't just about DO either. Similar stories about google and other
services are many to be found.

~~~
duxup
I'll toot my own theory and it is that nobody wants to pay for enough capable
support staff, pay to keep the support staff at the ready often enough, pay
support staff who want to stay in that role.

They want to automate that all away as much as possible.

The future is everyone who isn't somebody chatting about how they run their
application on X... because they're mysteriously banned form Y and Z and the
next guy talks about how he was banned on X and is on Z now... simply for that
reason.

~~~
treis
>I'll toot my own theory and it is that nobody wants to pay for enough capable
support staff, pay to keep the support staff at the ready often enough, pay
support staff who want to stay in that role.

The crazy thing is that it's really not that expensive. AWS, for example,
offers 24x7 support w/ <1 hour response time for production issues starting at
$100 a month. At worst it's 10% of your bill.

~~~
duxup
It's very doable, but sadly even skilled management (in my experience) tends
to avoid / flee support organizations due to their low prestige / resources
pattern that is across a lot of industries.

I fear it is something very doable, but I'm not sure there are many in
leadership that can.

------
fhood
Arguably Raisup should have been better prepared for this eventuality. _But_ ,
I suspect that many of the people here talking about how _their_ system is
infallible made it that way because they, had time and/or money to spare
working on it, have a very portable system, or have had something like this
happen before.

I feel like HN comments are often sanctimonious to the point of ignorance. A
solo dev, scrambling to get his project to the point where there is even the
tiniest chance it will succeed, is likely to have a complex network of
hardcoded filepaths, hostnames, and other magic numbers, strings, and config
files that would make it very difficult to make portable without significant
time and effort.

Or maybe the guy was an idiot. But possibly entertain the thought that maybe
his situation wasn't exactly comparable to your eh?

~~~
DKnoll
> Or maybe the guy was an idiot. But possibly entertain the thought that maybe
> his situation wasn't exactly comparable to your eh?

I have never encountered a business that does not require backups. Not having
a DR site as early in the game as they were might be excusable, but even then
suggesting they get one is sound advice. It's an additional expense but it has
a cheaper implementation cost while your infrastructure is small. Having a
good DR plan is a selling point as well, this is the first thing I ask SaaS
providers when I am evaluating them.

What is a total failure is not having backups on another provider. Before
anyone says it's just a 2-person startup... surely that makes it easier? I
can't imagine their DBs are all that big. Even just a three line cronjob and a
Backblaze B2 ($0.005/GB) account could have secured their business continuity
in the event their DO account didn't come back up.

>But also DO Spaces for our object storage with ~500k media and our database
backups.

Based on that comment (from @w3nicolas on Twitter) they actually could have
used the same script they used for DO Spaces to back up their DBs to S3 (both
use the S3 protocol). The 500k image files should be relatively small and
cheap to back up too, appears they are just logo thumbnails for the tracked
companies (500k image files, copy says they track 450k companies).

If you are building infrastructure and you don't have IT Ops experience
consult somebody who does because they will save you from yourself in
scenarios like this. If you are a dev you could probably throw a rock and hit
5 friends with enough ops experience to tell you the importance of redundant
backups.

I think this comment might read a bit more abrasively than I meant it, but I
don't mean to kick the guy after he was down. If you are in a similar boat to
RaisUp with your backups go fix it now. If you rely on any one vendor for
business continuity that is a problem. This PSA is sponsored by salty
sysadmins everywhere.

~~~
vardump
> I have never encountered a business that does not require backups.

Me neither. But unfortunately the most I've encountered don't either have
truly full backups (missing stuff), actually working backup process (some
issue with backup media, process, etc.) or any idea how to actually restore if
something _does_ happen.

Almost no one does drills to restore backups to some test system.

I can't talk about the other cases, but I can tell about one small startup I
joined a long ago. The worst one. They did backups on a 3-disk RAID-5
server... with one disk broken and one with SMART warnings. Their backup
process also failed to backup anything with too long path names, so in reality
almost half of the data was actually missing! There was also some unicode
issue that lost files with characters above 128 ASCII.

My first days went to just actually ensuring their data is backed up...

------
codingdave
You don't have to actually have redundant providers and monthly-tested DR
runs, etc. But you do need a plan. No matter how small.

If you are just a side project running locally, than plan me be as simple as
"If my laptop falls in a river, get a new one and clone the repo again." As
you start having production systems, it may still be "clone the repo again",
but this time to actual servers.

As you start having customers rely on you, your plan should get more robust.
Maybe you don't have a hot DR site set up where you can just flip a switch.
But you should know who your backup host would be, and have an account ready
with them. You should what steps would be needed to go from repo -> new
provider.

None of this needs to be set up and tested ahead of time if you are just
getting started. But if you have paying customers, you have to have thought
about it. DR starts tiny and scales up, just like everything else.

------
ianlevesque
> Do have some backups outside of your primary cloud provider. You'll sleep
> better.

This is really the takeaway from this whole incident. Even if the account
doesn’t get banned you’re just one accidental action away from your database
and backups disappearing simultaneously otherwise in most cases. Or if you are
using a blob store there may not be any backup if the original is deleted
(it’s like RAID not backups).

~~~
cm2187
And have a separate DNS provider.

~~~
crazygringo
No reason to.

If your cloud provider does your DNS and you switch to a different cloud
provider...

...then just set your registrar to point to the new DNS provider. You've still
got total control.

~~~
cm2187
That transition requires authorisation from the original provider. If they
locked your account and don't pick up the phone, you won't get the move
approved and won't be able to repoint your urls to a different cloud.

~~~
bradyd
No it doesn't. Name servers are controlled by the registrar not the DNS
provider.

~~~
cm2187
By provider I mean the registrar (which typically provides both services).
Moving to another registrar requires an authorization code, and good luck
getting that on a short notice if your provider doesn't talk to you.

------
duxup
Digital Ocean's statement on the events:

[https://blog.digitalocean.com/an-update-on-last-weeks-
custom...](https://blog.digitalocean.com/an-update-on-last-weeks-customer-
shutdown-incident/)

Lots of support related issues here.

As someone who worked in support for a long time it isn't surprising to see
this play out and the whole "Support and Security Operations leadership will
create new workflows to allow abuse-related events to leverage the 24/7
structure of Support." while probably a good action to take, is one of those
things that you see in support time and again and so rarely do you see an
appropriate response as much as a patch for "this one thing won't happen
again".

Nobody cares to staff, fund, and support the support teams until something
goes wrong, and then it is usually "new workflows" for support.

------
Operyl
When I joined Digital Ocean, back when it was 2-3 months old, I hit a KVM bug
in their stack with my network traffic that caused them to rate limit my
droplet to 1Kbit/s because it kept taking the host machine down I guess(?),
and then them ultimately terminating it without a ticket asking me to even
investigate. That was enough to pretty much never consider using Digital Ocean
for anything “production” ever again. They didn’t terminate my account, and
apologized with credit, but wow.

Another thought: That was also back when support responded in a reasonable
time frame, and they actually spoke on IRC and everything too.

------
turrini
I've got a stable setup:

    
    
      ---------------------------------------
      Vultr ========================== Linode
      ---------------------------------------
      Los Angeles ==================== Dallas
      Mastercard 1234 ============= Visa 4567
      ---------------------------------------
      VMs with custom Linux (ROOT/ZFS Debian)
      Replicating snapshots every 5 minutes
      CloudFlare + HAProxy Load Balancing
    

Total cost: 6 VMS at $20 each: $120, CloudFlare $20

Edit: backups to B2 and Wasabi. ~$10 month

~~~
lohszvu
Are you using a managed host for HAProxy? What if it goes down.

~~~
turrini
I have a failover setup that changes Cloudflare DNS when necessary.

------
cdumler
Pretty much the standard "the cloud is just someone else's computer" issue: if
you don't have the ability to reproduce your work somewhere else and your
provider decides to go away, you're going to have a Bad Time(TM).

Also, for as cool as Digital Ocean is, their primary focus is on low-use,
shared cloud resources. From my experience, they over subscribe CPU resources
so "noisy neighbors" was a problem sometimes. They do not provide or work with
people as if they have production services, and they don't seem to like people
who really want to use their system. I would never use them for production
unless the work was ephemeral.

------
cyberferret
What scares me the most these days is that a lot of vendors have a policy of
"we're shutting you down, but we won't tell you why or how you can fix things
so that it doesn't happen again".

I have had friends who have had their Twitter or Youtube accounts shut down
with no explanation from the companies involved. Just recently, MailChimp shut
down a brand new account for my SaaS (we had only 4 subscribers added, and
hadn't sent out any email campaigns yet!) with a refusal to give us an
explanation why. We set up another account and did everything exactly the same
way (including tediously replicating lots of email templates manually) without
the same thing happening. Why was the first account shut down? We will never
know, it seems.

I am all for using AI to detect dark patterns and raising a flag, but I think
these companies need to bring a more human aspect into the after effects.
Surely a legitimate operator will always email in with a "WTF?" whereas a
nefarious actor will simply automatically move on to the next thing?

~~~
bcooks
Hi cyberferret. Two quick trys to provide some clarity for you here:

1\. On your vendors don't say why they took action point. I think a lot of
vendors, us included, worry about the bad actors knowing the _how_ of our
algorithms because they will then have intelligence on what to try and work
around to avoid detection. It's a constant adversarial chess match we adjust,
they adjust.

2\. As for the human element being needed for the after effects I totally
agree. That said, I looked at our numbers and while many of the nefarious
actors simply move to the next thing, a very large percentage of the, "WTF"
replies are actually from bad actors hoping that is enough to be re-enabled
and to keep going for a while. Our goal is to bring more people into the after
effects and give them better information to make decisions and work with
customers to avoid this kind of mess in the future.

~~~
cyberferret
I appreciate you giving an insight into the reasons behind your decisions
Barry - I've learned a bit here myself, and can relate to the quandary that
you must have to tiptoe around, in between being open and communicative,
versus keeping your cards close to your chest. Thanks, and best of luck.

------
tnolet
OP here. A dude in the devops Slack channel had a great comment on the multi-
cloud thing. "People can't even get redundancy and failover working on one
cloud, why bother with two."

------
siffland
If you are running a business then always have a good disaster recovery
strategy. Split services between cloud providers and make sure your backups
are good and offsite. You can do this all with opensource software if you dont
want to pay for anything too (and if you have the staff that can do it).

If your business can be "killed" by your provider losing your data you have
set yourself up to fail. I am not saying the provider is not to blame, but the
burden should also be on the customer for not protecting themselves, what if
DO burnt down?

We have yearly DR exercises, we have to bring up about 40 of our 120
production instances, it is a difficult task, but we know we can do it.

Never depend on your provider to save you, always make sure you have planed
this out.

------
exabrial
I really liked their blog response. I logged a support case about the
incident, and a rep personally called me and spent 30m on video chat with me
answering my questions. I don't think that would _ever_ happen from GCP.

------
chrismeller
> The initial account lock and resource power down resulted from an automated
> service that monitors for cryptocurrency mining activity (Droplet CPU loads
> and Droplet create behaviors).

So despite the fact that you can get the same or more resources at other
providers who advertise their shared CPU resources for less than half that
charged by DO I’m still buying a shared resource?

I think this is something that needs to be much more front and center. With
AWS I can spin up as many “boxes” as I want on a moment’s notice. As far as
anything on DO’s site seems to advertise I can do the same.

Turns out that’s not so. If they wanted to shut down an account because they
suspected it was compromised then that’s one thing. Shutting it down simply
because of suspected cryptocurrency activity... not so much. If I’m willing to
pay for it then that’s what I’m willing to pay for and there should be no
limitations.

Now that becomes a little more clear as you read...

> determine if automated action is warranted to minimize the impact of
> potential fraudulent high-cpu-loads on other customers.

So clearly we are not talking the same resources that were expected.

~~~
EdwardDiego
> So despite the fact that you can get the same or more resources at other
> providers who advertise their shared CPU resources for less than half that
> charged by DO I’m still buying a shared resource?

The account that got locked down triggered automatic checks that used payment
history as a "these people are okay" check. They didn't have a payment
history, they were running solely on credits, so got flagged. IOW, they hadn't
paid for anything yet.

> With AWS I can spin up as many “boxes” as I want on a moment’s notice

You can, but they can also shut you down for "abuse".
[https://aws.amazon.com/premiumsupport/knowledge-
center/aws-a...](https://aws.amazon.com/premiumsupport/knowledge-center/aws-
abuse-report/)

~~~
scarface74
You have 24 hours to respond and why would you have a business and not be on
at least their business support plan?

~~~
EdwardDiego
That's a very good question for their CTO. Maybe we should ask him on Twitter.

~~~
scarface74
I’m referring to AWS’s policy compared to DO’s.

If he were using AWS or Azure and he had the same set of issues, I wouldn’t
blame him at all. But he is using a third rate cloud provider because it was
cheaper and then he acts surprise that they don’t have the same level of
competence and support as AWS or Azure.

Yeah I purposefully left out GCP. I wouldn’t trust their support anymore than
DO.

------
EdwardDiego
> Why isn't everyone on bare metal?

Is this how we're referring to running on servers we have contractual control
over these days? Part of managing risk is being sure your infrastructure will
stay there, and if you don't have a contract, your risk is significantly
higher.

------
INTPenis
Something tells me it scares DO too because I've been getting emails lately
asking if I'm happy with their services.

------
mabbo
It feels like the need for multi-cloud designs and infrastructure are becoming
more sensible. If costs are best with one provider, by all means put 95% of
your traffic there... But keeping 5% running smoothly on a second cloud, with
a simple means to rebalance between them would be a huge win. It would mean
the difference between an annoyance (argh, one of my cloud providers doesn't
work) and a disaster (my sole cloud provider just ended my company). It would
also save you the next time AWS or Google's cloud have an outage.

Are there any good resources for deploying multi cloud architectures?

~~~
thruhiker
For a business, especially a startup, the opportunity cost often makes multi-
cloud infeasible. You'd rather spend that engineering time on delivering
features or addressing tech debt. I'm of the opinion that multi-cloud doesn't
make much sense if your monthly cloud spend is less than at least $500k.

Pick one of the strong public cloud providers such as AWS, Azure, or GCP and
then be judicious about which PaaS offerings you use to avoid unnecessary
vendor lock in unless it adds enough value to justify the lock in.

------
yardstick
I’ve asked this in a previous thread but didn’t get much of an answer: what
can a business do to proactively fend off issues with cloud providers shutting
them down.

I’ve heard GCP (or was it AWS?) is less likely to shut you down if you bill on
account vs using a credit card.

Are there any other things we can do to reduce problems?

Would asking for an account manager help? Would buying reserved instances
help?

Is there a minimum spend that gets you more attention and priority? Eg
100/month vs 1k/month vs 10k/month etc?

Any other advice?

~~~
dathinab
\- Use billing methods which are less likely to be used by scammers (e.g.
likely not credit card).

\- Keep a steady balance.

\- Have luck (sad but true).

Minimum spend and similar might help with customer service (if they can see
your spend). But most problems start with automated services.

Don't switch cloud providers every half year because another one is slightly
cheaper now (i.e. have longer running/existing accounts), might help.

 __But most important make sure that you don 't get locked in by the could
provider __

Even a permanent ban from the could provider should just cause some, not to
high, short term (mony) loss. This means:

\- Do not ~use~ relay on cloud providers proprietary tec (sure use there
management console, etc. but don't integrate it into your business to
tightly).

\- Have control over your DNS names so that you can map them to different IP
addresses if needed.

\- Make sure you can migrate to a different cloud provider in matters of
hours/days/weeks (depending on what you use it for) at any time.

\- Have backups outside of the could provider for all your stuff. \- don't get
trapped by doing backups on a different service of the same provider (e.g.
Amazon) ;=)

In the end how much this costs you depend a lot on what you do and how you do
it. And yes deciding and not doing any continous shot term investments taking
the risk of loading everything is fine, you just should make the decision
conscious.

~~~
yardstick
Some good advice, thanks

------
weaksauce
I made this comment in an older thread but haven't gotten an answer on... the
circumstances seem off to me:

The thing that has me scratching my head is how this chain of events unfolded.

I get that your fraud algorithm flagged it because of lack of established
payment. how is this possible if what the tweet referred to as "locking us out
of all of our backups and work"? surely an account history of any significance
would have an established payment record. From their tweets they mention that
they had 5 droplets and some storage of a not insignificant number of records
(~500k) and that a script is required to be run every 2-3 months to process
some data and that script spins up 10 droplets during that time. seems like it
will take 13 hours to process the data based on row count and per record
time.[0] I am struggling to see how they didn't have payment history. can you
elaborate?

In addition another thing I'd think would help assuage fears of a complete
lockout is some process where you can request and download the db or a
snapshot of the virtual machine.

[0]
[https://twitter.com/w3Nicolas/status/1134529322902007809](https://twitter.com/w3Nicolas/status/1134529322902007809)

~~~
bcooks
Hi weaksauce... sorry I must have missed your earlier question.

The account had been live for some time and in that sense had history but
because of credits it didn't have payment history. As some others have
commented lots of startups use credits to get their business going and
depending on your usage they can last you for quite a while. Payment history
indicates a willingness and capability to make payments.

Part of the issue here was what triggers the algorithm used when looking at
remaining credits, payment history (none), workload deltas (the new spin ups),
and effective run rate (think of that as the amount of money they would be
charged for the workload they were spinning up). The bug in this case was both
simple and super impactful. Raisup did nothing wrong, everything right in
fact. We just blew it.

Thanks for the comment on request for download of backups or snapshot. That is
a great idea, I guess we just never expected to actually go shoot a real
customer and the fraudsters don't ask for their data.

~~~
weaksauce
I appreciate the response. a followup question to that would be how they got
enough credits to be running for that long without any payment?

~~~
bcooks
DO has a startup program called Hatch:
[https://www.digitalocean.com/hatch/](https://www.digitalocean.com/hatch/)

That is the starting place for many folks.

~~~
freehunter
So a startup was getting their hosting entirely for free, their paying
customers were Fortune 500 companies, but they didn't have the money to pay
for off-site backups.

What the hell was their cost then?

------
Sawamara
There is a heck of a difference between "eh, it only costs $5-$10" and between
"the survival of my project relies on the site/service being under my control.
If it was the latter, you go in having a plan B and a contingency Plan c.

GCP, AWS, DO, Azure... they all bank on that sweet combination of huge clients
and clients who do not really need them but due to opportunity costs, cant be
bothered to find viable, self-reliant alternatives.

------
reading-at-work
I recently tried to sign up for a DO account to deploy a proof of concept app
for my startup, and my account was immediately locked within minutes of
opening it. I hadn't even spun up an instance yet. That combined with this
means I'll be sticking with AWS for now, I suppose.

------
whitepoplar
I _want_ to love DO so much. I think they really do mean well and their
products are great, but yeah, this kind of stuff scares me shitless for
mission critical workloads.

------
tasssko
Don’t use DigitalOcean for anything serious. My team and i frequently support
DigitalOcean customers with issues like crashed applications failed builds and
in many cases the challenge is how they (DigitalOcean) provide resource
allocations for droplets. 1cpu != 1cpu which is vital for vm to perform at its
best. AWS and GCP are better. i’d argue that most IaaS platforms are better
than DIgitalOcean. You get what you pay for. i’m not a customer.

------
docker_up
The one thing that the author forgot or didn't realize to mention is:

Test moving your environment between service providers in production.

You should be able to failover from one provider to another, in a predictable
way. You should actually do so every few months so that you're well versed in
what needs to happen. If you practice this, you will be prepared in case the
absolute worst happens.

------
BinaryArcher
My account was banned two years ago just for accessing it from abroad. This is
real, don't use DO.

They finally gave me temporary access even though they didn't believe who I
was. This was the icing on the cake, they gave me access to a cluster, they
could've given it to anyone.

It's cheap but seriously not worth it. They also have many more outages than
other larger providers.

------
DenisM
Any stories of being screwed by AWS in the similar manner?

That is, excluding the stories about the same account being used to sell thing
on Amazon.coma _and_ run a AWS setup. We know those are risky.

------
ksec
I think Scaleway and UpCloud are two biggest contender to DO. ( Not sure why
Linode is not getting traction, may be its security incidents )

~~~
freehunter
Before I moved to DO I used Linode. My day job is a security consultant, I
switched from Linode because of their repeated security incidents.

------
mobilemidget
'before switching to AWS completely'

why rely on a single party, you are putting all your eggs in a single basket
again.

------
egorfine
> most appeals are true bad actors

is that really the case?

------
the_common_man
Was there a post mortem by DO?

~~~
aquaticsunset
This was linked above

[https://blog.digitalocean.com/an-update-on-last-weeks-
custom...](https://blog.digitalocean.com/an-update-on-last-weeks-customer-
shutdown-incident/)

It doesn't explain any detail behind why there was radio silence until social
media support stepped in, which is what I'm very curious about, but it does
have a timeline of events and an apology.

------
mark-ruwt
I'm a one-man startup and I couldn't disagree with this more.

If you want to "play" startup, feel free to put all your eggs in one basket.
If you actually want to build something meaningful and long-lasting, and you
have Fortune 500 customers, take a breath and invest time in an infrastructure
that can't be destroyed by a single rogue algorithm.

After all of the horror stories we've heard over the years, how is that time
investment (and ability to sleep well at night) not worth it?

~~~
luckylion
> take a breath and invest time in an infrastructure that can't be destroyed
> by a single rogue algorithm

As in? You can have your own cage, but somebody will have a cable that
connects to the hardware you own. If they pull that cable because a single
rogue algorithm told them that you're most likely abusing their service,
that's it.

------
scarface74
_I do not have all the details, so caveat lector, but apparently it was due to
spinning up a dozen droplets and running a batch job._

If this is considered “abuse” Digital Ocean is not a platform you should be
running a business on.

If you can’t afford to pay for reliable big boy infrastructure, you might need
to rethink your business plan or how much you charge.

------
TheSpiciestDev
All this Digital Ocean talk... I spun up some droplets to play with a few
months ago and liked what I saw and then "deactivated" them or turned them
off.

What I did not expect to see a few days ago was the bill for the stopped
droplets. Not just a little for the dormant disks I have had but seemingly
full-prices for the entire droplets, as if they were all still "active".
Nonetheless I deleted all my droplets, discarding whatever I had setup a few
months ago, out of pure frustration. I never plan to return.

AWS may take me a little more time to setup and their interface may not be as
pretty, but they will at least bill me fairly.

~~~
freehunter
I just tried to turn off one of my droplets and had to click "okay" on a
message that said "You will still be billed for Droplets that are turned off
because your disk space, CPU, RAM, and IP address will still be reserved."

If you clicked okay without actually reading the message, that is entirely
your own fault.

~~~
TheSpiciestDev
While I don't remember any such message, after a few months it's certainly
felt wrong and I've felt the sting. It'll be something I take into
consideration whenever I shop around again.

