
Limited Availability of Spaces API in NYC3 - turrini
https://status.digitalocean.com/incidents/rc3wwtvms36t
======
hemancuso
What is the replication factor on these objects? 0.1% objects unavailable is a
worrisome number as it suggests a meaningful number of objects are under-
replicated and the cluster is poorly configured. I hope DO speaks to the
durability of Spaces in a meaningful fashion after this is sorted out.
Multiple days of having a large amount of data unavailable should shake the
confidence of anyone considering Spaces for object storage. Are objects stored
with 3 replicas? Are they on heavily striped volumes? What is going on that
could cause this failure mode and what's stopping a much larger failure?
Lastly, nowhere in their status updates are they saying "don't worry, no data-
loss, just unavailable for a bit" \- which I read as "we hope to not lose any
data, but no promises"

------
Operyl
This is why I have consistently seen people stay clear of Digital Ocean.
They’re great for pet projects, and staging/development environments (as was
their original intention when they came out if I recall correctly) but many of
these “small” issues can quickly become a huge headache. Sure, anybody with an
ounce of sanity has a backup plan, but the reality is none of us want to have
to jump into it unless absolutely required.

On a side note: you should test your backups and plans often! Don’t let
outages catch you off guard.

~~~
holdenc
I'm a customer at linode, vultr, and digital ocean, and digital ocean (over
the last 6 years in their NYC datacenter) has had exceptional uptime and
physical host integrity. I've never had cycles stolen by "noisy neighbors" (as
has happened at Linode) or lost a server (as is to be expected on EC2). Their
unixbench scores are great (fast disks and CPUs).

All that said -- you want use a new product, expect to pay the pioneer tax.
Their CDN is even greener (no built-in GZIP, broken CORs header forwarding,
only pulls from DO spaces, no CNAME mapping) but the bandwidth is free/dirt
cheap. They are very upfront about their offering, and don't seem to be over
promising, or over selling. You get what you pay for, and I am ok with that.

~~~
Operyl
I guess my problem is they’re not ultimately some new product. Based on
previous outages for this same product they’re basically reselling Ceph.
There’s no new technology here.

------
alphabettsy
I transferred a 1TB Cryptomator vault to Spaces to test it and I’m finding
quite a few files unreadable with errors. This is not an issue I’ve had with
any other storage provider. They’re easily reproducible files, but I’ve lost
dozens of files at this point, which is unacceptable for a storage offering.

