
DreamObjects is down - oal
http://www.dreamhoststatus.com/2014/06/13/dreamobjects-outage-2/
======
epistasis
The current status update says that its now resolved:

>Update June 14th, 2014, 10:20 AM PDT: Stability on the DreamObjects cluster
has been restored. Requests appear to be resolving properly now that the
system has had time to re-balance itself. Our test are reporting properly now.
If you do have any questions or concerns, please contact support contact
support.

I'm quite interested to know what went on in particular, as I'm far more
interested in Ceph than in commercial object stores that I can't extend.
Librados is pretty damn awesome too, and I can foresee implementing some
highly distributed storage through that directly.

With DreamObjects, it sounds like some API servers went down, and failure
happened such that it couldn't serve some requests until the appropriate nodes
came back.

It appears that with Ceph it will be easy to keep enough replicas such that
data is not lost, but high availability is still being hashed out. Hopefully
the lessons from this failure guarantee that this particular failure mode
doesn't happen again.

------
tedchs
Status update says they're running Ceph, an S3 work-alike. I don't want to say
"clone" because nobody outside Amazon really knows how S3 works, and Ceph
merely has an external HTTP API that supposedly works the same way. Is anybody
running Ceph at scale and might be able to comment on what broke down?

~~~
avtar
I'm interested in Ceph-in-production stories as well. I subscribe to this feed
[http://www.sebastien-han.fr/blog/](http://www.sebastien-han.fr/blog/) which
follows Ceph development and have been toying with the idea of testing it when
time permits. Currently we use local storage for VM hosts provided by LVM but
something distributed and designed with HA in mind would be helpful.

------
ilaksh
How is the performance of DreamObjects?

For people who are actually on a budget for their projects, which is more than
you might think, if the reliability is OK, this might make a lot of sense over
paying much more to use the real S3 for backups.

------
Tarang
The name of the service is DreamObjects. No affiliation to Dreamhost but doing
something as basic as storing files as a service hardly deserves to be called
a clone.

~~~
tedchs
What do you mean "no affiliation"? It's one of Dream host's services.

~~~
phpnode
I think he meant that he had no affiliation himself

------
hyperliner
It seems they copied AWS S3 _too well_ , including its "feature" of going down
every now and then and it's overall lack of reliability. Nice!

~~~
thezach
I've only had very minimal issues with S3... one of the things thats holding
me back from switching to a cheaper provider from Amazon is if its not broke,
don't fix it.

~~~
zimbatm
Individual blob access is not very reliable on S3. In a previous company
uploading big-ish (1-10000MB) files was part of the service. We would see
failing uploads or slow writes all the time. Make sure to queue and wrap your
upload script with retries.

