
GitLab.com melts down after wrong directory deleted, backups fail - cryo
https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/
======
vonklaus
I just really like gitlab. I am not personally affected by the outage, but as
far as collosul fuckups go, I think they will do their level best to fix the
issue, be transparent about it and do what they can to make it right.

We'll get a really good measure of the company by how this turns out.

------
pmontra
Is anybody using a procedure to automatically perform restores and check that
they match production data? I guess it's not easy for complex systems and non
trivial amounts of data. Do you want to share the details and give us an idea
of the costs? Thanks.

~~~
paktek123
I've setup a postgresql backup before that would use pgbasebackup and get the
latest xlog segments then I would start up postgres and run select 1 against
it. That simply wasn't enough as I still got corrupted indexes but at least
data was there. If I were to do it again I'd probably run a whole production
test suite against it. The cost is extra hardware and time as backup boxes
aren't made for performance for a 1Tb+ database it would take more than 24
hours.

------
jwilk
[https://news.ycombinator.com/item?id=13537052](https://news.ycombinator.com/item?id=13537052)

