Hacker News new | comments | ask | show | jobs | submit login
CamelCamelCamel: $44.6k disk failure, down until Feb 3 (camelcamelcamel.com)
66 points by lwhsiao 15 days ago | hide | past | web | favorite | 72 comments

As others have pointed out using consumer grade SSD drives for a database is a bad idea. Databases are going to do a lot of reads and writes so that's going to lead to problems down the road.

The second issue, is that since this was setup in a RAID I'm assuming that all of the drives were purchased at the same time when it was originally setup which means depending on the supplier you are probably going to receive drives from the same batch of production. Because of striping with RAID if you have some fault in a batch and multiple drives from the same batch then you can have failures on multiple drives that occur at approximately the same time.

The simplest solution is of course to buy enterprise grade drives but the obvious issue there is the price is much higher. Especially if you are going to be purchasing the largest drives as you always pay a premium for max capacity drives and the price jump on large drives is not insignificant.

When we first built out our SSD cloud for DigitalOcean at the end of 2012 we went with consumer drives because we were bootstrapped and SSDs were also significantly more expensive so we couldn't justify the price jump without knowing if we had product market fit.

But rather quickly we ran into a whole host of issues. Another common issue is that performance can degrade when consumer drives get to near full disk capacity so you can't really use all of the available space if you plan to do a significant number of reads and writes, which is pretty much what a database server is designed for.

That was the first upgrade we did after raising money, which is switching to enterprise grade drives, unfortunately you have to pay the premium but it really reduces problems significantly.

I do believe your experience, but started wondering what makes an enterprise drive more reliable than a consumer-targeted one. What's preventing the manufacturers from selling the same drive with two different names and prices? Reputation?

Enterprise class disks have significant behavioral differences when dealing with a bad sector, among other differences. Intel has a good paper on this subject. See 3.2.1:


One difference is that enterprise drives have lots more spare capacity. Like >>100%.

This can be replicated in consumer drives by underprovisioning.

OK, fair point. Although I don't know enough about SSDs to know whether there's more to it.

But what else is different?

I think mainly warranty, although other posters are saying enterprise drives have capacitors to help with writing during power loss. I'd have thought a good battery backed raid controller would handle this but "enterprise"="belt and braces" so I can see why a lot of businesses pay the extra.

Right. Capacitors. Especially for write cache, I think.

But yeah, battery-backed RAID plus UPS would be good enough for most, I guess.

So the enterprise drives perform as specified and the consumer drives don’t?

Sorry, late to reply! The warranty differences are things like with enterprise they'll send you a new one as soon as the SMART data suggests it's going to fail. Consumer is usually you post it back and they send you a new one.

If you actually read the specification on consumer drives you'll see they're "guaranteed" for like 100 rewrites and you don't have any right to anything if they fail.

Which should be a hint as to what's coming.

It would quickly become very obvious. And then they'd be sued for a lot of money.

Disagree, Consumer grade hardware is a fantastic idea because it's so bad it forces you to write robust software.

Enterprise hardware is 3x the cost of consumer hardware, but on average it's at the most twice as reliable. If your software robustness and redundancy is high enough, the total TCO will be lower and no less reliable.

The real wtf with CamelCamelCamel is not that they were running consumer drives, it was they were running 3 replicas on consumer drives with what seemed to be a too heavy workload.

Proper infrastructure is expensive, along with competent ops. Clearly not prioritized. If infra was neglected I would wager security is too.

Oh boy I got triggered, wall of text incoming.

Competent ops being expensive wasn't part of the discussion, Good people are expensive period. In the long term, competent people provide better value. Competent ops would figure out to provide the same reliability guarantees at a lower price. Security likewise, nothing to do with it. That's just using circular logic to say using commodity hardware = bad security (wtf!?).

Sofware and by extension infrastructure gets expensive because of incompetent dev/ops, who are too incompetent to implement efficient systems and use a 'hardware is cheap, throw hardware at the problem' mentality in ops. This especially shows today with the fact that we have have mobile phones with 1000x the processing power of desktops 20 years ago that are sluggish when running a simple TODO app. LET THAT SINK IN. Those same 20 year old computers running the first pentiums were running Quake 1(!).

Back to ops, Google before they were "Google" were using custom built commodity hardware using a 'nodes die, deal with it' approach to software development. Obviously this pattern is a win with the popularity of k8s today / ephemeral nodes but it requires a more complicated development process.

This extends to incompetent ops as well. Who are outsourcing all their work to the cloud, paying a crazy premium for it, calling it a day, whereas the competent guys are able to do it properly.

Closer to the subject, Backblaze engineered a from scratch design to create a highly reliable/available/secure system based on non enterprise class storage that lets the offer storage at a fraction of existing solutions.

Naive question, but is using old school spinning HDs completely out of the question? In other words, can they play a role in a modern deployment?

Unfortunately for databases under a heavy load it's out of the question.

You can have a server pegged at 100% CPU waiting on disk with spinning disks, and without changing anything else on the server RAM or CPU, and just replacing the drives with SSD you can quickly see that load drop down to 10%.

Random reads are an order of magnitude slower so for fast access there's a notable difference. Still a role for hard drives with sequential reads/writes, such as the regular backups they should have been taking! Slower in all areas but cheaper for the same space.

Spinning hard drives are the tape drives of the modern era. Boot, logs, backups, but no databases.

>On the evening of Saturday, January 26th, our database server had three hard drives fail. It was designed to handle two disk failures, but three failed disks made the situation catastrophic.

Like in the joke.

Our data storage was so well designed, that when first disk died, we didn't notice. When second disk died, we also didn't notice. When third disk died, we noticed.

It’s likely that the three drives died concurrently.

And they are probably setting themselves up for a similar fate in the future by replacing them all (failed and working alike), and at once.

They don't fail that reliably, unfortunately. It's not a light bulb with a certain number of hours on it. See Backblaze's drive statistics.

Edit: I just saw it's about ssds and not hdds, but while Backblaze might not have stats on those, I'm fairly sure my comment still applies. Not 100%, but I assume they account for failures due to predicable issues like write wear.

>I'm fairly sure my comment still applies

There are other comments in this thread that go into this, but your comment doesn't apply. Another poster used the light bulb analogy well to describe SSD failures.

It's very common to see SSDs that were purchased at the same time, that were likely manufactured in the same batch, fail within hours of each other. Im' pretty sure I even read this fact on Backblaze.

At least they should try and get from different vendors/models


I mean, I can see a power issue taking out all at once. It could be that. What other way would 3 disks fail at once?

All SSDs were from the same batch and had the same amount of writes on them and all failed in a very small window. It's a known issue with consumer drives.

Good to see that I'm not paranoid for explicitly using drives of different vendors in my RAID at home.

Luck? And hence it makes the news because such bad luck messes things up? We all know drives fail, I'm not surprised this happened somewhere on the planet. I would be surprised if it happened to me.

Is there a good Linux tool for logging RAID status?

I just have scripts that write key data from "mdadm -D /dev/md$N" to logs. But then I need to remember to review the logs regularly.

That's what you'd have monitoring and alerting for. The nagios/icinga/icinga2-world has check_mdraid, which works out of the box. Zabbix should have something similar

checkout freeNAS

To me, using consumer-grade SSDs seems less the problem compared to not having a database replica (for quick recovery) and PITR -capable backups (for near-zero data loss). These are not difficult to set up and are definitely worth it if your data matters.

In most cases your disaster recovery scenarios should include the complete destruction of a single server.

Yeah I was surprised by that too. A write-only replica might even be doable on cheap spinny rust disks if it's just product and account info.

Using consumer grade SSDs is a huge no no. For a company this size a simple server rack with something like production ready Seagate drives would do better in the long run. In my view, the SSDs they bought make CCC look like a school side-project.

I can't believe a company of this size is asking for donations...

I never got the impression they were big at all.

Yeah I always thought it was a one man band until now

Why do they use Samsung consumer drives on serious things?

Consumer SSDs lack IOPs and should be never used on database drives.

Also, I never use consumer SSDs on things that are not a joke. Serious things need enterprise SSDs.

I think some people rely on the historical fact that consumer 7200rpm SATA drives used to be nearly identical to their enterprise 7200rpm counterparts. Consumer SSDs are very different from enterprise SSDs; the former typically lacks power loss capacitors, which should be a nonstarter for anyone using them to run databases.

Surely a battery backed raid controller would be enough? That's how we did it with spinning drives in the past.

The reason to avoid consumer SSDs in "enterprise" situations generally isn't their IOPs... the IOPs of higher-end consumer drives is more than sufficient for a lot of enterprise-y use cases, especially read-heavy workloads (which I would guess is CamelCamelCamel's workload)

With enterprise drives you're paying for things like beefy capacitors to provide safe shutdown in case of power loss, write durability, etc.

Samsung SSD 960 Pro is a consumer SSD? What is e.g. a common enterprise SSD CamelCamelCamel should use in the future?

When I was doing managed Colo we used Intel "Datacenter" drives (ex. Intel S3500 SSD) which were priced at around $1/GB (so ~3-4x as expensive as the consumer counterpart). They were performant and, luckily, we never had an issue with them.

>so ~3-4x as expensive as the consumer counterpart

This is likely the answer. If CCC has to ask for donations to afford the consumer version, I imagine they are in no position to purchase an equivalent amount of storage in enterprise SSDs.

I guess they're busy now trying to fix things, but since they didn't mention how to avoid this in the future, I feel like streaming replication [to much cheaper spinning disk] or some other form of point in time recovery is the way to go.

Consumer SSDs can absolutely be the right choice in some situations. Great for scrappy development environments or things that are refreshed regularly like running ETL migration runs. I'd advise underprovisioning them by 10% and keeping good (regular) backups though. Also being aware of the failure of two drives of the same age in RAID1 being common (same number of writes=similar failure time).

A shame that CamelCamelCamel seem to be running on a bit of a shoestring budget as it's a very useful tool. To be fair to the recovery costs though the majority of them are from a professional data recovery company and that ain't cheap!

Does anyone know whether they got unlucky with 3 simultaneous failures, didn’t notice earlier failures, or had some other issue that killed all the drives?

Well, from what I see in the pictures they are using consumer grade ssds to power up an enterprise like grade database. This is not bad luck, this is just bad by design.

And they are doing the same ’mistake’ again.

I would guess the failures were errors on rebuild.

This article summarizes this kind of scenario fairly well: https://www.zdnet.com/article/why-raid-6-stops-working-in-20...

What is CamelCamelCamel?

As dddddaviddddd said, it's a price tracker for Amazon. To go into a little more detail: you can enter a product name or Amazon URL and it'll show you first and third party pricing history. You can also set a price threshold on a particular product and it'll either tweet or email you when the product price drops below that point. You can even do it without signing up or logging in.

I've set notifications on a couple items that I regularly order and when they drop below a certain price I order enough in advance.

> You can even do it without signing up or logging in.

Then what's their business model? Do they sell aggregated customer traffic data to merchants?

I'm guessing that referral is a big drive

> To go into a little more detail: you can enter a product name or Amazon URL and it'll show you first and third party pricing history

They also have browser add-ons to make that more convenient. They give you a CamelCamelCamel button that you can click while on an Amazon page to pop up the price history right there.

Price tracker for things on Amazon.

wouldn't using a raid card mean that the wear on the SSDs will be almost the same? i.e. they are likely to fail together?

This is why you shouldn’t roll your own hardware (unless you have very, very specific needs).

Wondering how much their annual cloud bill would have looked like compared to the bill they now got

56TiB of gp2 SSD storage on Amazon would cost $6,500/mo (so $78,000/yr) - so if its a 1:1 setup they could afford to fail a couple more times. Now there's the question of would they need all that space in a cloud setup, but thats another story.

What concerns me, from the image, is that they seem to be using cheaper, consumer grade drives (Samsung 860 PROs I assume). This is asking for trouble - as they went through - 3 drives failed simultaneously. I've only ever dealt with Cloud infra, but even I know that drives that are bought together, tend to fail together. It's likely their new batch of drives will also fail simultaneously - backblaze has done a ton of research on drive longevity.

Just seems penny-wise, pound-foolish to me.

> drives that are bought together, tend to fail together

It's exactly the same as the lights in your house, all of these were made at the same batch with the same quality of material and used the same way, they tend to fall apart within a few days from each other.

I wonder what their dedicated hosting bill would have looked like compared to both cloud and colo.

My guess is that it would have been a tiny fraction of their cloud bill, and their downtime would have been a smal fraction of the colo downtime.

You can spec an OVH server with 14x3.8TB SSDs for roughly $2800/month.

What wonders me most is the fact that 44k $ is a great economical loss for them: i would expect CCC to earn at least a milion/year so that 44k $ would be nothing more than a little annoyance

$1M/year seems high, but even if that's right, $44k isn't small. That's most (or maybe all) of someone's annual salary. There only seem to be 7 people associated with the company [0], so that's a big loss.

[0] http://www.cosmicshovel.com/

If they say 44k $ is a significant loss then i agree they maybe gain less than 1mln. That said i expected them to be a lot more profitable

it's just 3 people and some servers hosting what looks to be one of the top 5k sites in the world based on traffic.

They didn't have a backup?!? Look how much they are paying for data recovery!

Did you read the very short post?

If the only backups they have are so old that they are deemed obsolete, that's essentially having no backups at all.

They said they had backups, but it had been long enough since the last backup they lost enough data for a ~$30k recovery to be worthwhile?

This suggests whatever backups they had might be months old. Given that they are using a roll your own storage solution with consumer grade hardware, I doubt they are rolling in revenue.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact