Hacker News new | past | comments | ask | show | jobs | submit login
How Backblaze Got Started (2017) (backblaze.com)
101 points by slyall on Feb 3, 2018 | hide | past | favorite | 68 comments

I see a lot of praise for Backblaze’s services and support and I just wanted to warn that I’ve had the opposite experience and am in the middle of switching to another service.

I’ve used Backblaze since 2010 and in the past year contacted support twice. In both cases, they were only interested in quoting esoteric policies and technical limitations rather than actually solving my issue. As in, “Oh that’s because our system works like this and this and has this limitation.” Ooookay, that’s not what I asked; I asked how to solve this problem of mine that your poor system is causing.

For example, the most recent time was when I started getting emails warning that all of my external drives were offline and my data would be soon deleted. Instead of “Very sorry about that, here’s how to fix the issue,” I got this long response about the ways their system looks for new files in serial and it can get jammed and start ignoring everything, with no apology, no acknowledgement this was their issue, and no solution. I had to go fishing for solutions and drag the information out of them to finally figure out what I needed to do. Which it turns out is to get back an internal drive (totally unrelated to the other drives Backblaze abandoned) I had physically removed and repurposed, put it back the computer, wait a long time for Backblaze to see it, then uncheck that drive in Backblaze and remove it again. This was a good example showing how the client design is poor and the support is worse.

The other two major technical issues that convinced me I need a new service asap are:

1. The client will lie to you and you never know what’s really backed up. Even if you use the secret alt-click to force a full drive scan, it can still miss files and tell you fully backed up when files from days ago are still nowhere to be found. Luckily I’ve never actually needed to do a restore, but I almost thought I did one time and would have been furious at all the missing files I noticed.

2. This might not matter to some people but absolutely no meta data is backed up.

I hate to badmouth them since the company seems like a really good citizen, they project a very positive image, and their software seems good on the surface. But when you dig in, there are a lot of issues with the software and service, and I recently realized having Backblaze wouldn’t actually reassure me in the event of a local disaster, so migrated away and I wanted to warn others too.

I had an extremely disappointing experience with Backblaze's support team (documented here - http://www.cederman.com/2016/12/tracking-down-rogue-bandwidt...), and in the process of researching the issues I was having, discovered a rabbit hole of reports of folks having issues restoring their data.

Backblaze have not acknowledged my issues at all, even though I have tried to engage with the team numerous times.

I switched to CrashPlan, and have had to do two full restores since doing so, both of which went flawlessly and quickly. The Java client is a bit cumbersome, but it's reliable, and I am much happier with the performance and support I've received from them.

Just another data point on CrashPlan...

I've used them for a long while and previously had good experiences.

I was recently adding a new backup to CrashPlan and was a bit miffed that my backup is capping out at around 3 Mbps (and down to about 500 kbps at times). Opened a support ticket, went through a whole bunch of things, and they basically came back saying 1 - 5 Mbps is the expected range.

In 2018, I really don't think that's acceptable when you're trying to backup a couple terabytes of data. I have not had this issue with other backup services I use.

I loathed Crashplan’s client. On a cheapish Windows laptop it would absolutely clobber the day-to-day performance of the machine. The machine was 2-3x faster when I finally uninstalled Crashplan. It seemed to be polling small files continuously.

Crashplan is killing off their home service, though. I've switched to Arq + Backblaze's B2, we'll see how that goes.

I gave Arq a go but usability wise it has some catching up to do. It's hard to find an arbitrary file that's been deleted to restore unless you know exactly where it lived on disk. The search didn't work too well for me.

Yev from Backblaze here -> After the CrashPlan announcement, we've seen a large uptick in that exact combination! Seems to be working really well for folks!

Yev from Backblaze here -> Hello Tim! I'm very sorry to hear you had a bad experience with support. If you still have it and are able to respond w/ a ticket number I'd love to take a look and get it to folks who can use it for training.

Not suggesting you switch back off of CrashPlan (we like those folks and they do good work) but we've changed quite a bit since your last interaction with us and the issue you were experiencing sounds like it may have been one of the burrs we've able to knock off. Again, I'm sorry you had a poor experience with us, but glad you're using an online solution that's working well for you!

Backblaze is so overrated; I really don't understand why so many high profile computing people have endorsed it.

Last year they lost one of my drives — the whole backup of the entire drive 100+ GB and support just threw their hands up. Of course I only discovered that they lost it after the physical drive failed.

Still looking for a better replacement. Arq didn't quite do it for me.

> I really don't understand why so many high profile computing people have endorsed it.


Yev from Backblaze -> Ted - it sounds like you hit the 30-day retention period with the drive (if we don't see a hard drive and/or data for 30-days we remove our backup copy of it). Is it possible that hard drive was disconnected before failing? We'd normally send warning repeated warning emails when we see the drive not connected for an extended period of time.

The drive was connected regularly frequently specifically because of this particular issue with Backblaze which was a pain point for years. The software did not report any of my drives as missing and the other partitions on the same drive were not lost :/. IIRC it was a bug in the OS X client. Still, what good is a backup service that can't recover one's backed up files? The least support could have done is refunded my plan which is much less than the cost of the lost data but they didn't even do that. This combination is how you lost a customer. -Taylor

Ted thanks for the additional information, I'll make sure it gets forwarded to the right folks.

I use Arq with no problems. Any specific issues you could list?

Yes, from my other comment:

> I gave Arq a go but usability wise it has some catching up to do. It's hard to find an arbitrary file that's been deleted to restore unless you know exactly where it lived on disk. The search didn't work too well for me.


Having to do a few restores like that was pretty painful with the Arq UI. I ended up giving up on it and finding the data another way.

Thank you for sharing your experiences with Backblaze. I've been debating which cloud back up service to go with and your comment has started to sway me away from Backblaze. It is also disconcerting to see the employee from Backblaze only responding to the positive comments below.

Yev from Backblaze here -> non-chronological view makes it tough, but those were the only comments on this thread when I check it last night! I've been sweeping the rest this morning! :D

Your comment describes my recent interaction with them perfectly. I suddenly got a big fat warning about something, yada yada long story short “oh yes but that’s just how our system works, and I can’t give you access to the diffs because that’s how our system works, you have to do X Y Z now.” No “we understand our design decisions are causing you a problem which shouldn’t be yours” or anything like it.

Let me know if you find anything worth switching to :)

( tangent: word of warning: it seems to silently ignore folders with the same path except for case on case sensitive disks. Fringe issue, but I only accidentally saw my two most important folders weren’t backed up.)

Disclaimer: I work at Backblaze.

> I suddenly got a big fat warning about something, > yada yada long story short > ... I can’t give you access to the diffs because > that’s how our system works

I would love to understand what the specifics were. Backblaze's whole design is based on a "log format" where you really SHOULD be able to understand the diffs and exactly what occurred at what days and times. In other words, a complete, total, absolute history of everything that occurred is found both on your local system, and then (encrypted) on our servers.

What I mean by "log format" is that when a file is backed up, a line is added to what we call the "bz_done" files that says something like this:

+ /pictures/puppy.jpg

The "+" means it was added to your backup. Then if you delete that file later we don't REMOVE that line, we add an additional line to the very end of the log like this:

- /pictures/puppy.jpg

The "-" signs means it was removed FROM YOUR LOCAL DISK (but not yet from the backup). Then 30 days later, the final line might appear like this:

x /pictures/puppy.jpg

That means the "30 day roll back history" period has expired and the file has been cleaned up from the Backblaze datacenter.

The whole point is we can determine exactly what occurred at what times, and you can understand the diffs yourself.

BackBlaze is very similar to AWS and similar services - it's not made for the consumer. You need to read and understand their entire stack, process and features available. If there's a single limitation that won't fit the bill - find something else.

They'll suit 80-90% of customers, but I'd say there's a portion that don't do their due diligence and test every use case they have (which is fair enough, most people don't and most people don't know every use case they will have going forward).

> This might not matter to some people but absolutely no meta data is backed up.

It matters to me. There's a lot of metadata on MacOS files and this is the reason I don't use Backblaze.

Disclaimer: I work at Backblaze.

> absolutely no meta data is backed up

> There's a lot of metadata on MacOS files

To be perfectly clear, Backblaze backs up and restores TYPE and CREATOR metadata (on the Macintosh), and restores the file creation date and the file last modified date. Backblaze does not backup anything else, for example, what color you chose for your icons in the Finder, and Backblaze does not restore the file and folder permissions.

This works for most customers, but some customers won't be happy with it. We just want to be clear what we do and what we don't do.

The concept of "file creation date" and "file last modified" are fairly universal, and can be restored "across platform". What that means is if you backup on Macintosh you can restore to Windows and vice versa. If you restore a Macintosh TYPE and CREATOR to Windows it will lose that data since Windows (NTFS) does not have that concept.

Windows has the most sophisticated (complicated) set of ACL (Access Control Lists) that anybody has ever seen, and frankly I've never met anybody who could fully explain why Microsoft went crazy on these (and Microsoft. Backblaze does NOT backup the ACLs on Windows. But NTFS and Microsoft ACLs will simply never survive the test of time. In 30 years your iOS phone or Android tablet will never be able to understand what occurred and why "joe" had permissions as "System" while "jane" only had permissions as "Administrator" on a photo of a puppy even though those logins are long gone.

Most people who are worried about backing these sorts of things up should look into "Drive Cloning" software, which is different than "backup against catastrophic long term photo loss". On Macintosh you can look into "Carbon Copy Cloner". Not sure what to use for Windows?

Yev from Backblaze here -> Not sure it would have fixed your particular issue, but we recently introduced a new client that fixes a few outstanding issues with the way external drives were being handled - it sounds like it might have helped in your case.

I'm sorry you had a bad experience with support, if you can post your ticket number, I'll take a look at it and make sure that the head of support sees it so they can do training on it if necessary.

> Then one of our co-founders Tim wondered, "what if we didn’t ask any questions and just backed up everything?"

But that is not what they do exactly:

"we do not backup your operating system, application folder, or temporary internet files that are transient and would not be useful in the future. Backblaze also excludes podcasts in iTunes."


Is there an online backup service that makes a complete image that you can just pop a boot CD in, and restore from online? That seems like the holy grail.

Yev from Backblaze here -> Backblaze isn't intended to be a bare-metal restore, just the data that's on the machine. In addition to the ones listed below, you might also take a look at Acronis - they do good work. Not sure if they have an offline component, but you could use their software to make a system image, and then toss that image in the cloud (like with Backblaze B2) and then grab it whenever needed. Might work well for what you're looking for!

It's not online, but you might checkout RedoBackup... seems like it could be modified to be used this way.

Check Veeam.

The rise of cloud storage has totally transformed the "backup" problem. Just by using cloud storage, you get near-realtime offsite backup without thinking about it. Vendors like Apple have built it into the OS. It's really one of the nicer things that's happened in personal computing lately.

Granted, there's still benefit to doing a traditional "snapshot" style backup. Something like Backblaze is still a good idea. But arguably it's more of a belt-and-suspenders thing at this point. (Especially if your cloud storage offers version history.)

While I've loved Backblaze's blog posts over the years, seems like this was not on their radar ten years ago. They were seeing it just as a traditional backup/restore problem. I see they've since introduced a cloud storage product (B2), but it doesn't sound that competitive.[1] Here's iCloud for comparison:

- $0.005/gb/mo (2TB plan) with unlimited data transfer

- Baked deep into the OS, totally automatic

- Intelligently purges local files when storage runs low

- Integrates with Photos app for optimized media storage

- Syncs while the device is asleep (Power Nap)

- HSM-protected keychain backup (geeky but groundbreaking security feature)[2]

(Not so nice: doesn't do version history. Dropbox does.)

Again, I've really enjoyed Backblaze's posts and hope they continue thriving in this new landscape.

[1] https://www.backblaze.com/b2/cloud-storage.html

[2] https://www.blackhat.com/docs/us-16/materials/us-16-Krstic.p...

> The rise of cloud storage has totally transformed the "backup" problem. Just by using cloud storage, you get near-realtime offsite backup without thinking about it

Then an unaccountable algorithm closes your Google account, and suddenly all that data is inaccessible. Your cloud storage provider's backups are not your backups.

> Then an unaccountable algorithm closes your Google account, and suddenly all that data is inaccessible.

Then a fire destroys your local backups, and suddenly all that data is inaccessible.

Like I said, traditional snapshot backups are still a good extra step, but cloud storage has brought very good backup practices to the vast majority of the population. The difference versus ten years ago is probably several orders of magnitude.

I agree. Cloud is OK as backup, as in if it goes away that’s ok, I can still create a new backup from the live data. But not for live data. I found having a synology NAS to be a good solution. I sync my data mostly from home anyway so it is fast (runs locally) while also being accessible from the outside.

Yev from Backblaze -> we're seeing a lot more folks go the NAS route, and so we're trying to work with all of the NAS providers to get B2 integrated into their ecosystems. We love the idea of having "working files" on the computer a "local archive" on the NAS, and then files forked off to B2 for an offsite component. 3-2-1 Backup at its finest!

You’ll find that this happens a lot less often when you’re a paying customer instead of a freeloading target for ads.

As a CrashPlan customer, I'll soon be moving to some other service when they shut down. As a Mac user, iCloud is an option. However, I'm unlikely to choose it because Backblaze or similar costs $5/mo instead of $10/mo for the amount of storage I need, and because I don't like the "cloud" storage model of treating your local drive as a cache and assuming a fast, always-on internet connection. I like to know that my files are on my local SSD, not on the other end of a flaky or nonexistent internet connection. "Intelligently purges local files" sounds like a nightmare.

> iCloud is an option. However, I'm unlikely to choose it because Backblaze or similar costs $5/mo instead of $10/mo for the amount of storage I need

Note: Backblaze charges a fairly hefty $0.02/gb to download, I believe, so restoring your 1TB backup will actually cost you $20.

But my bigger point is, people are moving to cloud storage for other reasons (i.e. syncing devices, optimizing local storage) and they're getting pretty good offsite backup along with that "for free". iCloud is nicely priced if you're actually interested in cloud syncing. At 2TB it's even the same storage price as Backblaze, which is amazing, because it's a full-on intelligent syncing service that doesn't charge for data transfer either.

I'll also note that intelligent purging is completely optional. There's a checkbox to turn it off for documents and/or media (separately). But if like most people you have fairly reliable internet, it's a real benefit. It makes it feasible to put your whole photo library in the cloud (mine is >1TB) and have it all immediately accessible without having to splurge on massive local storage.

Disclaimer: I work at Backblaze

> Note: Backblaze charges a fairly hefty $0.02/gb to download, I believe, so restoring your 1TB backup will actually cost you $20.

Backblaze has two product lines, priced two separate ways:

1) Online backup for $5/computer. THIS IS FREE TO DOWNLOAD.

2) "B2" - this is "object storage" for half of one penny per GByte per month. This costs 2 cents/GByte to download. If you want to use B2, you cannot use the traditional "Online Backup Client" from Backblaze, instead you can choose a piece of software from this integrations page: https://www.backblaze.com/b2/integrations.html (scroll down, it is long). You can see pricing "per transaction" here: https://www.backblaze.com/b2/b2-transactions-price.html

Side note: the $0.02/GByte download price is definitely over-charging, and we're working to lower that. We pay 1/10th that amount wholesale for bandwidth. The fear was that if we drop our prices much more, we would become the world's cheapest CDN (Content Delivery Network) and if that occurs too fast it will overwhelm our download servers until we build out that infrastructure and gain some experience with running the world's least expensive CDN.

Thank you for the correction -- I wish I could edit my comment. $5/mo for unlimited storage and data transfer is definitely the best price in town for traditional backup.

> I wish I could edit my comment.

:-) I share your pain. I just found my own typo in a different comment and now my shameful proof reading skills will be preserved for all of time.

Yev from Backblaze -> "Intelligently purges local files" sounds like a nightmare." Anecdotally we've had a lot of customers really confused and get bitten by this when they're traveling and try to access a bunch of things that are no longer "local".

iCloud is definitely a good option but we wrote this up a while back -> https://www.backblaze.com/blog/sync-vs-backup-vs-storage/. We view it still as more of a Dropbox/Syncing solution. If you work 100% out of your iCloud account you could make the argument that all the data's there, but in all of our research and polling, that's not the case (even when folks have the 2TB plan - they still tend to store data all over their computer). It'll definitely be interesting to see how the service develops. We have a lot of folks that are integrating with Backblaze B2 that are doing interesting things as well! It's definitely an exciting space to be playing in right now!

> If you work 100% out of your iCloud account you could make the argument that all the data's there, but in all of our research and polling, that's not the case... they still tend to store data all over their computer

How recent is this research? Starting with Sierra (late 2016) Apple is including the entire Documents and Desktop folders in iCloud Drive by default.

Fairly recent (we do yearly surveys). We certainly do see those services getting more use, but there's still a long way to go.

Backblaze is one of the few companies where I pay for a service and am 100% satisfied. I've never had an issue with the service and when I've had questions they are prompt and informative. I use them for personal and business backups.

They are more than happy to share what they build and we've used their storage pods plans to build 2 for work.

I felt that way until I needed to complete a restore of a backup that was roughly 500GB in size. It took around two weeks to restore on their servers for me to download as a zip. I posted on here about it at the time and the CTO reached out to let me know the server the restored started on was very old and they were going to decommission it afterward. I hope they did. I hope others don't wait that long.

I also really wish they would let customers buy a hard drive with multiple backups on it instead of one hard drive per backup. It would be great to purchase one drive every year with all the backups on it and not have to worry about downloading backups for retention.

Yev from Backblaze here -> sorry to hear about the long restore process. We definitely take restores seriously and our latest release 5.0 (https://www.backblaze.com/blog/cloud-backup-5-0-rapid-access...) addressed a lot of bottlenecks that could affect some customers. You can now also restore-by-mail and send us that hard drive back for a refund, so should you try us again in the future, you'll have more options!

Wouldn't just asking them to send a hard drive of your backup have been faster? You could have either kept the hard drive or returned it and got your money back.

Since the server (and not the Internet connection) was the bottleneck, it wouldn't have helped, as they would still have to read the files to copy them to the new drive.

Disclaimer: I work at Backblaze

> Since the server (and not the Internet connection) was the bottleneck, it wouldn't have helped

As Yev mentioned above, we MASSIVELY sped up restores fairly recently (August 2017). We understood waiting for restores for a few days was an inconvenience for customers and I'm personally very proud of the results. Would LOVE for you to try it again, but I fully understand if you don't want to. Here is the blog post about the faster restores (and faster backups): https://www.backblaze.com/blog/cloud-backup-5-0-rapid-access...

I never really thought about this until just now but your 100% right. I use both B2 and their consumer service and never had any issues.

Yev from Backblaze -> Great to hear that from both of you! :)

In general it's a bad look to respond to the good reports but not the bad ones.

Happy to chat about all of them - these were the only ones I saw when I check here last night if you look at the times!

There's a good moral here. Everytime they had a question, they asked their users. Users don't lie.

Yev from Backblaze here -> That's right! For better or worse they usually tell it like they see it, and that helps us inform decisions!

Yev, I’m a new Backblaze customer and am worried by some of the negative reports by other HNers above.

Would you mind taking the time to respond to the less positive comments about Backblaze rather than just thanking the two positive commenters? That would make me feel much more comfortable relying on Backblaze in the future.

Hey there sorry - if you look at my reply history you'll see I try to hit most topics, but when I looked last night those were the only comments there :D I'll get to the rest!

Such a great service that I’ve never used it for its intended purpose (thankfully, except for very occasional verification), and forget about it until I get the “we’re charging your credit card” email. I then delete the mail, happily let the charge go through, and proceed to forget that it’s there. I backup to a NAS, from which important stuff is backed to another, single drive. Backblaze is a small price for the “house burns down” scenario. Yeah, I could rotate drives offsite, but I’m not going to do that, and neither are most of us after a couple months, if at all.

Yev from Backblaze here -> glad to see you're at least trying the restore process! You'd be surprised how many folks forgo that in its entirety and then panic should they need to restore. Always good to check the system out from upload to download! Kudos!

I use B2 (sparingly so far) for personal storage and it's been a good experience. I prefer B2 since it's permanent and very inexpensive per GB.

I cancelled their account. It has a weird policy of deleting backups from hard drives that have not been connected within the last 3 months. Very weird policy.

It’s one month, not three, but the policy is totally reasonable since they charge one flat rate to back up your whole computer (not everything your computer has ever seen). If they didn’t delete disconnected drives, they’d be offering unlimited storage for $50/year which is obviously unsystainable. The plan is already as impressive as it is considering how much data you can easily connect to one computer.

I also have plenty of complaints about the service (posted above), but there’s nothing weird about this policy as written.

What’s weird is when permanently connected drives get classified as disconnected (see my other post).

Whilst I agree the policy is reasonable given the amount they charge per month, it's something I've only learnt about from reading comments and not from their marketing.

In fact, their marketing claims "Never lose a file again. Get unlimited cloud backup for your Mac or PC. Just $5/month."

Only on reading the help does it seem that if the client doesn't connect with the server within 6 months, then all your backups are gone. I'm not saying this is unreasonable, but it seems a bit counterintuitive given you are still paying for the service. I just still haven't found the bit about disconnected drives.

I really like Backblaze's articles on HDDs and they really seem to be customer focused, but I had to dig in the help to find this.

I don't use Backblaze, but I suspect from the above comments that the client does not even display some kind of warning saying, "Hey, you haven't connected this drive for X months. We just wanted to let you know that this data will be deleted from our systems in Y days. Please connect the drive to make sure we retain the data." and then proceed to display this warning periodically a few times, like a countdown (but not annoying enough to be shown on every run of the client). This kind of warning would be my suggestion to the Backblaze team. This becomes more important with SSD based machines that generally have a lot lesser storage than before while people are taking a lot more pictures and videos and resorting to storing them in cheaper (per GB) external HDDs (at least the people who do like to have a local copy).

Yev from Backblaze here -> If we detect a missing drive we send a warning email at the 14-day, 21-day, and 28-day mark to remind folks to plug them in - or create a restore if they no longer have the drives. If they still don't plug in the drive after those days, then the backup does get removed, but we do try to warn them ahead of time!

I _Believe_ the client displays this information, buy I could be wrong about that. I know I get a few helpful emails, starting with a couple weeks notice though.

Slightly tangential. I usually don't trust any company that offers something "unlimited" for a limited amount of money. I usually suspect, and believe, that they're overselling their capacity and then have the low volume users subsidize the high volume ones (so the former are paying a lot more than they should be). Such companies also tend to have undocumented limits that you'd know only when they tell you that you've hit them (and that limit would be dynamic).

For something as critical as data backups, I prefer that those companies stay in business and charge something that's a bit more proportional to their underlying costs. That's honesty and fairness, in my view.

Yev from Backblaze here -> We've been running on the unlimited model for a long time now and it's working well for us. It's like a buffet, some folks eat more, some folks eat less, but as long as the average is below our costs to provide the service we'll keep doing it. We DO have a cloud storage offering (similar to S3) which allows folks to store data indefinitely (or as long as they want it) and that one costs $0.005/GB!

Yev from Backblaze here -> We do have a 30-day retention window. So if the data is on your computer, or the externals are connected we'll keep a copy of it, but if the drives are removed and/or the data is removed from your computer, the 30-day timer starts. For an archive solution that keeps data for as long as you want we built out Backblaze B2, which is $0.005/GB!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact