
Backblaze 7.0 – Version History and Beyond - mrchucklepants
https://www.backblaze.com/blog/backblaze-7-0-version-history-and-beyond/
======
renaudg
In case someone at Backblaze is reading this : I had a less than ideal
experience in the past couple of days, trying to report through normal support
channels a potentially critical bug breaking the "inherit backup" feature in
the Mac client.

Here goes : a few days ago the Backblaze Mac client started prompting me to
upgrade urgently to the latest version (6.1.0.370). I went ahead and did that
but got "Installation could not complete, please contact support" each time.
Support advised me to wipe /Library/Backblaze* , reinstall, then inherit the
existing backup.

Well, we never managed to get the inherit process to complete : it failed with
"ERR_error_unknown" every time. And I started freaking out.

That's where things get interesting : I became frustrated with the canned
responses from support and decided to delve into the client logs myself. I
quickly found a potential smoking gun (in bzbmenu.log) : a spelling mismatch
in an XML attribute name between the server response (support_inherit="true")
and what the client was expecting ("ERROR could not read attribute named
supports_inherit"). Notice the extra "s" ? Pretty obvious typo and one that
could definitely explain the failure (client can't confirm that the selected
backup is eligible for inheritance, bails out)

I was rather happy with my findings and shared them in plenty of details with
support (including log excerpts), but instead of a "thank you and here's some
free months of service !" response, I got "meh, this log file is irrelevant",
and worse: no promise to escalate the issue (until I insisted a second time).
They suggested updating to the 6.1.0.372 beta, which turned out to have the
exact same bug.

Now, I'm left wondering if should try upgrading to 7.0 and inheriting my 6.1
backup from there. But the point of this message is to share my disappointment
with this support experience. I went out of my way as a customer and spent a
couple of hours researching and finding something potentially useful to the
company, but had no appropriate way to report it. I even tried security
contacts, but was (rightfully) told it wasn't a security issue. No word on
whether they were going to pass it on to the relevant team.

EDIT: I've just confirmed that the same issue is present in 7.0.0.386. Still
can't inherit my existing backup.

~~~
atYevP
Yev here -> I'm reading! If you're able, please send me the ticket number so
that I can review that exchange and escalate it. I'm sorry that support didn't
appear grateful for your debugging efforts, but I'll make sure that it gets in
front of the developers who can examine the issue.

I would update to v7.0 - it does have a lot of minor fixes in there as well
(not sure if inherit is addressed, but worth trying).

~~~
renaudg
Wow thanks Yev ! It's #499226

No hard feelings at all against the support guy, who probably has to deal with
dubious customer claims all day. But it was pretty frustrating when he told me
the log file I looked at couldn't possibly have anything relevant in it, when
I had in fact just showed him plenty of relevant things from it :)

~~~
renaudg
Well, not trying to turn this into a support forum any further, but I've just
upgraded to 7.0 and right now SMS 2FA seems broken ("The verification code is
invalid" every time)

~~~
atYevP
Yev here -> I'll do support wherever I see my bat-signal :D Do you have an
alternate method set? If you reach out to support ->
[https://help.backblaze.com/hc/en-
us/requests/new](https://help.backblaze.com/hc/en-us/requests/new) (they have
live chat right now) - they can try an alternate method if our primary
provider isn't reaching your device!

~~~
Operyl
It's completely broken here as well, which was really frustrating. I already
wasted 117 days of a yearly subscription on one machine because it ran
Catalina Beta (which was my fault, I needed a test machine, and I ate the time
lost without complaint), but not being able to inherit that back up really
sucked afterwards.

~~~
atYevP
Yev here -> Have you reached out to support about the issue? They are working
with our devs to track all of the issues and could use the additional data
points -> [https://help.backblaze.com/hc/en-
us/requests/new](https://help.backblaze.com/hc/en-us/requests/new). It's
working for most users, so we're trying to whittle down the issue.

~~~
Operyl
It’s too late now, I already wiped the old backup and reassigned the license.
I didn’t feel like wanting to waste the time much further after past
experiences with support and what I read here.

------
Trias11
Are you (BackBlaze) still happily deleting user's backup files if user's
computer does not contact BackBlaze servers within some arbitrary time?

Yes/No?

Source: [https://help.backblaze.com/hc/en-
us/articles/217664898-What-...](https://help.backblaze.com/hc/en-
us/articles/217664898-What-happens-to-my-backups-when-I-m-away-or-on-
vacation-)

This is malice.

If you are still doing this to your users - your service is a non-starter.
Backup service that deletes user's valuable files under some TOS excuse should
not exist.

~~~
theshrike79
Backblaze is a backup service, not a file storage service.

If the computer you are backing up isn't online for 6 months, it clearly isn't
that important. If it is, you can use B2 to store files indefinitely.

~~~
paranoidrobot
> If the computer you are backing up isn't online for 6 months, it clearly
> isn't that important.

Internet connectivity in some parts is particularly terrible.

My grandmother's ADSL 12/1Mbit connection was never particularly reliable and
would go down any time it rained for days at a time. Every 6-12 months it
would go down hard and not come back requiring a visit - the techs would
fiddle around, find a different working copper pair and get it going again.

Finally, about two years ago it died completely, and the Telco threw up their
hands and said they couldn't fix it, there just wern't any working pairs.
There will be, at some point, a FTTC rollout, but in the mean time we're
getting her limping along with an overly expensive and even less reliable 4G
connection. The data limits are absurdly low, so I can't afford to let
Backblaze actually run backups.

But the data that was backed up prior to the DSL outage we want to keep - I'm
paying the license for it, they should keep it.

~~~
cerberusss
That setup is simply not suitable for using Backblaze, or actually any kind of
off-site backup.

What would be an interesting solution, is to run a simple gigabit ethernet
line to a neighbor or even the backyard shed. And set up a backup endpoint
there. It's not totally off-site but sure beats the alternative.

~~~
Dylan16807
It's not suited for it right now, but it very much disproves "If it's offline
for six months, it clearly isn't that important."

------
elfchief
I've only ever had one problem with Backblaze, in my ... 7 or 8? ... years as
a customer.

The problem: The UI for excluding directories is _horrific_. It uses the
Windows "Browse for Folder" *SHBrowseForFolder) dialog, which requires you
click down through each and every level of your directory structure to get to
the directory you want to exclude. You can't cut and paste, it doesn't start
from the directory you added your last exclusion in, it's just lots and lots
and lots and lots of clicking.

This doesn't seem like it would be hard to fix, so I'm guessing they don't
consider it broken?

If the Backblaze folks are still reading: Could you please, pretty please,
with sugar on top, for the love of god, fix your fucking exclusions interface?
Please?

~~~
atYevP
Yev here -> if you don't want to use the UI (meant for non-tech savvy folks)
we do have advanced exclusions via XML - take a look ->
[https://help.backblaze.com/hc/en-
us/articles/220973007-Advan...](https://help.backblaze.com/hc/en-
us/articles/220973007-Advanced-Topic-Setting-Custom-Exclusions-via-XML).

~~~
elfchief
That's the thing -- I _do_ want to use the UI and I don't want to have to
hand-edit XML. I just want the UI to not use the single worst dialog box in
all of windows.

I am baffled at why Backblaze clings so tightly to the existing dialog box.

------
zomg
i've been a customer of backblaze for a about a decade now. everyone has their
opinion, but for me, it's always just hummed along in the background, doing
it's thing. i've had to restore almost a terabyte of data (on more than one
occasion, derp) -- and had no issues.

it's up there (again, for me) as one of the annual software subscriptions
that's worth every penny, if not more.

~~~
pfranz
I'm curious how you did the restore and roughly how it went? I've done a few
restores years ago. I don't know if this reflects the current process, but my
experience was a bit rough. However, I'm still a long term customer with
little interest in going elsewhere.

In late 2016 my laptop's drive died suddenly. I tried to restore everything
via a shipped drive. I'm not sure if it was because of copy on their website
(at the time) pointing out that they overnight USB drives or me just expecting
that's the fastest way to get my data, but I was disappointed at the
turnaround. Because it was a relatively large amount of data (a few hundred
gigs?) there was a "staging" process that took a day or two. It got
interrupted and had to start again (a server reboot on their side?). After
receiving the drive everything went smoothly; I believe the data had been
encrypted in transit, but wasn't onerous to restore from. I even think I was
outside of their 30-day return and it wasn't an issue (the problem wasn't my
hard drive, but a cable...with no hot spares and a special cable it took
longer to resolve). At work I had a similarly annoying experience with Amazon
Snowball where physically locating it and "testing" it took a few days longer
than expected when expecting tight turnaround...maybe I need to adjust my
expectations for physical logistics.

Many years previous I noticed my music files were showing up as 0-size--my
hard drive was failing. Thankfully I could pull my collection from 30 days
previous and I chose a download option. It was a bit annoying that they had
created a series of zip files (which makes sense), but I believe I was
restricted to the web interface for downloading them which made managing it
difficult.

~~~
yreg
For me the restore experience was painful.

I attempted to restore about 1TB of data (on macOS). Since I have 500mbit
internet I assumed I'd be able to download it through the app, but that didn't
prove easy. Even when split into smaller zip archives, the download would go
terribly slow. Often, the archives would be corrupted and I'd have to re-
download them.

In the end I had to order a drive to Europe (and pay the tariffs), which is a
pain in the ass. (But they did return the deposit even though I missed the 30
day window.)

Edit:

Oh yea, and when you loose your data you have only 30 days to get it back
before Backblaze deletes it as well! That's what happened to me right before a
multi-week trip, so I was pretty unhappy that I wasn't able to download my
backup in the 5-or-so days I had at my disposal, thus having to order the
physical drive.

~~~
buildzr
> when you loose your data you have only 30 days to get it back before
> Backblaze deletes it as well!

Uhhh... this one could be a significant issue for me right now. I've got a
laptop that's been offline for about 2 months now due to a motherboard failure
due to liquid damage. I haven't bothered dumping the drive yet because I
figured even if it's got issues Backblaze has a copy.

You're telling me that if I login to my account right now, the data is gone?
If so, they really need to make that one more obvious.

EDIT: Just checked Backblaze, still looks like the data is there after 70+
days. Has this policy changed at some point?

~~~
brianwski
Disclaimer: I work at Backblaze.

> Has this policy changed at some point?

It is more complicated than just "after 30 days" (and always has been). If
your laptop is entirely offline (or you simply uninstall the client from your
laptop), then the policy is ACTUALLY that we keep your backup for 6 months as
long as you keep paying your bill. But to be honest, it's more like a year or
two. The 6 month policy is we guarantee the backup will be preserved, not that
we will immediately go out and delete it at exactly 6 months and 1 day.

This is a completely different situation than files you delete but the backup
continues. For cost reasons, Backblaze purges the files that you deleted from
your local drive after 30 days. HOWEVER, with this new 7.0 release you can pay
a little more and increase that retention time to one year, or forever. This
was a highly requested feature for the situation you ran into.

Or, if you have a hard drive that you disconnect and aren't willing to
reconnect for more than 30 days. Since your backup is continuing, Backblaze
assumes the drive will never come back, so the files are then deleted to save
money in the datacenter. Of course, this changes with the 1 year retention
policy, you can unplug the drive for up to 1 year and still dial back time and
restore all your files.

~~~
justsubmit
> HOWEVER, with this new 7.0 release you can pay a little more and increase
> that retention time to one year, or forever. This was a highly requested
> feature for the situation you ran into.

I'm glad to hear that Backblaze has finally made such an option available.
However:

> For cost reasons, Backblaze purges the files that you deleted from your
> local drive after 30 days.

As someone who once almost lost his PGP key and had to recover it from old
physical backup media, I'd like to point out that that is not a "backup"
service. It's sort of like a lazily expiring mirror, but it's definitely not a
backup service.

Do you look at all of your files every 30 days? How long would it take you to
notice that a random file had disappeared from the filesystem 7 layers deep?
Have you ever needed to restore a years-old file?

(You need not explain why you do it; I understand about users who could use it
as a cloud storage service by deleting files after they're backed up. The
point remains that it's not a _backup_.)

~~~
pfranz
What's your criteria for backup? I completely agree that something like RAID
is not backup, but 30-60 days seems comparable to other backup services and
covers most scenarios. The old-school manual process of swapping a USB drive
at work every week/month has the similar retention.

The only consumer-facing system I've used with multi-year retention is Time
Machine. Every few years it has trouble "verifying" the backup and I have to
start over (it also deletes old backups when you run out of space). Right now
my backup only goes to August.

~~~
sjwright
Backup means that if it takes me 6 months to realise that the 2014 finances
folder had been accidentally deleted, I can still recover it.

I'd love to see BackBlaze offer the option to select "mission critical"
folders that have super-long-life (Time Machine-like) version retention. Give
everyone a few gigabytes of this for free, charge a premium to increase that
space.

~~~
Dylan16807
Though that sounds tricky to set up all the UI for. For now you can set up a
separate backup of that stuff. Set B2 as the backend and you get 10GB free.

~~~
sjwright
Maybe—and the UI could be little more than a script that does exactly what you
described.

The point of making it a UI is, first and foremost, to get people thinking
about this question. And it gives people the option to acheive what you just
described without stumbling upon a post like yours on Hacker News.

~~~
Dylan16807
A post like mine? But it's your post that has the idea...? The only thing I
said about how to make it possible is "separate backup", which anyone can
think of in two seconds and is only a hint of a tenth of an explanation. I
don't understand.

------
maikhoepfel
Cool changes! Any plans to ever add Linux support? That’s a dealbreaker for
me. I need to backup both Linux and Windows machines. I wanted to use B2, but
restic seems not ideal on Windows.

~~~
atYevP
Yev here -> No plans on the Computer Backup side. We haven't been able to come
up with a way to do it affordably and in a way that makes sense for us - so we
recommend folks use B2 Cloud Storage and any of the integrations that work for
both. Cloudberry and Comet backup work pretty well. You can see more on:
[https://www.backblaze.com/b2/integrations.html?platform=linu...](https://www.backblaze.com/b2/integrations.html?platform=linux&use-
case=backup) (I set it to use Linux + Backup as the parameters).

~~~
cweagans
B2 isn't really a solution for people that are looking for a robust computer
backup service like Backblaze currently offers for Mac and Windows. I went
down that road and ended up very frustrated.

I ended up buying a Synology NAS (which _does_ offer the same computer backup
features + revisioning and all that), and then syncing the backup out to B2.
It's not an ideal solution by any means, and I would get rid of my NAS in a
heartbeat if there were a real solution for Linux desktops.

It really sucks to have had to go through all of that setup (+ have another
machine to maintain) when I just wanted to get things done and know my data
was safe.

Would you consider working with the community to find a better solution? Maybe
offer a closed source library (a la libspotify, but for Backblaze) for the
storage bits + get the community pointed in the right direction for an open
source frontend for Linux or something?

~~~
calcifer
I imagine the problem isn't closed vs open source, but the fact that on Linux
it would be trivial to mount a massive NAS (think dozens of TBs) in a way
that's completely invisible to the Backblaze client and pay $6/month for
unlimited backups. There were people on /r/datahoarder who stored _petabytes_
on Amazon's "unlimited" backup before the whole product got canceled.

~~~
clSTophEjUdRanu
You can do that on Windows too can't you?

~~~
calcifer
I'm not familiar enough with Windows to answer that, but I imagine HTPCs are
much more likely to use Linux and would make a $6/month backup a very
attractive target.

~~~
brianwski
Disclaimer: I work at Backblaze.

> Can you abuse Backblaze on Windows or Macintosh?

Backblaze lives on the "averages", and so far the averages for Windows and
Macintosh have worked out for us. We don't have any deep pockets (no VC
funding) so we have to stay profitable or die. If you are curious what the
distribution of customer sizes is, here is a histogram (you will need to zoom
in, then use the scrollbar):
[https://i.imgur.com/iVEuwUT.jpg](https://i.imgur.com/iVEuwUT.jpg)

Backblaze does not allow running the "Personal Backup Client" on Windows
"Server" OS flavors to help the averages. Macintosh are really super
successful in the laptop category, but very very few people or businesses run
them as servers so again, the averages work out for us (so far).

However, in all of our market research, the Linux users averages would quickly
drive us out of business. By definition, Linux is pretty much dedicated to
almost exclusively servers, and exclusively dedicated to SUPER technical
people who understand exactly how much data they have. We built "Backblaze B2"
expressly for these technical customers running Linux. If you know how much
data you have, and you have less than average, you will literally save money
with Backblaze B2.

~~~
social_quotient
I appreciate the directness and transparency of your reply and backblaze in
general.

I also appreciate your focus on providing the value you know you can to your
core audience. Compromising that attention which, as you noted, would possibly
jeopardize your ability to exist at all. A ton of companies get this
completely backwards.

------
sanbor
Any plans to restore an encrypted backup without having to send the private
key to Backblaze servers?

~~~
Dylan16807
Looks like yes, but it's not a particularly high priority, based on
[https://news.ycombinator.com/item?id=21205948](https://news.ycombinator.com/item?id=21205948)

------
ryanmercer
I really miss Backblaze. I was a customer for years and years and actually was
fortunate (I guess?) enough to have to use them to recover around a terabyte
once but when I switched to using a Chromebox as my daily driver (then a
Chromebox 2) I found I was turning my Windows machine on once a month just to
keep kosher with the Backblaze servers.

Such a great service and I still always look forward to the emails, stuff like
the drive failure rates is always a neat read!

~~~
lostlogin
> I found I was turning my Windows machine on once a month just to keep kosher
> with the Backblaze servers

I’ve been burnt badly by this, I wish there was a better way. I was in the
habit of doing this and when I fired the drive up for some reason it failed to
be backed up and the backup was deleted at their end. 1TB had to be re-
uploaded on a 4Mbps connection. I know I shouldn’t have done the backup on the
last day (no data had changed) but wow was that irritating.

~~~
ryanmercer
Fortunately for me I realized none of the 5~ terabytes on that machine
mattered whatsoever as I never actually used any of it.

I would archive podcasts I liked, download YT videos I found worthy of it, I
had 100gb or so of 9/11 news coverage from the day of the attacks and the next
few days, insane amounts of exes for pretty much every revision of every scrap
of software I used in case I ever wanted to go back to an old version, insane
amounts of images from where I'd rip entire tumblr accounts based around
different fandoms/topics, I had half a century of Lodge meeting minutes for
one of my Lodges scanned as high-resolution OCRd pdfs, entire websites I'd
wget for offline (why?!) viewing etc. It's all presumably still there but I
haven't turned that box on in about a year now.

Man, switching to a Chromebox was so freeing.

------
tpmx
Do they still have a very slow upload performance with their backup service?
That's been the main complaint I've been reading over the years. (I'd probably
be happy with something in the range of 50-100 Mbit/s.) (My home connection is
250/100.)

~~~
atYevP
Yev here -> try it now. We've spent the last few updates honing in the way we
handle uploads and have introduce threading to make the process more
efficient. We'll always be bottle-necked by the available bandwidth to your
machine and latency, but it's worth trying again if you haven't for a while!

~~~
tpmx
Sure, but you surely you have stats? And why is latency a factor - surely you
can work around that by improving your network protocol (think less TCP, more
QUIC or something similar, etc)?

Anyway, okay, I'll test your trial again. :)

~~~
atYevP
This speedtest is somewhat accurate and is pointed to our data center so
should give you a decent idea of speed ->
[https://www.backblaze.com/speedtest/](https://www.backblaze.com/speedtest/).
The reason latency is a factor is that it affects how quickly data leaves your
computer and gets to our data center, so if you're far away, it can play a
factor!

~~~
lubblig
Unless I missed some setting it seems like the speedtest is only for the US
data center(s). Is there any way to test the speed to the EU data center?

~~~
brianwski
Disclaimer: I work at Backblaze.

> Is there any way to test the speed to the EU data center?

Create a free Backblaze account in the EU datacenter region. Then
upload/download some data! The first 10 GBytes of B2 is 100% free, you don't
even need to give us a credit card so you can try it all utterly risk free
with no way we could mess with you.

If you run the Backblaze Personal Backup, make sure you manually configure it
to use 30 threads, then let it rip and watch your bandwidth meter to see what
it can use. The first 14 days are completely free, again no Credit Card
required! Personally I get about 100 Mbits/sec from my home in California to
the European datacenter (capped by my ISP), but if you are in Europe you
should be able to hit 500 Mbits/sec using the Personal Backup Client with 30
threads if you are close enough to Amsterdam (where the datacenter is). Oh,
one hint -> don't judge Backblaze Personal Backup until you have been backing
up for at least 12 hours. Backblaze backs up small files first, and the
latency to push 1 byte files murders performance. But after we get through all
your small files, it should rip.

In the end, everybody has enough bandwidth to reach everywhere in the world
now, and you really shouldn't pick a service based on total throughput. I
think you should pick it based on cost, comfort with the security model and
how sensitive your data is, ease of use, etc. Unless you have a Petabyte or
more of data, you can backup hundreds of TBytes to anywhere in the world
nowadays. Easily. No problem.

------
semicolon_storm
Very surprised it took this long to implement. I’ve had a good experience with
B2B + Duplicati, but the main reason I went with that over the simpler
Backblaze Cloud was the retention policy. 30 days to recover files was just
too short of a time span to be a guarantee that my files were safe.

~~~
atYevP
Yev here -> Glad you're enjoying the Duplicati integration with B2! It did
take us a while, but we wanted to make sure that we did it the right way and
in a sustainable fashion. It's been an ongoing project for the past 10 months
or so - and came after a lot of customer demand following CrashPlan leaving
the consumer market. We're hoping people dig it, apologies it took so long!

------
novok
Does backblaze co. have access to the contents of your backups or is it client
side encrypted?

~~~
atYevP
Yev here -> the backup service is client-side encrypted before transmission.

------
CharlesW
Yev, can you clarify whether this new release supports NAS? I see that it
allows you to "Back up all your attached external drives", but my drives
happen to be connected via Ethernet.

~~~
scarface74
Probably not. They’ve said repeatedly that unlimited storage for $5 wouldn’t
be a sustainable business model if they supported NAS. For that there are
third party solutions that support backing up to B2.

~~~
CharlesW
Thank you, I know the history and what they've said in the past, and also know
about their B2 product. I'm hoping for a real answer from Yev, but if not I've
been reasonably happy with CrashPlan (which is a NAS-friendly solution).

~~~
scarface74
TIL:

Crashplan doesn’t support backing up NAS on Windows but does for Windows and
Linux. Interesting....

That being said, I think if I had a NAS, I would want a solution that runs
directly on the NAS instead of depending on a host computer.

~~~
CharlesW
> _…I think if I had a NAS, I would want a solution that runs directly on the
> NAS instead of depending on a host computer._

Yeah, you could totally do that using a VM on the NAS. The only downside is
that it'd be charged as an additional device, assuming you still wanted to
back up your main workstation.

------
jbergens
Hm, if I turn on 1-year-history or history-forever and includes my
node_modules folders in the backup will I fill their whole storage cluster?

~~~
atYevP
No, but don't do that. :P

