
I Nearly Lost All of My Data - kevq
https://kevq.uk/i-nearly-lost-all-of-my-data/
======
rsync
Hard to believe we are 129 comments into this and there are zero mentions of
'borg' in the comment threads ...

For those that don't know, borg is a backup utility[1] that has been called
the "holy grail of backups"[2].

It takes your plaintext files and directories, chops them into gpg-encrypted
chunks with encrypted, random filenames, and will upload (and maintain) them,
with an efficient, changes-only update, to any SFTP/SSH capable server.

My understanding is that the reason people are using borg instead of duplicity
is that duplicity forces you to re-upload your entire backup set every month
or two or three, depending on how often you update ... and borg just lets you
keep updating the remote copy forever.

[1]
[http://borgbackup.readthedocs.io/en/stable/](http://borgbackup.readthedocs.io/en/stable/)

[2] [https://www.stavros.io/posts/holy-grail-
backups/](https://www.stavros.io/posts/holy-grail-backups/)

~~~
m3nu
Let's also mention Vorta, an open source GUI for Borg to make it as easy to
use as commercial backup tools. We just started translating into different
languages. All input and PRs are welcome.

[https://github.com/borgbase/vorta/](https://github.com/borgbase/vorta/)

(I'm the original author, but Thomas, the current Borg maintainer is also very
active.)

~~~
techload
Any plans to have a Windows client of Vorta? [Edit: Saw it now: "Windows is
currently not supported by Borg, but this may change in the future."]

~~~
m3nu
Vorta can run on Windows (it's in Qt), but Borg can't (for now). They are
working on it though.[1]

You could probably run it today using the Windows 10 Linux subsystem, but it's
not fully tested and will need some small fixes. Maybe later this year.

If anyone is interested in working on this platform, just post at [1]. It's
probably very doable.

1:
[https://github.com/borgbackup/borg/issues/936](https://github.com/borgbackup/borg/issues/936)

------
derekp7
The best solution I found as an alternative to an off-site copy is to set up a
USB drive and something like a Raspberry Pi in my car. When I pull into the
garage, the house server senses it and syncs new data to it.

This can be supplemented further by having it auto sync to a computer in your
office whenever you pull into work. Add in some monitoring so you can get a
phone alert if any of the synced copies are more than a few days out of date,
and you are almost golden.

~~~
gtirloni
Does the RaspberryPi have enough CPU for disk encryption?

~~~
clanrebornyes
I am trying to find cheaper device between NUC and Pi.

Lot fewer choices are available.

NUC is expensive. I wonder when will 8core ARM based NUC will arrive in the
market.

No idea is NUC can be powered by Xiomi Mi 2i powerbank but a Raspberry Pi can
be

Priorities:

1\. Gigabyte ethernet

2\. Small form factor NUC or smaller.

3\. 8+ cores

4\. Very low power consumption

~~~
8fingerlouie
You should give the Odroid HC2 (or HC1) a look
([https://www.hardkernel.com/shop/odroid-hc2-home-cloud-
two/](https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/))

It's an 8 core ARM SBC, with 2GB Ram, Gigabit ethernet on one USB3 bus, and a
SATA connector on another USB3 bus. It can fully saturate spinning rush,
though it might have problems with SSD drives. In any case, it will fully
saturate your GigE link with even a slow harddrive.

It runs a standard Linux kernel, with the "driver" for the hardware added,
which is why it requires a special build. Latest kernel version is 4.18. There
is also an OMV build for it.

Mine uses around 4-8W, depending on how much the harddrive is being used. I've
got a couple of them running as backup targets (Arq, Borg, etc) with a 4TB WD
Red and Btrfs, and they've been rock solid.

If you need hardware AES, you should probably be looking elsewhere.

------
ttmb
There really isn't a story here, Kev. You blew a power supply on your
Synology. You even said it yourself: "maybe this is a problem with the
enclosures and the disks are fine".

If you'd put your disks into a replacement Synology unit, you would have been
back online - config, data, and all - within a few minutes.

~~~
pimeys
This is the reason I upgraded our Synology with a self-built NAS with six
disks, a low-power Atom CPU and a really really good PSU connected to a UPS
unit for extra security. Never save money with your NAS power solution.

~~~
vonseel
Which OS and file system did you choose?

I recently did similar with an old server and Freenas, but I am still not sure
if I have a safe (enough) system overall. I chose RAIDZ2 with 4 4TB WD Red
drives, ECC RAM, seems to be working well so far.

~~~
pimeys
FreeNas with ZFS, RAIDZ2, 6 x 4TB WD Red, 16 GB ECC RAM.

System has been running a year now without any bigger maintenance.

------
setquk
Many years ago I watched someone stuff the DVDRW (remember them?) which
contained all his stuff into a work PC’s pioneer slot loader drive to get some
music off it. We stood there and the drive went bzzt, clang, bzzt, clang then
sped up way faster than it was supposed to go. This was followed by a large
bang and bits of DVDRW coming flying out and then a crunching noise.

From this I learned about single points of failure.

Edit: also in the decade and a half since I learned that you should never
trust a magic box, magic piece of software or magic container file system for
backups. A plain file system you can just copy your shit back from is the
closest thing to a guarantee. Also it’s cheaper to curate your data carefully
than end up with 4TiB of crap you’re too scared to deal with on your hands.

~~~
ValentineC
> _A plain file system you can just copy your shit back from is the closest
> thing to a guarantee._

These days, I'd suggest using a file system with some form of file checksum
metadata. If one values the integrity of their data, bit rot is a thing.

~~~
setquk
Very true. alternatively SHA sum all your data per directory. That’s easy
enough to write scripts to validate

------
sbov
> What are the chances of both a 4 disk RAID failing AND a USB drive at the
> same time?

Probably close to 100% if your house catches on fire.

~~~
Retric
Probably closer to 10%. House fires rarely completely destroy the house and
cars are often somewhere else.

Wildfires for example often destroy homes, but generally allow most people to
evacuate.

~~~
cyphar
Their USB drive was plugged into their NAS. The probability (as they found out
in TFA) is very high that a power surge will knock out most of their devices.
A house fire wouldn't hesitate in destroying both. I think "close to 100%" is
being too generous.

If you don't have 3 copies of your data (with at least 1 offsite) then your
data doesn't really exist.

------
linsomniac
Kudos to Synology for having a process to recover on a Linux box if the
Synology box craps the bed. I was reading this story thinking "this is a good
lesson on why you don't buy an expensive SAN, unless you can afford to buy
two".

But, the other lesson is: Backups. Sounds like backups were shut off 6 months
prior.

The other lesson is: Monitoring. Backups were going to the USB drive, but it
that stopped working at some point. Unless you have some tested monitoring of
your backups, you are likely to lose data.

Glad this story had a happy ending.

~~~
Karunamon
The nice thing about Synology storage stuff is that it's a nice GUI
(seriously, they're about the only company I can think of that's doing
appliance management right) on top of standard and battle-tested open source
tools.

This was one of the reasons I was okay with paying their prices. Even if the
device completely craps the bed, I'll be able to hook the drives containing
absolutely normal LVM/btrfs volumes up to another machine and get my data out.

~~~
petre
Btrfs is battle tested?

[https://btrfs.wiki.kernel.org/index.php/Gotchas](https://btrfs.wiki.kernel.org/index.php/Gotchas)

We use a box with 2x2TB mirror zpools. It runs Nas4free off an 128GB SSD.
Initially we wanted to use Freenas but ir needs 16GB of RAM, while Nas4free
works fine with 8. It also does SMART monitoring sending emails when something
is up.

~~~
doublepg23
FreeNAS works fine with 8GB RAM, especially with such little data

------
mholt
These kinds of scenarios are why I built Relica [1]. It backs up to local
disks, network drives, remote computers (on LAN or anywhere with a public IP
address), your own cloud storage, or our own special formula we call the
Relica Cloud: one upload, five independent cloud providers -- replicated in
real-time.

And restores can use the open-source tool restic [2], so you don't have to be
locked into Relica for accessing your data.

We're working on the ability to do byte-for-byte copies of a repository to
other destinations to make data even harder to lose in these kinds of disaster
scenarios, as well as a new UI to make it more pleasant and powerful.

Anyway, our goal is help make robust backup strategies like what this guy
needs really, really painless, because I'm as paranoid as he should have been
about losing data.

[1]: [https://relicabackup.com](https://relicabackup.com)

[2]: [https://github.com/restic/restic](https://github.com/restic/restic)

~~~
xtagon
Can this also be used as a replacement for sync services such as Dropbox or
SpiderOak ONE? In other words, can I sync accessible folders between computers
in near-realtime, or is it only archival/backup storage?

~~~
mholt
No, Relica is not a sync service, because we do not want to sync your deletes
-- we recommend using backup and sync together, because they serve two
different functions.

Relica is archival software, but you can backup+archive your synced folders of
course.

------
x220
>I don’t know what happened for sure, but I think it may have been a power
surge that fried the boards on both the Synology and the USB, as they were
plugged in to the same socket.

He didn't have a surge protector? Sweet Jesus. I don't plug my _backed-up_ PC
into anything not surge protected.

~~~
timthorn
Surge protection isn't a big thing in the UK these days. You can get arresters
but it's really rare now to hear of computer damage as a result of mains
spikes.

~~~
x220
Yeah but don't you think having a raid array but not having a surge protector
is missing the forest for the trees?

~~~
blattimwind
Power surges, brownouts etc. don't really exist in large parts of Europe. I
also never unplugged my computer(s) in thunderstorms (some of which are on a
USP, most of which were simply plugged into mains, no surge protector, no USP)
and lost exactly zero computers to that.

One of the few things that actually routinely comes with real surge arresters
(i.e. gas-discharge tubes, not the MOVs plastered all over the place) is
anything connected to a telephone line, e.g. A/VDSL modems. On some old PBXs
for analog lines these were even contained in field-replaceable modules.

~~~
distances
Can confirm, I think it's over ten years since my last power glitch. I don't
think I've ever even seen a surge protector; UPS I've seen a couple of times
at people who run servers at home.

------
bornabox
It's a common scenario, unfortunately. Synology RAID is nice. External USB on
the same power circuit is not a good idea. Actually, I'd look at least into a
surge protector. But it's quite possible to have those blow up and still fry
your hardware.

I'd run some batteries in the basement and run my Synology (or similar) off of
those. Additionally, the USB Backup of backups should be in the garage or the
attic, if at all possible. Also, a nice cloud-backup solution that is capable
of delta-uploads is a very good idea (cover against fire in the house, or any
other "catastrophic" failure).

If you don't have DC experience or if you didn't do much hardware, it's common
to over-focus on software. And to be fair, vice-versa :)

~~~
tapland
The issue is probably the Synology PSU and nothing else. This is the third
Synology NAS breaking down with intact disks I hear about this month, and it
seems to be quite a common issue.[0]

[0]
[https://www.youtube.com/watch?v=K7ly8zde3dE](https://www.youtube.com/watch?v=K7ly8zde3dE)

------
zbuf
I had a similar experience with a relative's Seagate NAS.

Except the ext filesystem was unreadable because it used a different page
size. Required some shenanigans in userland by thankfully I was able to
recover the data. Seemed like a software fault on the box.

The chassis had to be destroyed to remove the drive and it was interesting to
see the warranty explicitly mention the customer was allowed to do this to
recover their data.

~~~
ce4
Have you written up the solution or do you have some link? I have that same
problem currently but havent had time to look at it yet. Thanks!

------
abrowne
You need one copy somewhere else! What if there's a break-in or the roof leaks
or whatever? My low-tech solution is two external drives, one of which is at
my Mom's place and gets swapped every month or two.

~~~
cellularmitosis
This solution is great because of its simplicity. Keeping a second drive on
your desk at work is also great if mom is too far away.

------
fencepost
Things that jumped out at me from this:

First, the USB external is probably OK except that its USB circuitry has taken
a power hit. If it's a standard SATA drive inside that could probably be
shucked and accessed. Counter: Some of these have drives that are no longer
SATA but have a bunch of the USB connectivity built into the drive. At that
point, you'd probably be looking at a few hundred $ of data recovery costs
(yes, that little). Professional recovery of the RAID would be more expensive
because pricing is often based on the number and capacity of the drives.

Second, RAID5? I know these were only 1TB drives, but be very wary of anything
with only a single parity disk if you're looking at drives of 1TB or larger,
particularly if they're sequentially-numbered drives from the same lot. With
modern TB+ drives there's a not-insignificant chance of drive errors as you
hammer the remaining drives to rebuild the array. If building one of these
now, the price difference between a RAID5 of smaller disks and a RAID6 of
larger ones is probably only a few dollars.

Third, if actually doing recovery the first thing you want to do is image the
drives and work from the images. ddrescue is probably your simplest option
there, but yes you're going to need a big chunk of drive space available.

~~~
shittyadmin
The fears of parity raid disk error rates are greatly overblown - this post
does a good job of summing up the problem:

[http://www.raidtips.com/raid5-ure.aspx](http://www.raidtips.com/raid5-ure.aspx)

A bit of annecdata: I've been running raid5 and raid6 for years without issues
on 4 and 8TB drives, scrubs come back successful every month despite claimed
error rates, the drive deaths I've had have been sudden whole drive failures
or write failures to a large portion of the drive.

~~~
fencepost
The linked article is working through some of the numbers on a pure math basis
but the real world is messy in ways that article doesn't consider.

Most notably, if a drive has failed there's a reason for it and a lot of the
possible reasons will be shared with other drives in the array. Was there a
manufacturing issue and all of the drives are from the same lot (pretty
common). Is the RAID in a hostile environment (heat, vibration, bad power,
etc.)? Heck, is the RAID normally very lightly used and going to have heat
problems if it's under full load for hours (days?) during a rebuild?

There are also factors like how the RAID controller is going to handle another
read error - will it drop a second disk if the drive reports a failure? For
that matter, if an array drops to "degraded" due to an error are you going to
immediately replace the drive or are you going to write it off as a one-time
fluke and let the array rebuild? Do you keep a pool of spare drives around
that you'll drop the failed one into after testing? I've regularly seen an
array drop a drive due to something transient, then rebuild onto it and not
have any more problems for years.

Even with a 2-drive failure in a RAID5 you're unlikely to lose much data -
almost everything is likely still there on the disks unless there've been
catastrophic failures (e.g. an array of the _old_ "Deathstar" drives which
were prone to head crashes). You just may need to do recovery which will
generally mean imaging each of the drives and doing recovery based on working
with those images instead of the original drives.

------
saahbs
Here is what I do to hedge the risk of electrical failure (raid+usb on the
same circuit), fire (both copies in same structure), malicious compromise of
my machine and theft:

\- My workstation has linux software raid-1 of 2x 6TiB drives (this provides
robustness and uptime in case of single drive failures and ease of recovery).

\- Another machine in my garage doing incremental daily backup pulls over the
network. It is setup as multiple discrete hard drives thus partitioning single
drive failures (low cost, the garage is a separate building, the host machine
is an arm board that actually turns off HD PSU when not backing up, so hard
drives are fairly isolated from power spikes).

\- I make a monthly incremental backup (three sets) onto an external 6TB usb
hard drive encrypted with luks. This drive spends 99% of it's life powered
down in a cabinet at my office at work. It is protected against theft, fire,
electrical spikes, etc... by my employer.

\- The is not a *ucking cloud anywhere in this picture. I can get access to my
backups within ~1hr in worst case (round trip drive to office to pickup my
drive).

You Kids need to learn how to take care of your shit - now get off my lawn!!

~~~
8fingerlouie
I use Arq Backup on my workstations, along with Resilio Sync.

The NAS holds all our media/documents/music/whatever, and where possible, this
gets auto uploaded from workstations to the NAS, mostly through Resilio Sync,
but also ChronoSync. A local Raspberry Pi (different building) acts as a node
in the Resilio Sync setup, adding more redundancy.

The NAS backs up to a local USB drive nightly, as well as a remote (4km
distance) Odroid HC2 with a WD Red drive. This device also runs Resilio Sync
as a redundant node. All machines run Btrfs where possible, with smart
monitoring, daily short smart tests, weekly long smart tests, monthly scrubs,
and log monitoring emailed to my inbox every morning.

Finally i make yearly archive discs (100GB M-Disc) with the data from the past
12 months. I burn these in 2 copies, one is stored locally, the other is
stored remotely. Along with these drives, i also maintain a couple of 4TB USB3
drives, which i freshen (nondestructive badblocks) yearly, and update. Again,
one is stored locally, the other remotely with the M-discs.

Even with the above setup, there is a theoretical possibility of losing data,
but as most data lives on both the NAS and the client machines, as well as a
remote target, i would need to lose all 3 at the same time. The only
irreplaceable data would be our family photos, and those are also stored on
optical and magnetic media (spending 360/365 days powered off), adding at
least a couple more layers of redundancy to the equation.

------
AdmiralAsshat
Over Christmas, I backed up all my stuff to a 3TB external hard drive that I
took with me when I visited my parents and intentionally left the drive on a
shelf at their house.

This is what I call a poor man's offsite backup.

~~~
ocdtrekkie
I assume you either told them what it is or are reasonably confident they
won't throw away the random thing on their shelf that neither of them know
about.

I've seen a few cold backup setups like this, but I would both worry about the
significant gap in coverage between the drive you left at Christmas and next
time you update your backups. And also the shelf life of mechanical hard
drives not in operation is poor.

------
steelframe
As the saying goes, nobody wants a backup. What they want is a restore.

~~~
FabHK
Very good point. As I pointed out in another comment in this thread, I had my
backup disks encrypted with a password safely stored on the disk that was
being backed up. Backups were just fine, but the restore depended on that
password. Fortunately I had access to it via another means, otherwise: no
restore. Sobering.

------
mirimir
Something like that happened to me. I had several TB of data on a RAID10 array
on an LSI MegaRAID controller. A nearby lightning strike took out the server.
And the server manufacturer had gone out of business.

I had backups of the data itself. But I'd been doing lots of data massaging,
and didn't have enough storage to keep copies of every step.

Anyway, so I bought a couple new servers. One to replace the dead one, to be
setup with SQL Server. And a low-end one that would accept the controller from
the old server. I just left the drives in their cage, and jury rigged power
and data connections.

And it worked.

------
RcouF1uZ4gsC
Couple of observations.

First of all, given the price of storage, for backups, I don’t think anything
other than mirroring makes sense. Just get 2 big hard drives for your NAS and
set them up for mirroring. In the event of a filmier, you can read directly.

Next, you don’t have a full backup without offsite storage. Even if it wasn’t
a power surge, there could be flooding or fire at a single location.

Always remember the basic 3 2 1 rule!

~~~
ocdtrekkie
Indeed. I'd rather have a bunch of RAID 1s around than a RAID 5 I have to
worry about rebuilding. I sync my files to two different RAID 1 setups in two
different cities. No hardware failure has ever been worrying.

------
xupybd
I've always thought of a NAS as high speed local storage with a little
redundancy in raid, but I'd never treat it as a backup solution. Only because
then all my data is under one roof. Fire, earthquake, flood or other local
event that's big enough and all my data is gone. I'm lazy so I haven't setup
anything fancy but backblaze and dropbox do all my off site backup for me.
It's very cheap. 10 years of backblaze would be cheaper than a NAS.

------
drtillberg
Power problems can be both stealthy and deadly.

Years ago I had a desktop with a 4-disk RAID 5 where the SSDs failed in quick
succession, it's common. I lost some data-- or rather recovered files manually
I think from a failed RAID and cold backup, switched to spinning disks, but
after a while the new disks started beeping and generating RAID errors.

After much time and anxious guessing, I swapped the power supply and never had
a problem since.

------
albertzeyer
I recently was also thinking about how to organize my data & backups. I still
have not really decided. Esp about the software. I collected lots of options
here: [https://github.com/albertz/wiki/blob/master/backup-
software....](https://github.com/albertz/wiki/blob/master/backup-software.md)

At the moment, I really like Perkeep
([https://perkeep.org/](https://perkeep.org/)). But I'm not sure whether this
is a solution for everything.

On the hardware side, I also have not really decided. I want to build up my
own NAS (custom hardware, no preconfigured thing), which should be quiet (if
it is not doing anything, i.e. most of the time), as it will be in my home.
Another NAS maybe at my parents home. And then maybe some cloud storage.

~~~
RealityVoid
Yes, I was also thinking about perkeep and giving it a spin. I was considering
a RAID 1 array for redundancy.

Another thing I was considering was M-Disk[1], that can, supposedly, hold
information for 1000 years. But I wonder what other people's experience with
it is.

Alternatively, I understand normal Blu-Ray disks should be able to have data
retention of about 40 years, and that sounds decent to me.

I was looking into Amazon glacier and/or google cold line, and the prices seem
decent, but I do not like the fact that you have to pay monthly, even if a
small sum, it's just one more thing to concern yourself with. I would like to
prepay and know my stuff is up there for a couple of years.

Normal Dropbox/Google drive stuff is too expensive to store big amounts of
data, so not worth it. Plus, you have a copy of it locally also, (at least in
normal use cases)

Thinking about it, I think this would be a great idea for a startup. Cheap
data storage for long term storage, with competitive prices, the ability to
per-pay, user-side encryption and a simple UI that grandma can use to drop
photos. It should be able to guarantee that the information you desire to will
outlive you.

[1]
[https://en.wikipedia.org/wiki/M-DISC](https://en.wikipedia.org/wiki/M-DISC)

~~~
ValentineC
> _Thinking about it, I think this would be a great idea for a startup. Cheap
> data storage for long term storage, with competitive prices, the ability to
> per-pay, user-side encryption and a simple UI that grandma can use to drop
> photos. It should be able to guarantee that the information you desire to
> will outlive you._

As a consumer, I _wouldn 't_ trust a startup for such needs, because there's a
likelihood that the startup would either raise prices, pivot to a different
service offering, or shut down entirely.

Google Cloud Platform lets you make a manual prepayment [1], so that's an
option worth considering. I know Google gets a lot of flak for shutting down
consumer services, but I'm inclined to believe that they wouldn't shut down
Nearline/Coldline without a significant amount of notice.

[1] [https://cloud.google.com/billing/docs/how-to/manual-
payment](https://cloud.google.com/billing/docs/how-to/manual-payment)

~~~
RealityVoid
> As a consumer, I wouldn't trust a startup for such needs, because there's a
> likelihood that the startup would either raise prices, pivot to a different
> service offering, or shut down entirely.

Fair enough. It's just that I think something in the field that can do these
things would be filling a need. And all companies have to start _somewhere_

But I agree, getting a promise of long-term availability from a new company is
a bit rich.

Thanks for the info about GCP - I looked mostly into AWS glacier and I know
AWS did not have a prepay offer.

------
jacobush
I have looked and looked since 1998, tried rsync, RAIDs, striped drives, ATA
over Ethernet, lvm, ZFS, all sorts of things.

What finally clicked for me, is
[https://www.greyhole.net](https://www.greyhole.net)

It's like magic. Decide how much redundancy you want. Then just add drives to
it. It balances files automatically across drives. You can have remote drives
in the mix.

What it is not good at, is many small files. But for my use, media files and
backups (tar archives) it's a breeze. And the files are stored as normal file
on the drives it distributes too, so there is nothing complicated to dig into
should disaster strike. (Not that it has happened to me.)

No affiliation, just finally in a Zen state of mind when it comes to my home
NAS.

Next step - make sure all of that is backed up off site too, but that is
another thing altogether...

~~~
2Ccltvcm
It's a really tough sell for most people when you tell them they need to have
at least two copies of their data if they care about it. The vast majority of
people will rather "take their chances." Usually that means another $100-500
bucks for most people.

~~~
jacobush
It was a tough sell for me too. After losing 6 years worth of photos, it was
an easier sell!

------
mysterydip
I had all of my backups on a 2TB external drive, which worked great until the
MFT got corrupted somehow. Suddenly all my eggs in one backup basket felt a
bit silly. Fortunately I was able to recover all of it. I'm in the process of
partitioning out a new backup system to avoid that in the future.

------
Havoc
After thinking about this kind of data apocalypse for a bit I realised that
the portion of my data that is mission critical is actually really small.

Throwing it on multiple clouds with version history isn't an issue.

(I'd recommend an O365 sub + duplicati...in theory you can push like 5TB to MS
cloud).

~~~
chrsstrm
And here's your recurring reminder that all cloud storage services fingerprint
your uploaded files looking for TOS violations. If you haven't encrypted your
backups first you are at the will of the service you are using.

~~~
toufiqbarhamov
I use Arq to backup, and it’s a lovely little program that isn’t a resource
hog, and handles the encryption. I use it to back everything up, a copy to a
removable drive, a copy that gets updated every day online, and another once a
month online on a seperate server. It seems to work well, and it really is
painless,after the initial seeding.

~~~
patrickdavey
Yip, I also use arq and love it. Everything backed up into my nas and Google
drive. The only irritating thing is that there's no Linux client and I suspect
my next laptop will be Linux based not a MacBook.

------
FabHK
Silly mistake I nearly made:

MacBook Pro in for repair incl. wiped disk. No problem, I have two external
drives with regular TimeMachine backups, so go ahead.

Having received the laptop back, I plug in one of the disk drives and - oops,
it's encrypted (with a good, safe, long password of random characters), which
had previously been stored in the local login keychain, so that the disk
drives have always just silently automatically mounted over the years, and I
totally forgot that they were encrypted!

Fortunately, I had the password saved somewhere and access to it. Otherwise my
backups would have been for nought (though of course they themselves had the
disk password inside... well protected by itself.)

------
baseballMan
Is it weird that I don't have GBs of my own personal files/data? I get if you
have movies/tv shows or whatever, that's different. I just don't know what
personal data you could possibly have that would amount to GBs...

~~~
mxuribe
As a parent you go through a period of time - when your kid is young - that
you take photos and videos of EVERYTHING that they do. Its silly now that i
look back. Certainly it is important to capture moments, but to over-document
everything is silly...now i take minimal photos/videos, and just be present in
the moment. Oh, and my data accumulation has precipitously gone down since
i've mellowed out. But, yeah family photos/videos adds up like crazy.

~~~
cptskippy
I have 3 kids and still don't have more than a terabyte of data.

------
fro0116
I currently have way to much data lying around for any metered cloud storage
to be economical, so I'm currently subscribing to both Backblaze and Crashplan
for their unlimited storage backup plans.

Crashplan killed off their consumer plan a while ago, so I ended up moving to
their business plan at double the price. In my case even at that price it was
still worth it considering the amount of storage I'm using.

Anyone aware of other services offering unlimited storage for a single user? I
know Dropbox, Google Drive, and OneDrive all offer unlimited storage in their
business plans, but they all require a certain number of users before the
unlimited storage kicks in.

~~~
Jedd
Your question - about data and transit volumes, and costs - would be
answerable if you provided some actual numbers.

~~~
stevekemp
I'm not the poster, but I have local and remote backups of photographs.

When I shoot a model I will take 400-600 images in an hour, each about 25Mb in
size. The shoot I had last weekend resulted in approximately 18Gb of RAW
files, and output JPG files.

In total I have just under 3Tb of RAW, JPG, and other media files. (Sometimes
I film shoots, or do some video-work at the same time.)

That kind of volume is not huge, but still painful to upload remotely. Its
also at the cusp of the kind of data you can backup to a cheap SAN-box
locally. I currently have two toy NAS devices each with 2x4Tb drives. If I
want to bump my local capacity to 8Tb, or similar, it'll get quite expensive.

~~~
Jedd
Okay, so if this is a professional gig, then off site backups are a must, and
raid on site is also. But for a pro photographer I'd have expected disk costs
to be a necessary expense. In Australia we have 10TB disks for AUD 450. Double
up for redundancy and add on chassis, this isn't trivial $s, but equally it's
presumably 'worth it'.

For my non-pro and far more modest collection of around 300GB of images, I
keep copies on three local machines, and one remote (family member) with sync
changes being able to be done over home grade ADSL. With your volumes you
could do off site sync via usb disk easily enough for new large ingests, and
propagate smaller changes over the wire. Having a friend or family in the same
city is very convenient compared to trying to hunt down the best all you can
eat deal du jour, with no need to handle the regular t&c changes those
services suffer. Good reciprocal opportunities too, of course.

------
cheeseprocedure
A surge from a lightning strike near my home travelled over the cable line to
kill a network switch and the WAN port on a firewall. (Strangely, the cable
modem was spared.)

Everything was on a decent UPS... but I’d completely forgotten the cable line.

~~~
ken
Most of the UPSs and surge protectors I've seen for home use include coax,
ethernet, and/or telephone protection, too, e.g.,
[https://www.apc.com/shop/us/en/products/APC-Performance-
Surg...](https://www.apc.com/shop/us/en/products/APC-Performance-
SurgeArrest-11-Outlet-with-Phone-Splitter-Coax-and-Ethernet-
Protection-120V/P-P11VNT3)

~~~
matthew-wegner
Lighting is pretty rough.

You _can_ completely air gap your network from the outside line by converting
to fiber at some point (probably between your cable modem and your router, in
a DOCSIS setup). Isn't foolproof, because lightning can induce current on your
wires directly, but it'll help these kinds of scenarios.

------
pmlnr
\- SSD + HDD in laptop, this latter for storage, because it actually tells me,
if it's unhappy and about to die, unlike the m.2 ssd. The cost of this is to
have an older laptop, in my case, a thinkpad x250.

\- synced to home server, which, at this point, is a thinkpad x201 on an
ultrabase with 2 disks - it has built-in ups, called a battery

\- all of this synced to off-site rented server in Germany

\- irreplacable photos are on blu-ray on yearly archives

This covers lost laptop, burglary, house fire/flood, etc. To avoid problems
with lost ssh keys, I have a few users on that rented server which can log in
with a password, in case of emergency.

~~~
zepolen
Synced how?

------
mey
My current approach.

Personal media archive, Windows 10 Pro, Storage Space with Parity across 4 HDD
in a sata jbod. This is a purely software RAID. Moving the drives (or part of
them) to another Windows 10 system allows for seamless recovery.

This archive, as well as all personal computers use Backblaze for offsite
backup (including versioning). Versioning is important in case
malware/accidents/buggy software. I don't consider any backup plan complete
without this and being off-site (fire/theft)

For my business servers I use Tarsnap. (Off-site and versioning, 45 days)

Edit: Oh and everything is on UPS

------
rthtrshfeqe
RAID is not for data protection. It's for availability.

------
brador
Data Storage is dirt cheap. Buy a 6tb for $100 every few months and throw
important folders onto it. Place old ones at relatives homes for offsite.

Never worry about this again.

~~~
pbhjpbhj
6TB drives are about £150 at the bottom end, average maybe £180. That's c.
$200-240. £600 a year on back up hardware is pretty hefty.

------
bastawhiz
My first thought is "why don't you keep a backup?" I really don't know what
the moral of this story is beyond "don't keep one copy of all your data,
especially in the same physical location."

Online storage is cheap. Bandwidth is cheap. There's a multitude of solutions
in the comments if a consumer solution like Dropbox or Google Drive isn't good
enough for you.

~~~
cptskippy
> Bandwidth is cheap.

Most ISPs in the US now have a 1TB datacap and charge ridiculous rates for
overages.

~~~
bastawhiz
Are you really pushing 1TB/mo, though? That would be a full quarter of OP's
lifetime of data. And in those cases where you are, could you not afford
business class internet for a bit of a premium?

~~~
detaro
It's on top of your other bandwidth usage though. Family of 4 with
Netflix/YouTube/Twitch can accumulate quite a bit. Still, using an online
service at least for an important subset is a good idea.

------
shakna
Sounds like a UPS with the standard protections is going to be somewhere in
your future... Just not somewhere a cleaning lady can touch it.

~~~
sliken
Most UPSs don't provide much more protection than a power strip. They have a
cheap "surge" protector, if you look it's often just running the tower through
a ferrite ring.

Generally the input it directly connected to the output, through the ferrite
ring. Only in cases of a power outage does a relay trigger and switch to the
batteries. But that doesn't prevent a surge from going through the UPS.

You can hear the relay trigger when you disconnect it from the wall power, the
latency is high enough to allow plenty of damage.

There do exist UPSs without this problem, but they are more expensive,
heavier, generate more heat, and consume more power. Look for double
conversion UPSs that in the normal state go from AC -> DC -> AC.

Kinda weird that while computers are 100% DC, they are fed with 100% AC. I was
pleasantly surprised that pretty much all ham radio stuff is DC fed. MUCH
easier to do UPSs, solar, wind, or have multiple pieces of equipment share the
same power supply. Imagine racks with a big power supply on top (and getting
rid of all the AC -> DC conversion heat). I've actually seen these, but they
were unfortunately cost prohibitive.

------
philjohn
There isn't really a good excuse not to have an offsite backup, especially
when services like back blaze are $5 a month.

Also ... RAID is not a backup.

~~~
basilgohar
Backblaze is great. But their desktop backup solution for $5/month is not
available beyond Windows and MacOS. Their B2 storage is via duplicity, but
that quickly exceeds $5/month with typical storage cases.

~~~
philjohn
Fair point. It would be great if Backblaze supported Linux with a desktop
client.

So I guess the options would be desktop backup on the various computers that
feed the current NAS solution.

------
codingdave
No matter what your backup tech is, if you have it all in a single location,
you don't truly have a reliable backup.

------
ashurov
Synology allows syncing of your NAS to a similar Synoogy NAS on another
location (Cloud station server). That's what I am doing. This doesn't prevent
issues with data corruption, but prevents these kind of issues where you
'complete' NAS fails due to a possible surge.

------
markgreene
The failure modes I consider most probable at my home are a whole-house power
surge, and a loss of structure (fire, natural disaster). With a 1 Gbps
(synchronous) FTTH connection, and a Backblaze subscription, that's about all
the peace of mind I need!

------
ganoushoreilly
I'm surprised it's not mentioned here but a lot of the Synology nas's have had
power supply issues in addition to the prior Intel Atom issues. I swapped out
four units twice each (1815+'s) and swore off their hardware permanently.

~~~
bhauer
To be fair, at least with the Intel Atom C2000-series problem, I find it
difficult to blame them. We had a network edge device from another vendor die
due to the C2000 defect. I figure the best the vendor can do in that case is
offer to replace/fix it, which the vendor in question did. Did Synology offer
to fix their C2000 devices?

------
absc
I use tarsnap for off-site backups. It has a reasonable price, even for some
GB of data

~~~
absc
Forgot the link:

[https://tarsnap.com](https://tarsnap.com)

------
randyrand
Pretty happy with my solution. Keeps me safe from floods, ransomware, and
losing google drive access:

\- originals: 2TB google drive + 50GB icloud

\- backup #1 (autosync from drive/icloud): 4TB internal drive

\- backup #2 (cold storage): 4TB external drive

~~~
keithpeter
How often do you do the incremental backups on backup drive #2?

Some degree of cold storage would have reduced the OP's stress level when
restoring their NAS mind you.

~~~
randyrand
maybe every 2-3 months.

------
lettergram
That’s part of the reason I keep my personal data backed up two two sets of
NAS. In addition, it’s mapped as a Dropbox drive on one of them.

From my families perspective it’s just one drive, but it’s replicated
everywhere.

------
fiatjaf
Me too, because I was storing it on IPFS. IPFS had a minor bug and I've lost
all the references to the data.

You know, the data is still there, I just don't know what hashes correspond to
each objects.

------
ricardobeat
I use Synology cloud software to backup to both Google Photos and Backblaze.
Way faster than Glacier, and costs me <$5 month (haven't reached 1TB yet).

------
dabockster
After watching Steve from Gamers Nexus talk about their second Synology NAS
failure in a single quarter, I instantly figured that OP was using a Synology
product. Their stuff must be bad if I'm making that association.

[https://www.youtube.com/watch?v=K7ly8zde3dE](https://www.youtube.com/watch?v=K7ly8zde3dE)

------
techload
Anyone knows an open source Backup Client that runs on Windows and that
support SFTP?

------
known
Bitter lesson learned: Invest couple of dollars in a surge protector.

------
cyberferret
Coincidental timing for this article! I just wrote in my own blog yesterday
about the level of importance I had placed upon the data I had stored on my PC
[0]. (TLDR - Data I thought was unimportant ended up actually being the
opposite right after I lost my external drive).

[0] - [http://devan.blaze.com.au/blog/2019/1/20/the-folly-of-
unimpo...](http://devan.blaze.com.au/blog/2019/1/20/the-folly-of-unimportant-
data)

------
gnulinux
4 disk RAID what? 4 disk RAID 1 or RAID 10 or... It'd make a difference.

~~~
cyphar
The article mentions that it's RAID 5:

> My Synology has 4 x 1TB disks in a RAID 5

------
2Ccltvcm
TL;DR: a power surge killed my PSU and maybe my motherboard, causing me to
freak out and btw I don't have any cold storage.

I'll bet this guy did not have a surge protector in front of his Synology PSU.

------
skookumchuck
I've been working with computers for so long that total failure is not a
probability, it's a certainty. Last year, in fact, my desktop burst into
flames. I've had smoke come out of desktops several times before, but that was
the first flamer.

------
Farradfahren
The best thing i found to enforce backup discipline on myself is to regularly
migrate between machines- who are regularly offline for more then 4 hours.

You will have mostly current backups of your projectdata and workdata on all
those machines you migrate from and towards.

The problem theire then becomes destructive automation. You must avoid
automating syncs with half-corrupted or full-corrupted instances of your work
environment.

Also backups. Always backups.

------
gammateam
He doesnt confirm that the drives are faulty, and it isnt clear that they are,
as the data is accessible while maybe some partition or boot data was
scrambled but repairable

The NAS should also have a warranty of some kind or the controller could be
repaired for cheap

He was never in any data peril, so just fix that and then add an offsite
backup to the mix

------
bunkydoo
M Discs brother, they are cleared to last 1000+ years. Long after most
people's cloud backups have turned to dust, my most treasured data will endure
due to my use of these discs (which may well be unreadable from a hardware
standpoint in 3019. But shit, hope they keep the schematics around.

------
hendry
Inspired by this post, I decided to document my backup strategy and
shortcomings here:
[https://natalian.org/2019/01/20/Data_I_would_not_like_to_los...](https://natalian.org/2019/01/20/Data_I_would_not_like_to_lose/)

tl;dr I'm trusting Apple.

------
bsder
Z ... F ... S.

And at least Raid-Z. Raid-5 and Raid-6 are now at probability of failure
levels that your rebuild is likely to throw an error.

~~~
Filligree
ZFS is great, but won't help if there's a power surge that fries all the disks
at once. To prevent that you'll want to store snapshots remotely somewhere.

~~~
2Ccltvcm
To prevent that you'll want to s̶t̶o̶r̶e̶ ̶s̶n̶a̶p̶s̶h̶o̶t̶s̶
̶r̶e̶m̶o̶t̶e̶l̶y̶ ̶s̶o̶m̶e̶w̶h̶e̶r̶e̶.̶ use a surge protector

~~~
derekp7
Don't forget to replace your surge protector on an annual basis (or more
frequently if you have really dirty power). Also, you can have a nasty fault
in your PSU that fries everything also (If anyone knows of a surge protector
that sits between the power supply and the motherboard, I'd really like to get
one).

~~~
wtallis
> (If anyone knows of a surge protector that sits between the power supply and
> the motherboard, I'd really like to get one)

I think the best you can do is just get a top-quality power supply. SeaSonic
will sell you a ridiculously overengineered box with a 12-year warranty for
$160; it's guaranteed to have a longer useful lifetime than any other
component in your computer, probably even including the case itself.

~~~
2Ccltvcm
Just use a surge protector literally right next to your PSU.

Cheap whole house surge protectors are pretty much useless. It will work for a
few times, but the MOVs will degrade and fail pretty quickly. Every time a big
inductive load switches on or off its going to cause a voltage spike that is
going to trip the MOVs. or the MOV's Trip voltage is so high (to avoid quick
degration) that they really don't provide any protect. Really a good whole
house protector needs to use SADs (Silicone Avalance Diodes) or a passive LC
filter. Transtector, Thor Systems (SADs) or Pricewheeler (LC Filter). These
protectors can take many more surge hits than the ones you listed above. Why
whole house Surge protectors don't work.[https://zerosurge.com/wp-
content/uploads/2016/10/USTech.pdfE...](https://zerosurge.com/wp-
content/uploads/2016/10/USTech.pdfEven) with a whole house protector it still
would be a good idea to use point of use surge protectors. Since the surge has
to reach the Whole House protector before it can be clamped. Surges can reach
your electrical devices faster than a whole house protector can clamp. It only
takes a few nanoseconds to destroy a microchip. For instance say you have a
vacuum cleaner plugged into the same circuit your computer is connected to.
the vacuum cleaner jams, causing a surge that will hit your computer (since it
closer) than the whole house proetector is (down in your basement).﻿

Source: comment section on
[https://www.youtube.com/watch?v=6PqO0aQaGDY](https://www.youtube.com/watch?v=6PqO0aQaGDY)

------
_bxg1
I just don't understand rolling your own backup drives in 2018 unless you're
up to something nefarious. I rest peacefully knowing my data will never be
lost by Dropbox, and the immediate backup means I don't have to mess with
slow, periodic all-at-once backups.

~~~
KingMachiavelli
It's much cheaper to build your own than to use cloud storage esecially a
secondary storage service like Dropbox. Even blackblaze B2 is $60/TB/year vs
you can buy an 8TB HDD on Ebay for ~$200 and if the drive lasts 5 years that
works out to be $5/TB/year and adding one or two redudent drives is still much
cheaper.

~~~
cyphar
How do you deal with the off-site problem? That's why most people use cloud
services -- as an off-site backup that they don't need to manage themselves.
Yeah, you could just put everything on an 8TB hard-drive from eBay (not sure
that's a great idea for important data) and stick it on a shelf -- but the
chance that drive will be alive and not have bit-rotted by the time you need
the data is smaller than most people would be comfortable with.

------
ams6110
If you have 4TB of data saved at home you are a hoarder. Why do you have that
much stuff. I don't have any data at home that I would miss very much if I
lost it.

~~~
xupybd
Family photos and videos can take up a lot of space. Also it sounds like the
authors work is backed up to this NAS as well.

I wouldn't have that much to backup but some do, depending on their hobbies
and work.

~~~
ams6110
Photos, videos... Yeah I used to save that stuff. I realized I never looked at
it again. So I just stopped. I literally don't worry about losing data at
home. Work is different of course.

~~~
xupybd
I think for some this might be the case but there are some photos people hold
dear. Wedding photos, children growing up and other major life events. Each to
their own.

------
1kGarand
I see some terrible backup strategies here.

1\. Backups should not be on a single drive. 2\. Backups without checksums
will result in corruption. 3\. Offsite is a must. 4\. Unencrypted off site
backup means someone already copied your data. 5\. Encrypted offsite backups
should have forward secrecy. So different keys for each file and keys file
gets backed up encrypted.

My backup strategy: File server runs zfs raidz with Daily/weekly/monthly
snapshots on disk.

Snapshots get copied to 2 external drives, zfs mirrored.

Files get encrypted and uploaded to backblaze using my custom software.
Nothing fancy, just standard authenticated encryption (chacha20poly-poly1306)
but with per file key management and argon2.

~~~
walterbell
_> Encrypted offsite backups should have forward secrecy. So different keys
for each file and keys file gets backed up encrypted._

Any references on PFS for backups? Was there no existing OSS backup solution
that implements PFS?

~~~
1kGarand
Most encrypted backup solutions are really bad with protecting keys. Fixed ivs
are ok for one file. Not ok for possibly millions of small files. Basically
exposed your private key along with your backup.

~~~
walterbell
How do you manage millions of keys, if you have millions of small files to be
backed up? Would it be ok to have something between 1:all and 1:1?

~~~
1kGarand
Sqlite.

