Ask HN: Data Usage for Family Exploding

[nytesky]

How do people manage their household reams of data that our family generates?

Each family member has their own laptops and phones, and take dozens of high resolution images every week, school presentations and projects, receipts, and tax documents, videos -- lots of videos.

As a family of 5 with 2 working parents, I'm not really monitoring data use and want a mostly hands off solution to their data storage. And every new phone has a higher resolution camera though maybe JPEGXL will help us there.

We pay for 2TB of icloud data, but I am already eyeing upgrading to 6TB -- but that adds another $22/month. Moving to other cloud providers may save somewhat, but then its another management of a cloud ecosystem, and I know my family will mostly stick to icloud.

On a side note, we store alot of data on google drive too, and google takeout fails everytime I try to download a 50GB file -- so really its trapped there?! All of these have a lockin factor since with more and more data its a huge effort to export!

[austin-cheney]

I set up a house file server. It’s a pretty minimal piece of hardware that came with a 2tb nvme disk. I added two 14tb disks in a RAID 1 array and they don’t even fit in the box physically. The RAID array is just for redundancy so that a disk can fail and the data is still available from the other disk.

On this file server I am running Debian12 with Docker Compose. Then I built a management dashboard in TypeScript/Node that provides web servers you can spin up in about 2seconds and each server provides Websockets and http with fully proxy support over a single port. The dashboard also provides a command terminal via browser interface and a bunch of other server and connection management tools.

* I also included Samba for remote file system access from Windows with permission management.

* I run Jellyfin with metadata caching so that I can stream video without need for internet access or subscriptions.

* I run Pihole for DNS management.

* I just got WireGuard set up so now I can VPN directly into the home network from anywhere in the world via IPv6.

Therefore I don’t need a cloud storage subscription.

[solardev]

It's probably mostly videos. Upload the ones you don't really care about to a private YouTube channel, which doesn't charge you for storage, and delete them from iCloud. Keep the ones you really want to keep on both (it's safer that way).

By contrast the photos and documents will barely take up any space. It's not worth worrying about those until you can do something about the videos. They probably wouldn't even be an issue afterward.

Or just make the kids do chores or get a part time job to help pay for the extra iCloud storage. $22 is not much for a whole family's worth of memories.

[shortrounddev2]

I would delete stuff personally. Not everything needs to be saved

[nytesky → shortrounddev2]

Finding the time to go through and make sure its not relevant (like tax documents) and deduping photos. The ease of capturing things means we have far more to sift through. I agree I would like to just toss, but I have used older documents many times.

[HenryBemis → nytesky]

There is a bunch of 'duplicate finder' apps for Windows (so I assume for every OS under the sun). It'll take you 10mins the first time, and 1 min every other time.

I run it every few months (or whenever I remember to do so). I don't bother for word/excel/pdf, even if there are some 20MB PDF files, a 5min 1080 shaky vid from a concert you went 5 years ago is a far worthier target for your time.

[curious_curios → nytesky]

Documents don’t take up much space. Videos absolutely do though, so you’ll get a pretty good return on investment in going through those.

[theGeatZhopa → nytesky]

for me, i just search for the biggest files in my devices, then dedupe the ones with same file-size. Then I take a look on thumbnails of the big files and dedupe in case the thumbnails are the same (^but the file-size is not). Then I sort all the left files on size and look out for general data of same file-size and then decide whether to delete without breaking things or just leave them.

but, I'm thinking putting a NAS in my home and do all the neccessary backup onto that.

[giantg2]

You can set up a home NAS like people are saying. Just one point many aren't saying as a part of that, is you still want a disconnected backup (probably in a fire safe or off-site) of anything truly important to avoid data loss in the event of a lightning strike, fire, etc.

I haven't set up a NAS yet, but I probably will in the coming years. Right now, I do periodic drive image backups to a secondary drive, a few periodic file backups to USB drives for documents but not images, and some periodic drive image backups to external drives. Seems to work for me so far, but we also don't have a lot of file sharing going on, so it's mostly just storage.

[brudgers]

We pay for 2TB of icloud data

For what it is worth, the cloud is not a backup because it is one or two failed credit card transaction away from oblivion.

As a rule of thumb, if it is easy or convenient it probably is not a good back up method. But even poor backup methods that are easy or convenient might still be better than nothing. If you are lucky.

Good luck.

[Someone → brudgers]

Also: iCloud syncing isn’t a backup solution at all. It makes cloud storage your primary storage, with your phone/laptop SSDs caching parts of that, retrieving data from iCloud as needed.

[romanhn → brudgers]

I think cloud is great for backup - as long as it's not your sole backup strategy. My personal approach is to have a local NAS with redundant drives that also replicates data to the cloud. I'm protected from drive failures, house fires, failed credit card transactions. Obviously not all at the same time, but as far as risks go, feels pretty minimal.

[brudgers → romanhn]

The cloud is great for backups in the something is better than nothing sense. The same is true for SD cards thrown into a shoebox — except SD cards don’t have a dependency on credit cards.

Obviously not all at the same time

That’s the “so long as you are lucky” part. The cloud is great for availability. Availability != Backup.

[nextts → brudgers]

The cloud is great as the failure mode is statistically independent of the home backup failure mode. So both backups together if they have a 1000 to 1 chance of failing per year would give you a million to 1. You'd need to lose your primary data then hit the 365 million to 1 to lose all data.

[nytesky → brudgers]

Generally they will hold onto data for a while without payment. It’s not like a hard drive needs to be replaced every couple of years — can you prepay cloud like domains?

[brudgers → nytesky]

Spinning disks are very reliable and write once is the only good long term backup strategy.

Backups to restore live systems can involve rewriting and pruning, but live system backups are about availability not archiving.

Prepayment makes financial issues more likely to be catastrophic because the timing of the issue is the mid term of the prepayment period…with a one year period the issue may go unnoticed for more than six month and gmail empties your spam/trash folder after 30 days.

You can stick hard disks in a box and there’s nothing to manage.