Not a great time price-wise to be building a NAS, but I have been doing so for the last two weeks. Inside a Jonsbo N6 case, which is pretty nice with an 8x SATA backplane and drive bays (unlike the earlier Jonsbo variants).
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
> Not a great time price-wise to be building a NAS
That under-states the matter. It is a terrible time, price-wise, to build a NAS.
I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)
I would kill (figuratively, anyway) to have no AI whatsoever. No slop machine threatening to replace my job, or turn my job into babysitting its stupidity, and hardware would be reasonably priced? That would be awesome. AI has brought me nothing but downside.
AI has freed most people I know from the tedious job of writing actual emails and birthday greetings - leaving then more time to e.g. wash dishes or clean the floor.
Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.
Was it a genuine birthday greeting when you bought a card? Or used an online card? I don’t really see the difference. The point is you thought of them, not the actual words
I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
> I’m running an 8-drive ZFS RAIDZ2 pool. I’m wondering if you know — are the free space recommendations around ZFS cargo or real?
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
ZFS will auto-degrade performance if there isn't enough headroom. In addition, if you use SSD, you also want the headroom, because otherwise you end up writing and rewriting on the same small empty space which kills SSD (unless you have enterprise SSD, which have built-in headroom).
You assert that ECC RAM being necessary for ZFS is just a myth but provide no justification for why that is untrue.
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
Yes, indeed. ECC RAM is better than non-ECC RAM, also for ZFS.
The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.
A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.
The only hangup with the myth "debunking" is that the point is that the corruption doesn't happen to a per-disk buffer, but to the in-flight data before it's persisted to your stripe.
Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.
But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.
What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.
Usually if you’re using a NAS you don’t want to lose data, ZFS is not significantly more sensitive than everything else.
But everything is actually quite sensitive.
We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.
You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.
My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA
It was €329 per hard drive (through a reputable store), and I chose the ASRock B650M-HDV/M.2 mATX motherboard combined with a Ryzen 5 8500G. Stock CPU cooler, replaced the Jonsbo case fans with Arctic P12 Pro PST LN. I only slightly regret the PSU (MSI MAG A650GL), which could be quieter. Not that it's very noisy, and it's a great PSU otherwise, but I should've chosen one that just shuts the fan down at low power usage.
I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.
--
(*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.
> I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall.
I'm also planning a new home server build now and the prices are definitely relatively ass. So far I've spent 820€ on two 22 TB WD Elements HDDs, 375€ on 2x16 GB DDR5 kit, and 520€ on two 2 TB M.2 SSDs (cache). About 1700€ and I still have no server to show for it. Doesn't help that I've been procrastinating on picking the CPU and motherboard.
I really, do not want to have a big server rack lying around at my house. I have a couple of SSDs that are not being used and 3 USB HDD drives. I also have a PN50 that I am not using as well.
I am thinking of buying a USB bay with 5 SSD slots in it and then 3 HDD drive.
My use case is very, very cold. It is mostly just readonly data with some rclone sync every week.
Does anyone have a suggestion? I am pretty much relying on ZFS to do all the redundancy for me.
I found cockpit to be incredibly useful for managing NAS and VPS. Even though any configuration is not an issue with LLMs, having a dashboard with toggles, formatted logs and such is helpful, and I can even run shell commands on phone without ssh. In the NAS case, you can easily see the results of backup/maintenance cronjobs, manage samba (which is a huge pain in the ass to get to work properly on iPhone), and monitor disk usage
Same, using NixOS as a NAS, though it kept growing and now I'm trying to share it with other people. I use BTRFS on MDADM though. It's ended up being an all-in-one home server and router now because I was tired of the noisy power hungry rack I had in my closet.
I still pay for snooty, and the reason for that is that when a disk goes bad (not if; when) I pop its tray out, replace the disk, pop the tray with the disk back in, click a couple of widgets, and that’s it. I know it will be rebuilt properly.
(And I know I have to do that, because when the disk fails it beeps and lights a led near the bad disk)
It’s easy to build a NAS such as the one described in this article, but in the long run, data loss is significantly more likely.
Also, any guide like this that doesn’t guide you through “disk 3 failed, this is how you safely replace it” is imho incomplete, even if it doesn’t go through telling you how you know a disk has failed.
That is kind of exactly how zfs works though. The guide isn't complete, sure, but "rebuilding" the array is just replace the disk and run a single zfs command.
That’s comforting to know; that wasn’t true in the past for ext4 over lvm (is it true now?).
But what is that command? And how do you know which disk has gone bad?
I am sure I can get an answer from Google / Claude / ChatGPT, but a guide is incomplete without it - and the failure report should be active like a beep or flashing hardware light - I typically log into my NAS only a few times a year. A motd or other banner isn’t sufficient.
To be fair, the "beeps and lights a led" part is very important in case of a dead disk in an array. The only time, I disrupted a service in production for longer than a few hours was when I didn't get the email of a degraded raid 1 array and the second disk died a few weeks later.
Just yesterday I was brainstorming with ChatGPT about this. I have an ancient QNAP plus a slightly less ancient NUC running PiHole, Wireguard and other services. Both need replaced, so why not combine them?
I don't know much about ZFS, but it sounds like I need to learn. Docker may have conquered the world, but I plan to stay with LXD for services.
The one thing I take issue with: an appliance like this runs 24/7. It should be low power and fanless. A processor like the N100 seems like the obvious choice.
One alternative for those who don't want any of the major NAS vendors, just use RHEL10. It's free up to 16 licenses, it's ultra stable, cockpit is a very mature gui for a lot of maintenance tasks.
It's a set and forget OS that will run for years without requiring your attention. But these days it has decent container support for hosting services on.
The problem with RHEL is that the free version exists at the sufferance of IBM, a company not well known for being motivated by keeping tech enthusiasts happy. I would use Debian, personally. Not as long of a support cycle as RHEL, but still quite stable and no possibility of corporate rug-pulls.
its probably dumb but what i do is use proxmox with a debian VM with docker installed and each docker container gets a virtual disk i make on the proxmox host
I've been thinking about setting up something like this for a while. I have a Broadwell dual socket Xeon workstation that I'm going to upgrade to Proxmox. Would it be reasonable to run something like this as an LXC or VM or would you put it in the base kernel?
> I am creating a RAIDZ1 (RAID 5) zpool. That means 1 drive redundancy in-case of failure
A friend once told me that RAID5 has a high latency cost, because every Write requires a Read to update the stripes across all drives, and while this made sense when drives were expensive, nowadays you might as well do a RAID10 instead, and trade space for latency.
It's complicated, but https://jro.io/capacity/ actually walks you though how it works, towards the middle of the page.
```
Unlike traditional RAID5 and RAID6 implementations, ZFS supports partial-stripe writes. This has a number of important advantages but also presents some implications for space calculation that we'll need to consider. Supporting partial stripe writes means that in our 7wZ2 vdev example, we can support a write of 12 total sectors even though 12 is not an even multiple of our stripe width (7). 12 is evenly divisible by +1 (3 in this case), so we don't even need any padding. We would have a single full stripe of 7 sectors (2 parity sectors plus 5 data sectors) followed by a partial stripe with 2 parity sectors and 3 data sectors. This will be important because even though we can support partial stripe writes, every stripe (including those partial stripes) need a full set of p parity sectors.
```
RAIDZ1 will add more stress to the consumer ssd's as to enterprise one due to surprinsingly different hardware. Consumer ssd's life times are calculated on a 8 hour shift (the tbw) and enterprise are calculated on 24/7.
Please - dont - use - consumer ssd's - with zfs raidz1
To this day I still use a ZFS array for my critical backup storage. I've migrated over to using snap raid + mergerfs for my larger Linux iso storage array. Simple enough and I can pull a drive on it's own without any other stuff.
I killed 2x 4tb Nvme drives on Pciexpress with ZFS raid1 under proxmox, got them replaced, tried again on both pci-express and usb adapter kept getting io problems there too with zfs, drive gets disconnected during scrub or high load, tried already kernel or power management options for pciexpress and usb without sucess, managed to reproduce with high load.
or proxmox ( especially community edition) is fucked or zfs is still unstable, surprisingly with different drives on different interfaces I obtained same results, didnt swap proxmox for clean os, that might being some changes since modules and options would be different i guess. ( drives firmware etc are fine and performing well when inspected with proprietary windows tools)
just search zfs nvme pcie or usb problema or disconnect and you see similar stability problem for different users in different cases (os, drives etc), but also unraid and others, maybe somewhere is rithe right combo of options/glitches but didnt find it yet
I'm sure there's better options now but the HP ProLiant MicroServers (used).
They support ECC ram, 4 caddies, one extra PCIe slot, and to my knowledge you're not cpu limited for a zfs file server usecase.
Keep in mind though, all you need is linux* support, iDRAC, ECC if you're a snob, and drive bays ... and that's basically any free server.
In my extremely opinionated opinion I would only get used enterprise server gear, because a zfs file server will just work unless hardware fails. And a UPS.
*ZFS will be a more natural choice on FreeBSD. It's better documented, and will meet Linux 1:1 in hardware compatibility for this.
I'm still running an old Gen 8 MicroServer. Modern drives can actually saturate the SATA controller, and because it only has a single PCIe slot I can't add both a 10Gb NIC and a storage controller - I went with the 10Gb NIC.
It works well enough though and has lasted me over a decade at this point. 16GB DDR3 ECC, an old 4 core/8 thread Xeon, 4x14TB drives and the Mellanox NIC.
Not strictly a recommendation, but Terramaster is a good brand to look at if you want Synology-shaped hardware which can run TrueNAS or Proxmox or any flavour of Linux you want.
Along with various other devices (including a large Synology which I wouldn’t buy today), I run Proxmox on a small two bay+two nvme Terramaster. I have a bare bones Ubuntu LXC running Samba configured for Apple Time Machine, an VM running Scrypyed, and PBS for Proxmox backups. Nothing on it is critical so I don’t bother with any storage redundancy.
I have the F8 Plus, great little unit. It did need a BIOS update when I first got it to enable Proxmox/other OSes to work properly.
Recently replaced the internal USB boot drive with a small NVMe USB enclosure; using a 90-degree USB connector and using a dremel to sand away an opening for the cable to come out so I could mount the enclosure externally.
It's a horrible idea likely, but I have an ancient old Dell PowerEdge R510. Probably sucks way too much energy, but it does what it does and the price of SSDs have skyrocketed so I'm not touching it.
this is really cool. I've been dealing with an aging Synology nas and this is something I can pick up, evaluate performance and how safe it is to serve as home for my data.
Yeah no. When it comes to backups and data storage, I would rather use a proven reliable system that’s been used and tested by millions of other users, keep these hacky stuff for your hyperland set up.
Not really, zfs is part of it, but not the whole picture, you as a user you need more than just a file system.
Truenas for example add replication, external cloud sync, gui (yes it’s important not to wipe your data with wrong command), HA, custom caching, containers to extend ot with other apps, raid-z expansion with openzfs, and other for monitoring, user management and a lot more.
Other solutions also add more than just a zfs with ssh access.
Run ZFS backed filestore on FreeBSD, have migrated it to/from Debian. At work and home, not petabyte scale but certainly multi hundred terrabyte. Over 15 years, on over 50 hosts/NAS/SAN instances, different hardware.
Run ZFS on Raspberry Pi, on home builds, on Intel, on AMD, on other ARM chipsets.
I think you're over-stating things. Debian is fine for this. I do think FreeBSD is a better platform for myself.
The code bases adhere (modulo ZFS version numbers) to a spec and you can safely migrate the pools between OS. I've done it multiple times both directions.
You can not do this with BTRFS and other Linux things, I consider this feature of (Open)ZFS a killer-context for me: It's OS portable. I wish Mac OSX hadn't walked out of the room when Oracle went legal.
There is actually a btrfs driver for Windows [0]. I've used it a few times before, and it works surprisingly well. You probably wouldn't want to use it for any serious work, but that's not because it's technically flawed, but more because it isn't extensively tested or commercially-supported.
Yet everyone is (again) lost in the details and missing the big picture, which is Linux is doing its best to rat fuck OpenZFS at every opportunity, the last of which was the elimination of write_cache_pages in 6.18 behind the GPL iron curtain a mere few months ago.
I don't know about you but I don't want to build my file storage atop hacks on top of more hacks. The kernel has made it clear non-GPL code is not welcome. Struggles will continue in perpetuity. There are better options.
> Running ZFS on anything but Solaris/Illumos/FreeBSD is asinine.
> ZFS is a permanent second-class citizen on Linux
Linux is the primary target of OpenZFS [0] [1], and has been since 2020 [2]. It may not be supported by the Linux kernel developers, but it's supported by the ZFS developers, and that's all that really matters.
> I don't want to trust my data to some half-assed out-of-tree solution that may or may not break in a week.
Sure, it's an out-of-tree module, but that doesn't mean that it will randomly break all the time; it just means that you may occasionally need to wait for a new OpenZFS release before upgrading your kernel.
> FreeBSD ZFS support has matured and is outstanding.
Agreed, but Linux and FreeBSD both use the same ZFS [3], so I don't really see how the ZFS in FreeBSD can be better than the one in Linux. The tooling and install procedure is certainly better on FreeBSD, but the actual filesystem code is the same (and is probably slightly more robust on Linux since that's going to be where most of the testing occurs).
If you run ZFS with an LTS kernel you're pretty much fine. Yes new Linux releases will break existing ZFS releases - but the LTS tree is in support for long enough that this is never an issue.
I am a BSD user yet I think this is largely irrelevant in the scope of a NAS which is the conversation we have here. Even in the cases bleeding edge hardware support is relevant, the latest Linux LTS kernel has probably wider hardware support than the freebsd-current at any given point in time anyway.
FreeBSD is a great choice, but there is no need to invent silly reasons to justify using it.
I ended up on shucking 4x the 14 TB WD Elements Desktop. They contain helium drives, the WD140EDGZ in my case, and are about a third cheaper than 4x the 12 TB WD Red Plus drives (which are air-filled). The shucking was easier than I expected too, and the performance seems very comparable. The warranty is a definite downside (European, so no Magnuson-Moss), but I think I can even get them back in their enclosure should they fail during the 2-year warranty period.
I've put some second hand 256 GB M.2 SSDs in there as boot drives. It was a bit of a struggle to get it to work in a way that failure of one of the drives doesn't hold up booting, combined with LUKS, TPM keys and ZFS on root. Learned a lot about systemd-boot which I have never used before, but feels a lot saner to me than grub ever was. So now I have a large script which debootstraps a Debian based NAS into being.
I noticed that there are a lot of ZFS myths and cargo culting. For example TFA mentions ECC RAM, which in some circles is a must-have because ZFS would wreck your pool during a scrub otherwise, which is a myth. It's also very expensive, especially this year. You also don't need much RAM for ZFS, L2ARC doesn't use much RAM at all, to name a few others.
Still doubting about setting `dnodesize=auto` (which is the default), because there are some horror stories about that [1]. And it seems impossible to find a cloud storage provider with reasonable prices that supports `zfs send`. Rsync.net upped their minimum order to 10 TiB recently, which is far too much for my use case.
[1] https://github.com/openzfs/zfs/issues/11353
[2] https://www.rsync.net/products/zfsintro.html
reply