
Ask HN: ZFS on FreeBSD vs. expensive NAS for 100 TB backup over NFS? - reacharavindh
Fellow HNers!
I have a chance to upgrade our backup server that is an aging NetApp Filer costing us big money. 
Is it a good idea to setup FreeBSD-ZFS on Directly attached JBODs and use it over NFS?<p>Or are there any practical reasons I should look into enterprise NAS vendors(NetApp, EmC etc al)?<p>I imagine the server and disks I&#x27;d buy would come with support, and I can replace them as needed. It&#x27;s usecase is stupidly simple:
4 NFS mount points to a single host that rsyncs our primary storage every night. 
One backup admin user and nothing fancy.<p>Any experiences, suggestions before I step into building this thing?
======
zwerdlds
Programmer's opinion here: I'm not a sysadmin, just like to do light HomeLab
style hobby work at home.

I can't provide a counterpoint for why you might consider a proprietary NAS,
but I can tell you that I'm 90% sold on FreeNAS. I've built two servers
running them from commodity hardware and never once had a problem with it.

I use rsync to backup to my older NAS no functioning specifically for that
purpose. It's a ReadyNAS x86. I moved away from that because I wanted more
functionality and redundancy.

I've had 2 drive failures on 5-disk setup. Live rebuild in degraded status
worked as well as it should have. I also use the SSD caching feature which
sped things up a bit for frequently used files.

My only qualm about the product is that whole versioning fiasco. I actually
almost got bit by it while building a server for my parents, but the install
wouldn't take so we used the older version. But by the grace of luck we didn't
use the bad version.

I use separate hardware for my service host which talks over NAS. I just use
the FreeNAS boxen for storage. Learned that lesson!

~~~
reacharavindh
Thanks for the comment. I'm a sysadmin albeit only recently changed into this
role. FreeNAS was my first choice when I thought about building this solution,
however, instinctively, it feels over engineered for my needs. Sure, if I want
a NAS with GUI, Jails and running VMs etc, I'll run to FreeNAS. But, I'd like
a minimal core (FreeBSD + ZFS + NFS + rsync) which lowers the failure surface.
I will write a few scripts to keep an eye on the health of the server, and
that'll be it.

Good to know that your NAS survived a 2/5 disk failure.

~~~
DKnoll
While I would personally prefer FreeNAS (or just to roll my own), remember
that you're supposed to be planning for the next person in your position.
Sometimes a commercial appliance really is the best way to go. If you do roll
your own or set up FreeNAS you need to build lots of docs and make sure
they're stored somewhere the next guy will find them.

Ultimately it depends on the company, it's just a compromise I have had to
make before so I thought it worth mentioning.

~~~
reacharavindh
Good point. I'm thinking of this custom solution only because of the
simplicity of this requirement(NFS backup). I'm planning to prepare a thorough
documentation of what I did to build this solution and how to maintain it.

------
nailer
There's a storage startup that posts their hardware designs openly on here. I
forget the name though.

Edit: [https://www.backblaze.com/blog/open-source-data-storage-
serv...](https://www.backblaze.com/blog/open-source-data-storage-server/) Cost
is $0.036/GB

Alas it's ext4. You could do XFS if you want something in the mainline Linux
kernel with a long history, btrfs or ZFS-on-Linux if you're happy with one or
the other, or just run ZFS/BSD as you mention if you've got BSD skills.

Edit: I know it's not the done thing on HN to complain about downvotes, but
why on earth would anyone downvote an entirely informational post?

~~~
bradknowles
With respect, Backblaze can’t be used as a NFS server replacement.

It’s fine if you want to use them for storing backups with their proprietary
protocol, or use S3-style bucket storage on their B3 service, but that is
totally not the same thing.

~~~
nailer
It's principally a hardware design. You can use it with whatever software you
like.

~~~
bradknowles
Take a close look at that hardware design — it is very much intended to be
used exclusively for the purpose it was designed for, with their proprietary
HTTP application on top. They are tightly coupled — it is very hard to
separate the hardware design from the software that would be running on it.

If you were to design a more generally useful storage server, it would look a
lot more like the FreeNAS or TrueNAS boxes than it would the Backblaze boxes.

------
olavgg
The biggest reason going for an enterprise vendor is support! As long you pay
them big money you will get something that just works without worries. If
something should go wrong, they will be on your door ASAP and fix the problem
for you. Very convenient.

If you have expertise and is willing to do self support, creating your own DAS
may be fine. Just keep in mind though, that things can go really wrong at the
worst time when you are busy with something important.

A lot of businesses do bad resource management and don't do proper planning.
This can hurt you.

Because we think most major storage vendors are way too expensive for us, and
we don't have resources for self support, we have landed on using Aws S3 and
Aws Glacier.

At home I've been running a FreeBSD ZFS system for 8 years. Oh boy I have had
many issues over the years because of bad hardware, cheap SATA cables,
consumer grade SATA controllers, consumer grade network cards, consumer grade
hot swap bays, consumer grade motherboards. But now I use only enterprise
stuff, except from the hard drives, which has been very stable for the last
two years.

~~~
reacharavindh
True. This is the reason why our primary storage is an Isilon cluster
supported by EMC. This is something we can't afford to have downtime with.

The NAS I was asking about is a passive backup server. If something goes wrong
and ZFS asks me nicely to do do a clean boot or a scrub, we'll be perfectly
okay. Even if it is down for a few days, we can catch up from snapshots on the
primary storage.

Good point about going for enterprise grade hardware instead of putting
together cheap parts. Duly noted. I will buy a decent server with good support
and lots of ECC memory.

------
sandreas
I recommend taking a look on borg backup
([https://borgbackup.readthedocs.io/](https://borgbackup.readthedocs.io/)) or
restic ([https://github.com/restic/restic](https://github.com/restic/restic))
instead of rsync... and use snapshots for zfs.

~~~
reacharavindh
I use BorgBackup for backing up my personal website (in the order of a few
MBs) to learn. But, my use case in this quest is to build a long-term backup
solution that we can rely on. (10 years+). My concern is Borgbackup being
updated so frequently and being actively developed to add more features. What
is the chance that I can read a backup that I make today 10 years from now? I
do NOT want to maintain a directory full of borgbackup binaries of different
versions just to be able to access older backups. Same goes with running
unupdated version of Borgbackup for 10 years.. Hence my interest to keep
things stupidly simple with NFS --> ZFS server.

~~~
sandreas
Well, that is understandable... but i think, that rsync has major drawbacks:
No history - overwriting means destroying (e.g. ransomware) || No
deduplication - restic transfers blockwise and only once || No encryption -
access to backup means access to all data || No error protection - how to
verify backup success?

In my opinion a good backup should contain a verified restore strategy :-)

You should also consider sending and receiving snapshots, when using zfs
([https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm...](https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html)).
Zfs also supports deduplication, but FreeNAS devs do not recommend using it
unless you have a massive amount of RAM and you know what you are doing.

------
sandreas
Well, if not FreeBSD, you could take a look at OmniOS
([https://omnios.omniti.com/](https://omnios.omniti.com/)). There also is a
project with a web based administration for this os called napp-it, that could
be used as is or, in your case as "minimal installation reference guide"
([http://www.napp-it.org/index_en.html](http://www.napp-it.org/index_en.html))

~~~
reacharavindh
Looks very interesting. However, I'm not familiar with Illumos/OmniOS to begin
with. Even though ZFS is still ZFS, I'd be concerned about dealing with
operation of an OS that I'm not familiar with. I've played with FreeBSD for a
few years, and I know its community help is great along with the wealth of
StackOverflow responses. I'd try OmniOS for a smaller scale private project to
learn it.

------
phil21
This is a perfect use case for ZFS. We run a number of ZoL (ZFS on Linux)
filers in various backup/archival roles, and the cost savings and flexibility
are great. These range in size from about 100TB to 1.6PB.

Since you are a single dev with limited experience with this, hopefully I can
think of a few common gotchyas.

1) Many disagree here, but I do not think ZFS is ready to run as primary
storage for say backing a vmware cluster. We've tried in the past, and there
are far too many "reboot the server" style bugs when you get into heavy load.
Many have been fixed, but simple things like certain types of drive failures
ZFS on FreeBSD still can have issues. We've even experienced this on
IllumOS/Nexenta as well. Data integrity will let you sleep at night, but don't
expect 100% availability comparable to a Netapp or the like. This is why we
use ZFS in a backups/archival role only, where 100% availability is not
required.

2) Understand your I/O needs and how that will impact your choice of mirrored
vdevs over raidz(2) vdevs. You will likely not get the space efficiency you
are mentally calculating either, which is something to keep in mind. I'd take
a look at the raidz efficiency calculations[0] and keep in mind you should
never fill a zpool beyond 90% capacity or so.

3) Shouldn't need to be said - but run ECC memory. RAM is cheap, buy a lot.

4) For rsync based backups (assuming backing up actual user directories and
such) you will have potential for a lot of small file writes. I do suggest a
small SLOG (write cache) ssd for these cases, as random writes against a raidz
vdev consisting of spinning rust can be rather slow. I don't see a reason for
any l2arc (read cache) here.

5) You're doing a one-off. Do not go exotic on the hardware. I recommend a
standard 1U server with the LSI HBA of your choice, connected to an
appropriately sized SAS JBOD enclosure. We've had good luck with both HGST
4U/60 drive units, and Supermicro 4U/45 drive units depending on the density
we need and if top-load is a good choice for a particular facility. I suggest
avoiding top-load as your first deployment, assuming you are not space
constrained. There are plenty of VARs out there who can give you a pre-
packaged deal.

6) Due to the way ZFS stripes over vdevs, it's better to build out capacity
up-front vs. add later if performance is a concern. This is likely much less
of an issue for your use-case but is important for planning. For example if
you know you will need an additional 100TB in 12mo, buy it all at once and add
it to the pool day one. Don't trickle in new vdevs 8 spindles at a time.

7) This is probably the largest pain point for ZFS right now - you will have
to build tooling to properly monitor and administer it. This isn't hard, but
is something to very easily forget or have break and not notice if you're a
single man "ops" team as your second responsibility. Remember to monitor zpool
space usage and to alert early to add capacity. Stuff like a hot spares while
getting better I wouldn't trust to automatically Just Work(tm) yet, so ensure
you have robust drive/pool health monitoring.

8) Snapshots! Remember to use them, they are magic. znapzend is a decent tool
for getting schedules going.

9) Sounds like you won't need it now, but zfs send/receive combined with
snapshots are one of those life changing technologies like Tivo I don't think
I could live without these days.

Overall ZFS is great, if you enjoy a little hacking this should be a fun
project for you.

Edit: links, woops.

[0]
[https://docs.google.com/spreadsheets/d/1pdu_X2tR4ztF6_HLtJ-D...](https://docs.google.com/spreadsheets/d/1pdu_X2tR4ztF6_HLtJ-
Dc4ZcwUdt6fkCjpnXxAEFlyA/edit#gid=804965548)

------
twunde
I'm just going to put this out there since we're discussing ZFS. There's an
open ZFS conference in Norwalk CT in April: [https://www.eventbrite.com/e/zfs-
user-conference-2018-ticket...](https://www.eventbrite.com/e/zfs-user-
conference-2018-tickets-38008213590)

------
myrandomcomment
ixsystems.com

Makers of FreeNAS/TrueNAS. Good team. Great support.

------
touchofevil
I was thinking of using FreeBSD for a large filer as well, but then I learned
that you can run ZFS on Ubuntu, which should have better driver support than
FreeBSD (at least I would think so). So you may want to consider Ubuntu with
ZFS instead. [https://www.howtogeek.com/272220/how-to-install-and-use-
zfs-...](https://www.howtogeek.com/272220/how-to-install-and-use-zfs-on-
ubuntu-and-why-youd-want-to/)

~~~
rleigh
"Better" is a little subjective. Linux has more drivers, true. But you only
need one driver, for the HBA you use, and is the quality of that one driver
better? I bought an LSI HBA specifically because the FreeBSD driver for it is
known to be good (and it also works well with Linux).

Drivers aside, ZFS on Linux is rather more rough around the edges than on
FreeBSD. Having used both, I'd place more trust in ZFS on FreeBSD. While I've
not had dataloss on either platform, I have seen odd glitches on Linux which
required rebooting to resolve, like zpool getting stuck in D state after some
pool operation, and it failing to mount filesystems after rebooting until I
manually reset the mountpoints for little reason I could see, and excessive
memory usage which has frozen the machine on a few occasions when under heavy
load. It's also rather more featureful on FreeBSD; the Linux implementation
has a number of annoying restrictions and missing functionality which aren't
essential for the basics, but make it much more pleasant to use. Like not
requiring root, priv delegation, NFSv4 ACLs, NFS export, ARC integration into
the kernel memory management.

