Ask HN: Btrfs or ZFS on Linux in production? - smilliken
======
patrickg_zill
ZFS but, tune it properly by limiting the size of ARC and buy some SSDs to use
as both log devices and cache devices.

Also, do not buy any controller cards that will get in the way of ZFS talking
directly to the disk (such as Dell PERC cards), where you see a "virtual" disk
instead of the real one, it will affect performance.

~~~
smilliken
Regarding the ARC, is zfs not as efficient about evicting the cache as, say,
ext4? With ext4 I've never had to worry about it consuming as much memory as
it pleases.

Thanks for the tips!

~~~
patrickg_zill
My layman's understanding: under Linux the ARC is not integrated into the
usual Linux caches which are unified and which the kernel understands. So
basically the ZFS ARC is a separate chunk of RAM, which can grow and grow
(thus you need to limit it) and it won't be evicted as perfectly as the kernel
would evict stuff cached on xfs or ext4 filesystem.

~~~
smilliken
Ah, that makes sense. Thanks for the explanation.

------
gopalv
They are both check-summed and copy-on-write.

This is great for consistency and storage, but the first part somewhat sucks
if you've built it into your layer as well (like hadoop fs). And it makes
writes & reads a tiny bit slower than it should be.

They have fat xattr inodes, which is great if you use it - but useless if
something like SELinux means nothing to you.

The point where they truly excel is in a multi-disk scenario.

And in that land, the zpool array implementation makes it awesome to introduce
an SSD+HD mixture with nearly the best of both systems.

BTRFS doesn't have that yet (well, bcache on the lower layer, perhaps).

But as someone who's fired up debugfs a bit too many times, if you really
don't know which one you need, pick the one that ships with your distro!

~~~
smilliken
> And in that land, the zpool array implementation makes it awesome to
> introduce an SSD+HD mixture with nearly the best of both systems.

With an array of half SSDs and half HDs, can I configure ZFS so that I never
hit the IOPS bottleneck of the HDs? How do you recommend configuring the
array?

~~~
gopalv
I have never used a setup like that - all I've done is throw an SSD in front
of 3 disks.

That said, the zfs log and zfs cache command supports mirror disks for log and
allows more than 1 cache per pool.

Using more than 1 SSD as a cache (and of course, using mirrored logs for
safety) would be recommended if you had a production box with this.

To repeat, I have never tried ZFS-on-Linux anywhere near a production setup -
but if I had to, I would use multiple SSDs just in case one locks up its
firmware.

------
hendry
Choose extfs every time. btrfs is bloated.

x220:~/tmp/linux-3.13.2/fs/ext4$ ohcount . Examining 39 file(s)

    
    
    							  Ohloh Line Count Summary
    
    	Language          Files       Code    Comment  Comment %      Blank      Total
    	----------------  -----  ---------  ---------  ---------  ---------  ---------
    	c                    37      31707       8586      21.3%       5038      45331
    	make                  1          8          3      27.3%          3         14
    	----------------  -----  ---------  ---------  ---------  ---------  ---------
    	Total                38      31715       8589      21.3%       5041      45345
    	x220:~/tmp/linux-3.13.2/fs/ext4$ cd ..
    	x220:~/tmp/linux-3.13.2/fs$ cd btrfs/
    	x220:~/tmp/linux-3.13.2/fs/btrfs$ ohcount .
    	Examining 81 file(s)
    
    							  Ohloh Line Count Summary
    
    	Language          Files       Code    Comment  Comment %      Blank      Total
    	----------------  -----  ---------  ---------  ---------  ---------  ---------
    	c                    79      77101      13378      14.8%      13025     103504
    	make                  1         15          0       0.0%          4         19
    	----------------  -----  ---------  ---------  ---------  ---------  ---------
    	Total                80      77116      13378      14.8%      13029     103523

~~~
smilliken
I'd be surprised if btrfs was smaller. Its feature scope is significantly
wider.

The btrfs performance isn't at par yet either, but I'm particularly interested
in filesystem-level compression which isn't supported in ext4.

------
stevekemp
Both have good features, both will be useful and interesting in a few years.

Right now I'd not trust either of them, in production use. Even if you never
spent 60 minutes restoring backups both filesystems are too new for me to
trust, with my conservative tendencies.

------
smilliken
My understanding is that btrfs is the likely successor to ext4, but that zfs
on linux is further along and more featureful.

Are there any data integrity, performance, or other issues to beware of?

------
k3oni
I would go with ZFS, at least for now. BtrFS is getting there but i wouldn't
use it production just yet,

