
ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015) - SXX
http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html
======
mrmondo
\- Very old kernel used (3.10!), makes me wonder how old the packages like
btrfs-progs are as well.

\- BTRFS not mounted with compression (compress=lzo)

\- Don't use QCOW2, just don't, it's slow and you're just adding extra layers
where you don't need to.

It would be interesting to see you re-run these tests using a modern kernel,
say at least 4.4 and either raw block devices or logical volumes along with
mounting BTRFS properly with the compress=lzo option

~~~
ryao
The benchmark configuration appears to be designed to evaluate the use of
storage technologies for a KVM host. Consequently, saying to use raw block
devices when giving tips for improving btrfs performance is contradictory.

Also, there are a large number of people that will not run a newer kernel for
several years because they are on RHEL6 or RHEL7, so while newer kernels are
interesting, we should not discount the results on the basis that the kernel
is old. The latest ZFSOnLinux code is able to run on those kernels, so while
btrfs remains stagnant there, ZFS will continue to improve.

As for rerunning the tests, using recordsize=4K and compression=lz4 on ZFS
should improve its performance here too. Putting the VM images on zvols (where
it would be volblocksize=4K) rather than qcow2 also would help. In ZoL, zvols
are block devices.

~~~
mrmondo
I disagree (politely) with your assumption about QCOW2 and BTRFS performance,
if I get some time this week I'll do some objective benchmarking (I design and
build Linux based storage systems) and see hey I might be proven wrong but
regardless I can let you know the results if you're interested?

People not willing to update their kernel in production environments is a
social problem, not a technical problem. The kernel is one of the most
reliable, well tested and reviewed software projects in the world. When you
upgrade your kernel 99.5% of the time you get new features, performance and
bug fixes without any negotiate impact. There are of course rare corner cases
especially if running propriety hardware when they are generally slower to
release updates that show the benefits of a modern kernel.

The problem there is the culture and traditional slow moving operational
engineered haven't all embraced the well proven fact that regular, small
charges are safer and have various added benefits.

There is also a serious language barrier between many engineers and management
/ project managers who clearly would not be likely to understand the benefit
of upgrading to a kernel version that say properly supported the new SCSI
blk_mq backend for storage, so there either needs to be degree of trust and
respect to (proven) engineers(ing) and teams or they need to be clearly taught
the value add of a fast release cycle and practising quick (hopefully
automated) patching. That's where I think books like Gene Kim's - The Phoenix
Project and his soon to be released book - The DevOps Handbook which may in
fact be more useful to people performing PM/PO tasks.

~~~
ryao
You said "Don't use QCOW2, just don't, it's slow and you're just adding extra
layers where you don't need to.". I was agreeing with you when I said that it
also has that effect on ZFS. What is my "assumption about QCOW2 and BTRFS
performance"?

As for newer kernels, code churn tends to break things that previously worked,
which upsets customers who want bugs to decrease as a function of time, rather
than go up. Vendors like Redhat will not update distributions like RHEL to
newer kernels because of that and they will not support kernels that they do
not ship. That is why people run older kernels.

~~~
mrmondo
That's complete fud regarding newer kernels - the linux kernel gets more
stable over time, not less - it's like updating your drivers - you don't hold
back your graphics card drivers because the new stable release might be less
stable than your outdated one that came bundled with your PC.

------
beefhash
Why this article had to be segmented into 10 pages is beyond me, however.

~~~
_Codemonkeyism
The reason I didn't read the article.

~~~
frou_dh
Very interesting - thanks for announcing your non-reading!

~~~
_Codemonkeyism
My pleasure!

------
matt-attack
What is with BTRFS? It stands out in every single test (and not in a good
way).

~~~
josefbacik
Kvm (or any overwrite workload for that matter) is the worst possible workload
for btrfs because of COW. We have ideas to address this but honestly it's not
high on the list.

~~~
loeg
Shouldn't ZFS have similar problems with overwrite workloads?

~~~
josefbacik
ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times
as many core contributors at any given time so I imagine they've solved this
in some novel way by now.

~~~
tolle
Wasn't ZFS even "production ready" before BTRFS development even began?

I don't remember the state of it when it was first introduced into Solaris,
maybe someones memory is better then mine. Was ZFS better of in 06-07 Then
BTRFS is now?

~~~
ryao
btrfs development started in 2007 while ZFS development started in 2001. If
you want to do a point in time comparison, look at how ZFS was in 2010. You
can get the last copy of OpenSolaris for the comparison. It would need to be
done on physical hardware due to poor support for virtualization back then.

By the way, ZFS was deemed production ready after 4 years of development.

~~~
tete
That's true. One quick side note though: BTRFS actually had a fairly long
"draft period" (multiple years) of design. I have no clue how long that was on
ZFS. I just know that the original author mentioned that somewhere.

~~~
ryao
If you have time, you can watch this video where Jeff Bonwick describes the
birth of ZFS:

[http://m.youtube.com/watch?v=dcV2PaMTAJ4](http://m.youtube.com/watch?v=dcV2PaMTAJ4)

I am the guy who asked the LZJB question. To summarize my recollection, formal
design work on ZFS started in 2001 when Matthew Ahrens started working at Sun.
Jeff Bonwick had promised Matthew Ahrens a job at Sun making a filesystem a 6
months to a year before then when Matt was still in college. I am sure that
both Jeff and Matt had some thoughts on it during that time, but there was no
formal effort until Matt's employment started.

------
ars
This is more than a year old. I've been told that btrfs has improved greatly
in that time.

~~~
ryao
ZFS has not stood still during that time either.

~~~
tux1968
But it had a lot less low-hanging-fruit to pick during that time.

~~~
ryao
I could argue otherwise, but that is beside the point on enterprise
distributions where these tests were done. The latest ZFS code is always
avaliable on those while the btrfs code is stagnant there. The btrfs
developers delegate responsibility of backporting their changes to
distribution developers, who never do any backports. Consequently, the newer
btrfs code is not very relevant as far as benchmarks on RHEL and others are
concerned.

------
LeoPanthera
*on Linux.

ZFS on Linux in particular is not representative of ZFS on BSD or Solaris.

~~~
georgyo
Not sure what you mean here. They showed that ZFS performed well on linux. Are
you saying that ZFS performs poorly on BSD and Solaris?

Love me some ZFS and it works well pretty much everywhere. OpenZFS shows good
promise of keeping (bringing) the BSD, Illumos and Linux versions in line with
each other.

~~~
protomyth
> Are you saying that ZFS performs poorly on BSD and Solaris?

I think LeoPanthera is saying it performs better on BSD (well FreeBSD) and
Solaris (well Illumos-based in particular).

~~~
tete
Another reason is that ZFS is tightly integrated with other things (for
example jails and zones) on those systems, having switches related to other
parts. I don't think that it is quiet there yet on Linux, even though I am
sure that will come.

------
tete
Here a few things to note:

BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole
idea of the FS is intended to be copy on write), which actually makes BTRFS
look even worse compared to ZFS, cause BTRFS writes once instead of twice and
has most of its features not work, while still performance pretty bad. But
then it's also younger.

Both ZFS and BTRFS (only know specifics of ZFS) can be configured to be better
for DB workloads. On ZFS, I know a couple of people using it cause it has nice
properties.

Anyway. You might not want to use ZFS or BTRFS for a (pure) database system
when performance is the important thing (compared to data security).

What I find kind of missing is UFS because of the BSD world (it's kind of what
Ext4 is in the sense of "your general purpose FS with good performance for
DBs"), but okay.

And yeah, that all may sound a bit biased towards ZFS, but so far my
experience with ZFS has been rather pleasent compared to BTRFS, but again. You
might wanna use UFS or Ext4.

It's also kind of "wrong" to compare those file systems. It's like while you
could use Redis and PostgreSQL for the same things it's probably not what you
want to for one reason or the other.

But then of course it's good, cause you might really have a thing where you
want to have a comparison of those two things to take the right decision for
your specific application. Like those cases where you say "It's slow, but it
makes a lot of thing easier" or "It doesn't guarantee that, but when I take
care of it in the design I will only need only a fraction of the resources.
And the other downsides don' t annoy me thaaat much".

Because the question came up. COW means Copy on Write and it really means what
it says. You copy the data (so have find and to write blocks, duplicating your
data on write) which in case of a full blown database which does that again
(WAL, Autovacuum, keeping lots of metadata, etc.) you really shouldn't be
surprised that it's a lot slower on write heavy sytems. It is more than
expected.

On the other hand because the FS does a lot of things in a similar way to a
database, other than snapshots and replication and all that cool stuff ZFS
does things related to metadata extremely quickly and that's why some CDNs
that have many small files use it. Besides just being an amazing thing for
managing your data in general.

Also if you wanna learn more a about ZFS and have a good understanding of how
you can run databases and other things on ZFS and really utilize it (not just
gaining more performance) then I highly recommend the book FreeBSD Mastery:
Advanced ZFS.

~~~
specialist
Is there something like Aphyr's Jespen for file systems?

[https://aphyr.com/tags/Jepsen](https://aphyr.com/tags/Jepsen)

I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the
technical chops, gumption, wherewithal to figure what's what. Reading posts
(comments) about the edge cases where they fail (data, performance, missing
features) leaves me more baffled.

~~~
ymse
Not the same, but there was an article about file system fuzzing recently:
[https://news.ycombinator.com/item?id=11469535](https://news.ycombinator.com/item?id=11469535)

The presentation is focused on how the fuzzing technique works rather than
individual file system performance, but the "time to first bug" is quite
telling. Copied here:

    
    
        ext4 (2h)
        XFS (1h45)
        GFS2 (8m)
        NTFS (4m)
        NILFS2 (1m)
        HFS (30s)
        HFS+ (25s)
        ReiserFS (25s)
        OCFS2 (15s)
        F2FS (10s)
        BTRFS (5s)
    

I've had miscellaneous failures with the bottom three, but XFS and ext4 have
been rock solid. Unfortunately ZFS and CephFS were not included.

Would love to see a Jepsen-style analysis of the various file systems too.

~~~
ryao
That sort of analysis is testing something on most of those that happen to be
two different things on btrfs and ZFS. Specifically, corruption that passes
checksums and corruption that does not pass checksums. btrfs' result was
obtained by modifying it to disable checksums. How btrfs or ZFS do without the
benefit of checksums should only matter for people exchanging disk images. In
that case, it would be better to have a userland driver regardless of the
filesystem. NetBSD is able to run its filesystem drivers in userspace for this
reason.

------
Zardoz84
Interesting. But in our case, we are running Hyper-V VMs (son sadly NTFS on
host), were the VMs uses EXT4, and a few uses BTRFS.

The only problem that we had, was with a uncontrolled poweroff a year ago.
Looks that BRTFS had some trouble recovering from it, but we manage to restore
all data from the partitions. However, the flexibility that offer BTRFS
(transparent compression, increasing hard disk space on demand, etc) is really
nice. I hope that it improve more, so we can use it without any issues.

~~~
tremon
_increasing hard disk space on demand_

Note that ext4 on LVM can do this as well, you just need to plan for it in
advance. All of my systems are configured as fs-on-lvm, even the single-disk
ones. I've found it's just less hassle: I can do all storage migration or
storage expansion without ever taking the machine offline.

~~~
newman314
This.

I just realized one of my Ubuntu VMs was built without LVM and only found out
because I was going to expand storage on it. Luckily, it's just a toy VM on
the home server so no biggie. But yes, having fs-on-lvm should just be the
default now since it makes expansion so much easier.

------
Already__Taken
I'd like to see NTFS somewhere in these comparisons just to shake some of that
"grass is greener" thinking. The amount I've read on the various file systems
I feel like there's a lot on the table there to get out of our hardware by
just using a better system. But I've not seen a whole lot about ntfs, because
it's all we've got in windows land I suppose.

------
yAnonymous
My ad blocker doesn't care that you separate the article into 11 pages. It
only puts more load on your server.

------
newman314
Got a Synology NAS showing up today. Based on the results here and the fuzzing
link posted several days ago (and elsewhere ITT), it looks like I'll be going
with ext4 for now.

Amazingly, I've had a ReadyNAS device last me over 10 years and here's to
hoping this one lasts a similar period of time.

------
yashau
Mirror?

~~~
ryao
[https://web.archive.org/web/20160403001544/http://www.ilsist...](https://web.archive.org/web/20160403001544/http://www.ilsistemista.net/index.php/virtualization/47-zfs-
btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html)

