
The potential risk to ZFS created by the shift in its userbase - zdw
https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSUserbaseShiftRisk
======
kev009
The nail in the coffin was Delphix switching away from Illumos for similar
reasons the author cites in another article. As they drive most of the complex
feature development directly with staff developers, and by virtue of having
Matt Ahrens on staff drive a lot of the leadership and organization of ZFS, it
makes sense to follow their lead as the repo of truth and lower the barrier to
entry (illusions-gate wasn't working out).

Since FreeBSD support is going to be merged directly into the ZoL repo, ZoL
devs will see the FreeBSD CI build results and it will be a lot closer to a
unified upstream than the past. Being a downstream instead of integrated into
the source of truth made feature development a laborious two way sync and
communications were spread over many mailing lists and bug trackers. I think
this move will be good for both communities, companies like iX Systems will
help get trim across the finish line in ZoL.

In terms of shifting userbase, FreeBSD might even win in terms of ZFS revenue
or other metrics like shipping systems due to FreeNAS (would have to spend a
few hours researching the ZoL companies business models since a lot of the
work is subsidized by the US Government via LLNL). I don't think there will be
any tectonic shifts in userbase, there are still pros and cons to both Linux
and FreeBSD in different areas and being behind in ZFS wont be wont be a con
for either now.

~~~
bubblethink
>I don't think there will be any tectonic shifts in userbase

Unrelated to this post, I'm quite interested in knowing how Canonical's move
has worked out so far. Does anyone know if they've got customers at any
meaningful scale who are using ZoL ? Or is Canonical contributing to ZoL ?
There may not be an immediate shift in userbase, but Canonical can certainly
make a more comprehensive bid to future customers with their overall offering
that also includes ZoL, compared to a specialty storage provider that supports
ZFS on FreeBSD.

~~~
kev009
I think it has mostly lowered the barrier to entry for Linux users already
interested in ZFS, which is a good thing for user experience, but I haven't
seen Canonical driving any feature dev and they aren't doing extensive
backports or uplifts into their LTS distros to stay current with ZoL AFAIK. If
they wanted to monetize it better, they could do a spin for the SOHO market
that either helped replace proprietary filers the same way FreeNAS does, or
highly integrate it into their private cloud stack.

------
barrkel
How real is the risk?

ZFS on Linux is loaded via DKMS. DKMS is basically the compile-from-source
alternative to binary blob drivers. Is there a reason to believe that ZFS via
DKMS is materially different, legally, to Nvidia drivers being loaded via
binary modules?

Indeed it appears that Nvidia proprietary drivers these days go via the DKMS
route. Are they also at risk?

~~~
joecot
The risk is in trying to keep compatible with the Linux Kernel, when not part
of the Linux Kernel dev team.

If ZFS was a module within the Linux kernel source tree, Linux kernel devs
would keep it in mind when making changes to the Linux kernel's APIs. They
wouldn't make potentially breaking changes to that API, or replace APIs with
ones that can't be made compatible with ZFS at all, without considering the
effect it would have on ZFS. The Kernel devs have to take the whole tree into
account, including all the modules.

But since ZoL is not part of the kernel, and is instead a separate independent
module that gets loaded into the kernel, the ZoL have to play catch up.
Instead of Kernel devs working with them to make sure things stay compatible,
or giving them a heads up when something needs to be changed for a new
version, the ZoL team is on the outside, trying to make sure they don't get
boxed out of the Kernel API.

It can be a tenuous position to be in. For a more everyday example, look at
the Firefox and Chromium browser updates. For Firefox Quantum they completely
revamped their extensions system, making all existing extensions worthless.
Either the extension devs rewrote them with the new API, or the extension is
just gone. Similarly Chromium is changing its API for how extensions can
interact with the DOM and networking, which would make Ublock unusable. If
these features were officially part of the browser or official extensions, the
browser devs would work with the extension devs to make sure things stayed
working; instead the extension developers need to keep up or watch their
product die.

If the Linux Kernel completely changed its module API, and ZoL was caught flat
footed and needed to make substantial rewrites, what would the projects
backers do? Would they commit the dev time and resources needed to make those
changes, or would they compare the benefits of ZFS vs the competition --
solutions based on btrfs or LVM, both built into the kernel -- and decide to
jump ship before committing more time? That's where the danger really lies. If
the Linux Kernel makes a serious breaking change, ZoL's supporters might
double down to fix it, or they might just cut their losses.

And while there is an official way to integrate external filesystems into the
Linux kernel, FUSE, it's extremely slow compared to Kernel native modules.
There was a userland FUSE implementation of ZFS, and there's a reason it's
fallen out of favor. ZoL is significantly faster because of its direct kernel
extension, but it's playing with fire by not being part of the kernel tree
itself.

~~~
DiabloD3
>ZoL was caught flat footed

Which would essentially be impossible.

OpenZFS is designed in a way that uses a shim layer between ZFS's internal
Solarisish API usage and the native OS's usage This allows the same ZFS code
to run essentially unmodified on many *nixes without major changes.

In ZoL, this is called SPL (Solaris Porting Layer), and is one of the two
kernel modules required to use ZoL. Linux does not export the proper APIs that
ZFS requires to work here, and SPL fills that gap.

ZFS modules do "break" often due to internal API changes, but the fix is
usually shipped in ZFS stable before the kernel itself is shipped. Only people
who follow ZoL development closely ever see the sausage being made.

It is highly unlikely that Linux can ever do anything that ends up with a
situation that the SPL and ZFS kernel modules can't easily #ifdef their way
out of. It wouldn't make sense for Linux to, either.

~~~
luke0016
> ZFS modules do "break" often due to internal API changes, but the fix is
> usually shipped in ZFS stable before the kernel itself is shipped. Only
> people who follow ZoL development closely ever see the sausage being made.

Eh, it depends on the distribution. I use Fedora, and there have been quite a
few times when a kernel update has resulted in ZoL becoming useless for a
while until they push an updated version. The solution, of course, is just
booting the earlier kernel until ZoL catches up.

In fact, this is happening right now. The 4.20 kernel was pushed to updates on
Fedora (at least my system) on 1/23\. The latest stable ZoL doesn't work with
it. They're on RC3 for the next release, though, and that does. Hopefully it
comes out soon.

In the mean time, I'm running the last 4.19 kernel. _shrug_

~~~
kbenson
> I use Fedora, and there have been quite a few times when a kernel update has
> resulted in ZoL becoming useless for a while until they push an updated
> version.

Alternatively, you can use the variation of a distro, if it exists, that
provides a more stable kernel and package version set. In this case, that
would be RHEL/CentOS.

For a workstation, that can be annoying, but as you said, just use an older
kernel for a bit longer. Perhaps mark it as to be skipping in normal package
operations, and have a cron job that runs to check the kernel specifically and
emails you if there are updated versions of the kernel that exist.

For a server, I imagine it's rarely a problem since those should be running
more stable distros anyway, since 99% of the time the kernel is older and
back-patched (it _may_ be the current kernel, but still weeks after it was
released right at point release updates), which should result in a stable API
for ZoL.

The trade-offs there don't seem too onerous to me. A little hand configuration
for a workstation (of which there's likely one or two for a person to deal
with as long as there's smooth server support which is generally both more
important to have solid because it can be harder to fix if there's a problem
and because it can scale from none to many per person.

~~~
vbezhenar
How does RHEL help there? They ship kernel updates. Those kernel updates once
broke HP b120i properietary driver (HP releases new version of this driver for
every minor RHEL release). I don't see how fakeraid driver is fundamentally
different from ZFS.

~~~
kbenson
Kernel updates are back patched. That means that in-between point release
updates (which generally happen every 6-12 months) the kernel version stays
the same, and any bug-fixes are ported into the older kernel that is shipped.
Point releases _may_ update the kernel version (I think?), but generally keep
it the same as well, but they will back port some features into the older
kernel as well, not just bug/security fixes. You can see here[1] for RHEL
versions and the kernels they ship with.

If a security fix breaks your ZoL integration, my guess is you're actually
better off waiting for that to play out and resolve itself than to expect it
to work. If a feature back port breaks it, that might be a little more
annoying, but I imagine it will be fixed in short order, and you only have to
worry about that once every 6-12 months (and it's well publicized).

> Those kernel updates once broke HP b120i properietary driver (HP releases
> new version of this driver for every minor RHEL release).

If HP is releasing closed source drivers for RHEL, I imagine they would want
to be on the certified hardware list and test, or at a minimum seek access to
the beta spins of the point releases (which I think is where it broke) so they
can test before it comes out. I'm not sure I blame Redhat for HP trying to
specifically support RHEL and failing to do so, given the systems I know they
have in place to help companies in just that situation (because it helps RHEL
users).

In any case, all I'm really noting is that between Fedora, which ships a new
kernel version every kernel update (AFAIK) and RHEL/CentOS, which ship larges
the same kernel with only the specific changes needed the majority of the
time, keeping ZoL working should be vastly easier on RHEL systems (and in
fact, any OS which does back patching of kernels, which I believe includes
SuSE and the LTS releases of Ubuntu).

1:
[https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_...](https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_7)

~~~
vbezhenar
It's not really the same kernel. They backport a lot of features with every
minor release. They still call it 2.6.x or whatever, but it really is
different. I know that RHEL has some subset of internal kernel API that they
promised to keep stable within major release, so if HP failed to rely on those
API, it's their problem, but it might happen.

------
cracauer
Speaking of all this, am I the only one where NFS clients with executables on
NFS (such as PXE booted machines) get bus errors when the server runs recent
4.19/4.20 Linux kernels with recent ZoL code?

As a reminder, those bus errors are supposed to happen when a client-side
mapped file has been unlinked on the server. So the ZFS export does not
provide pages to the client on a client page fault the same way NFS would on
deletion. Tried dedup and compression on and off, no change. A reboot of the
server clears it up for a couple of days, but then it starts the same way.
Something seems very broken in caching there.

This popped up during maybe the last 4-5 months.

------
stallmanite
Interesting article. So is the “unfriendliness” of Linux kernel developers
towards ZFS that is mentioned strictly due to licensing issues or are there
other points of contention?

~~~
dspillett
BTRFS is being positioned as an alternative to ZFS (amongst others) without
the licensing issues (and less rational concerns like the current/past
politics of it all), so perhaps there is a "why do we X it when Y is close"
with a bit of extra NIH syndrome mixed in?

~~~
CrLf
Isn't btrfs also at risk? I was under the impression that it lost some steam
in the recent past and was on its way to become the next Hurd.

~~~
vetinari
btrfs has a problem with it's image as being unreliable (some well deserved).
Redhat obsoleted it, because they use kernel 3.10 in RHEL7 and they were doing
nothing but backporting, without having any btrfs expertise in house anyway.
Btrfs continues to be developed by Oracle, Suse, Facebook, Fujitsu and the
rest of the usual suspects.

~~~
walrus01
RHEL is famous for using kernels that are _ancient_ , even by the notoriously
conservative and risk averse debian-stable standards.

I can't even imagine the hassle of backporting stuff from much newer kernels
into 3.10 to make btrfs work.

If you want to use btrfs in production (not the best idea, IMHO, but orgs such
as facebook do it) you absolutely need to be running a 4.9.x or better kernel.

~~~
snuxoll
Calling them ancient is a bit unfair, they regularly backport hardware support
and select other features from newer kernels. Unlike other distributions,
however, they have a stable kernel ABI for an entire release, so they can’t
bring in changes that break white listed symbols.

------
j16sdiz
Zfs is very modularize. Most of the platform specific code is in SPL. Don't
think this would affect freebsd

------
lmm
I would speculate part of Java's success is due to being available but not
deeply integrated on Linux-based OSes, for similar legal reasons. As with ZFS,
it became very easy to upgrade your version of Java without having to go
through an upgrade of the whole baseline OS (which was often the natural path
if you wanted to use a new version of, say, Python).

Of course ZFS has a lot more potential to be broken by kernel updates than
Java, but the parallel is interesting.

------
xxpor
Why was CDDL chosen for ZoL?

~~~
bromonkey
ZoL was forked from Oracle ZFS, if you want the real answer you need to ask
Oracle, presumably it was so that ZFS would be incompatible with their
competition (Linux GPL based work)

~~~
simcop2387
Even before Oracle, Sun had licensed it as CDDL. So Oracle itself may not be
able to answer that question anymore.

~~~
yc-kraln
CDDL includes patent grants which GPL (et cetera) does not. CDDL allows also
file-based licensing, which allows for projects with combined free/nonfree
sources to be distributed, which was a requirement for Solaris. It's not that
they chose CDDL to make it not work on Linux, they created CDDL because they
didn't have a choice.

~~~
BurningCycles
Sun created CDDL in order to be Linux GPLv2 incompatible, even the person who
was tasked by Sun management to write CDDL (Danese Cooper) said so.

And it's not as if that admission was needed, as anything else would have been
insane from Sun.

They were losing badly to Linux, so in the eleventh hour they decided to go
open source in an attempt at getting more mindshare = marketshare.

Solaris had some real technical advantages over Linux like ZFS and DTrace,
however if they licensed Solaris with a GPLv2 compatible license, Linux could
just incorporate their coveted tech, which would be business suicide. Hence
CDDL.

You can't blame them given the situation they were in.

------
fxfan
Speaking of filesystems- is there one that allows me to add tags? Or alternate
streams (like NTFS does), so I can add my own tagging system on top?

~~~
drbawb
As someone who has spent a lot of time on this problem: you don't want to
store tags as attributes/resource forks/streams/etc in a traditional
filesystem. The single most important characteristic for a tag-based
organization system is that search operations must be extremely fast and
cheap. Walking a hierarchical filesystem and stat'ing a bunch of files to
discover their tag information is anything but cheap.

What you want is to store the tag metadata heap in an easily indexed structure
(i.e: a search tree.) That heap then just contains pointers into content
addressable storage. You can kind of build such a thing on top of existing
filesystems which support hardlinks, however I'd advise against that for the
metadata itself. Once you have the power of tagged content a natural evolution
is querying that data using set operators. You will inevitably re-invent (some
approximation of) SQL once you arrive at this discovery.

~~~
the8472
It might still make sense to keep the canonical metadata in xattrs so that
tools can interoperate on it and build a search index from it when you
actually need to do search.

On a NVMe drive rebuilding the search index shouldn't be too costly, even on
hundred thousands of files.

I think there were also some ioctl additions for btrfs that let you
sequentially walk the filesystem by extents or inodes instead of by directory
tree, which makes indexing a lot easier.

