
ZFS is the FS for Containers in Ubuntu 16.04 - doener
http://blog.dustinkirkland.com/2016/02/zfs-is-fs-for-containers-in-ubuntu-1604.html
======
brendangregg
This is great news, and we're already using ZFS in production on Ubuntu in a
few areas at Netflix (not widespread yet).

Ubuntu 16.04 also comes with enhanced BPF, the new Linux tracing & programming
framework that is builtin to the kernel, and is a huge leap forward for Linux
tracing. Eg, we can start using tools like these:
[https://github.com/iovisor/bcc#tracing](https://github.com/iovisor/bcc#tracing)

~~~
tiffanyh
What about FreeBSD?

Does that imply Netflix is transitioning away from FreeBSD? If so, why?

~~~
brendangregg
Netflix cloud: AWS EC2, tens of thousands of cloud instances, mostly Ubuntu.

Netflix CDN (Open Connect Appliance): lots of physical boxes, FreeBSD.

~~~
tiffanyh
Any reason why you don't use FreeBSD everywhere (or Ubuntu everywhere).

Why two different OS'es?

~~~
brendangregg
It's really two questions: Why choose Ubuntu for the cloud, and, why choose
FreeBSD for the CDN. We believe that's the best choice for both environments.
I was trying to type in an explanation here, but that's really something that
will take a lot to explain (maybe a Netflix tech blog post).

~~~
tzs
If you do write that blog post, it would be cool if you not only covered the
FreeBSD vs. Ubuntu aspects of the choice, but also the Ubuntu vs. other Linux
aspects (particularly Debian).

------
ThePhysicist
ZFS is nice but as far as I understand the Linux version does not yet have
support for copy-on-write clones using e.g. "cp --reflink=always", which to me
was reason enough to choose BTRFS instead. Apart from this the two systems
seem to be quite comparable (from my limited user perspective), with BTRFS
having quite good Linux support as well. Maybe someone more experienced with
the COW functionality could comment on that as it would be very interesting to
hear how other people deal with this.

~~~
aidenn0
I've used both ZFS on both BSD and OpenIndiana, and I've used btrfs on linux.
as recently as 4.0.5

As of 4.0.5 btrfs was IMO completely unusable as a daily file system. Some
examples of issues I ran into:

1) System became unbootable with the version of btrfs I had installed and I
had to use either an older or newer kernel to recover

2) I have a periodic backup of my mailbox that runs, and when it runs my
system becomes completely unusable until it completes. The same script running
on zfs on bsd and with ext4 or reiser3 on linux would show I/O slowdowns, but
I could still use my machine.

3) In general I would run into other minor issues and the consensus in #btrfs
was that since my kernel was more than 3 months old, it was probably fixed in
the latest version, and why would somebody using an experimental filesystem
not be tracking mainline more closely?

[edit]

To be fair, here's some issues with ZFS:

1) Do not ever accidentally enable dedupe on commodity hardware; it will
slowly consume all your RAM if you aren't on a sun server (where 64GB of RAM
is a resource constrained environment), and there aren't effective ways to
undo dedupe, other than copying all the data onto a different pool.

2) You can't shrink a pool. Hugely annoying, apparently non-trivial to solve.

~~~
rsync
"To be fair, here's some issues with ZFS:"

Let me add:

3) Do not allow a pool to exceed 90% capacity ... and probably don't let it
exceed 85%.

ZFS does not have a defrag utility and it badly needs one. You can permanently
wreck zpool performance by running it up past 90% capacity - even if you later
reduce capacity back down to 75-80%. You can sort of fix it if you add
additional top level vdevs to the zpool, thus farming out some IO to the new
set of disks, but it's still going to be performance constrained forever. The
only solution is to create a new pool and export the data to it.

This is unacceptable, by the way.

It is not at all reasonable to require a filesystem to stay below 80% capacity
(our target "full" number at rsync.net) nor is it acceptable that hitting 90%
is a (performance) death sentence.

When you consider that you might have _already sacrificed_ 25% of your
physical disk just for the raidz2/raidz3 overhead, being constrained to 80%
means you're only using 60% or so of your physical disks that you bought.

 _ZFS needs defrag. Badly._

~~~
cthalupa
For ZFS to get defrag, someone with enough ability and knowledge would need to
bite the bullet and go through the fairly massive and difficult task of adding
in block pointer rewrite, which doesn't seem to be something anyone has been
willing to do, and I've seen a lot of concerns about the actual feasibility of
it from some very smart people that are knowledgeable with the codebase.

~~~
XorNot
Which also incidentally would solve the pool shrinking problem as well.

~~~
voltagex_
I wonder how much money you'd need to pay a very talented engineer to do the
work. This is yet another thing I didn't read about before building a home NAS
on ZFS.

------
baldfat
I don't understand why BTRFS isn't gaining more support in Linux. I prefer
BTRFS to ZFS with Fuse.

~~~
zzzzzthrowaway
Probably rhymes with "btrfs balance".

I'v had the misfortune of using btrfs in production with a few hundred
machines on Ubuntu 14.04. It's one of the most finicky filesystems I've ever
used. It's probably better in newer kernels, but if you have a lot of churn it
requires constant care and feeding and tends to cause kernel softlockups
fairly commonly.

Also this: [http://blog.pgaddict.com/posts/friends-dont-let-friends-
use-...](http://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-
for-oltp) and this:
[https://www.phoronix.com/scan.php?page=news_item&px=CoreOS-B...](https://www.phoronix.com/scan.php?page=news_item&px=CoreOS-
Btrfs-To-EXT4-OverlayFS) and this (old but a few of the points still apply):
[http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-
into-a-...](http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-
bar/posts/70)

That being said I'm sure there are plenty of use cases for which btrfs in its
current capacity is more than sufficient.

~~~
robszumski
At CoreOS and we tried really hard to make btrfs happen, but it really came
down to how different it operated than other file systems. It was mainly a UX
issue and thus fell into my lap.

The major issue is that regular debugging tools that folks have been using
forever like `df -h` aren't just non-functional, they actively misrepresent
the state of the file system. The most common example is indicating that you
have plenty of free space when in fact you're out. We had to write a lot of
documentation to teach people how btrfs works and how to debug it:
[https://coreos.com/os/docs/latest/btrfs-
troubleshooting.html](https://coreos.com/os/docs/latest/btrfs-
troubleshooting.html)

The second major issue is that rebalancing requires free space, which is the
problem that most folks are trying to fix with a rebalance operation. Catch-22
in the worst way. Containers vary in size and can restart frequently, churning
through the btrfs chunks without filling them up, leaving around a lot of
empty space that needs to be rebalanced.

~~~
ryao
I hit that rebalancing needs space (and therefore can ENOSPC) issue at work
when trying to compile ZFS on CoreOS on Digital Ocean before CoreOS switched
from btrfs to ext4 and overlayfs. Getting ENOSPC on btrfs rebalance when you
are seeing regular writes return ENOSPC is a really annoying problem.

------
melted
Apple should replace the abomination that is HFS+. They almost did a few years
ago, but pulled the code at the last moment.

~~~
e1ven
Apple's choice to drop ZFS may have been influenced by the lawsuit Sun was
dealing with at the time.

[http://www.computerworld.com/article/2539287/data-
center/app...](http://www.computerworld.com/article/2539287/data-center/apple-
s-support-for-zfs-may-drag-it-into-open-source-lawsuit.html)

I'd wager they wouldn't want to adopt ZFS without an explicit license and
legal indemnity from Oracle.

~~~
zanny
They could adopt btrfs. Darwin is open source, right?

~~~
melted
BTRFS is still not mature, and there's a license incompatibility as well. And
it's controlled by Oracle all the same. If Apple replaces the filesystem
they'll likely roll their own.

~~~
zanny
btrfs is not controlled by Oracle for one (its principal developers are
employed by Facebook, but its still regular Linux GPLv2 code) but I did check
and the APL is incompatible with the GPL.

And they obviously aren't going to make a new filesystem. That doesn't get
them sales like higher resolution screens or changing the color theme...
again.

~~~
melted
Looks like you're right. Oracle used to control it (as evidenced by (c) Oracle
all over the place), but that doesn't seem to be the case anymore.

~~~
ghaff
I'm not sure it's accurate to say it was "controlled" by Oracle but a lot of--
though certainly not all--active development came out of Oracle at one point,
notably by Chris Mason (who is now with Facebook).

~~~
melted
Whoever employs the core team, controls the project. For a while that was
Oracle.

------
mark_l_watson
Sorry in advance if this is a stupid question: my main Linux system is a
laptop with a small SSD drive. I would like to organize my entire digital life
on a 2 TB external USB drive, and be able to maintain a clone of everything on
at least one other 2 TB USB drive.

Is ZFS the right tool for this?

~~~
berdario
I read some time ago that ZFS is definitely NOT the right tool for
laptop/external storage, unless you actually have a zpool with mirroring/raidz
(which means you have to always keep the devices connected).

The reason is that when ZFS detects corruption, it'll lock down the whole
fs... and prevent reading/recovery data from it, as recovering data from raidz
is the expected solution in that case.

I tried to google again for the description of this issue, but I couldn't find
it... I found this otoh:

[https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs...](https://groups.google.com/a/zfsonlinux.org/forum/#!topic/zfs-
discuss/VEV-uKUzP14)

And things aren't that obvious, apparently:

> Even without redundancy and "zfs set copies", ZFS stores two copies of all
> metadata far apart on the disk, and three copies of really important zpool
> wide metadata.

Whichn means that this might not actually be a problem after all

~~~
ryao
> The reason is that when ZFS detects corruption, it'll lock down the whole
> fs... and prevent reading/recovery data from it, as recovering data from
> raidz is the expected solution in that case.

ZFS has duplicate metadata by default, so it can recover from corrupted
metadata blocks unless too much is gone. If the data blocks are corrupted and
there is no resundancy, you should get EIO. There is no code to "lock down the
FS", although if you have severe damage (like zeroing all copies of important
metadata or losing all members of a mirror), it will die and you will see
faulted status. That is a situation no storage stack can survive and is why
people should have backups.

------
nisa
Let's see how this works out. It's probably better and more stable than btrfs
but this is not complicated...

ZFS on Linux had issues with ARC (especially fast reclaim) and some deadlocks
and AFAIK cgroups are not really supported - e.g. blkio throttling does not
work.

Would be great is they got this ironed out but I would be wary. Still great
news!

~~~
AstralStorm
Additional problem is that in-kernel latencies of both btrfs and ZFS are on
the high end. Essentially a show stopper for professional audio work and maybe
some kinds of video streaming. Trying to completely escape disk IO in those
uses is very limiting.

A comparable solution using LVM and/or mdraid with ext4 on top has much better
latency behavior.

Sorry for no benches for you, but feel free to run a quick check using
latencytop and ftrace. Phoronix has some performance comparisons if you want
them.

~~~
semi-extrinsic
> Essentially a show stopper for professional audio work (...). Trying to
> completely escape disk IO in those uses is very limiting.

Could you expand on that?

I mean, an hour of mono uncompressed 192 kHz/24bit audio is almost exactly 2
GB. Compared to professional audio equipment, 128 GB of RAM isn't very
expensive ( < $2000), and that would let you keep 64 one-hour maximum-def
tracks in memory. Why do you need to read from the disk with any frequency?

------
shmerl
OpenZFS, not ZFS. Those are different beasts now.

~~~
ZenoArrow
Worth pointing out the licence is the same for both.

~~~
shmerl
Not at all. ZFS is not even FOSS anymore.

~~~
ZenoArrow
Fair enough.

------
heavenlyhash
This is great news. Among other incentives, ZFS has some truly excellent
features for improving reliability. ZFS's built-in checksums, for example, can
result in _much_ happier stories during the onset of disk failures: where a
RAID array can quietly return _incorrect_ sector contents without noticing,
and be unable to correctly differentiate between the correct and not-so-
correct sectors in the event of disk loss followed by disagreements discovered
during rebuilds, ZFS simply does the right thing by making checks during
normal operations, and uses the same checks to confidently do the right thing
during recovery. And snapshotting. Oh, snapshotting.

On the other hand, I've always wished we could get a modern re-take on ZFS. As
anyone who's tried it will tell you: dedup in ZFS essentially doesn't work.
ZFS, internally, is not built on content-addressable storage (or, it is, but
since splitting of large files into blocks doesn't take any special actions to
make similar blocks align perfectly, it doesn't have anywhere near the punch
that it should). As a result, dedup operations that should be constant-time
and zero memory overhead... aren't. Amazing though ZFS is, we've learned a lot
about designing distributed and CAS storage since that groundwork was laid in
ZFS. A new system that gets this right at heart would be monumental.

Transporting snapshots (e.g. to other systems for backups... or to "resume"
them (think pairing with CRIU containers)) could similarly be _so_ much more
powerful if only ZFS (or subsequent systems) can get content-addressable right
on the same level that e.g. git does. `zfs send` can transport snapshots
across the network to other storage pools -- amazing, right? It even has an
incremental mode -- magic! In theory, this should be just like `git push` and
`git fetch`: I should even be able to have, say n=3 machines, and have them
all push snapshots of their filesystems to each other, and it should all
dedup, right? And yet... as far as I can tell [1], the entire system is a
footgun. Many operations break the ability receive incremental updates; if you
thought you could make things topology agnostic... Mmm, may the force be with
you.

[1]
[https://gist.github.com/heavenlyhash/109b0b18df65579b498b](https://gist.github.com/heavenlyhash/109b0b18df65579b498b)
\-- These were my research notes on what kind of snapshot operations work, how
they transport, etc. If you try to build anything using zfs send/recv, you may
find these useful... and if anyone can find a shuffle of these commands with
better outcomes I'd love to hear about it of course :)

~~~
bryanlarsen
Note that ZFS isn't magic. Even with ZFS's checksums on read, you should still
be doing regular scrubs, just like you should be with LVM or btrfs. And once
you have regular scrubs, checksums on read don't really add much.

~~~
heavenlyhash
Agreed, turning on more safety features will always make you... safer :)

But it's worth noting that I've debugged corruptions in prod systems where:

\- corrupted data was read from disk -- a bit flip, with no error code at the
time -- by an application

\- the application operated on it

\- and the application then wrote the result -- still carrying the bit flip --
back to a new file on disk.

Ouch. The bitflip is now baked in and even looks like a legit block as far as
the disk is concerned. The disk failed not long after, of course -- SMART
status caught up, etc. But that was days later.

Checksums on read address this. I never want to run a system without them
again.

~~~
bryanlarsen
I don't understand. If the bit was flipped before or during read the scrub
would catch it. If it was flipped by the application then no file system can
help. How does read checksums help you?

------
weitzj
And what about machines without ECC RAM? I thought this is the idea for using
ZFS in the first place. Or is the ECC "requirement" only important for raidz?

~~~
Filligree
Without ECC RAM, you're far more likely to get uncorrectable / unnoticeable
corruption.

This is not unique to ZFS, and it doesn't make ZFS worse than other
filesystems. But since the reason you'd use ZFS is often to avoid any
corruption, it's tradition to advise the use of ECC.

~~~
tobias3
No it is worse with ZFS because it doesn't have a fsck tool. If you have a bit
flip in the ZFS metadata you have to export and re-import your whole pool to
get it to a writable state again.

~~~
pmarreck
Resilvering (which is basically a global data verify, similar to fsck) will
fix bits flipped the wrong way via error correction, assuming you've set ZFS
up that way. Are you saying this doesn't apply to the metadata?

~~~
ryao
A simple scrub will repair blocks that have checksum failures, but there's is
no guarantee that the checksum was calculated before the bit flipped if it
occurred in a buffer being written.

------
e40
Anyone know of Red Hat has plans to include it in default installations?

~~~
cjbprime
Opinion: Red Hat won't because it has better lawyers.

~~~
rwmj
... and billions more dollars at stake.

------
wstrange
Oracle could put this whole legal wrangling to rest by changing the license to
GPL.

I am dreaming I suppose.

~~~
protomyth
They don't own the copyright on the patches, so it wouldn't exactly be
effective.

~~~
dmm
They might not own the openzfs patches but they definitely own all of the zfs
code. They had all contributors sign CLAs to reassign copyrights. That's how
they were able to end opensolaris.

~~~
protomyth
I'm pretty sure ZFS as last released is not as valuable without the OpenZFS
contributions.

------
dekhn
I'm a conservative user, so I don't change my filesystem until my preferred
distribution (Ubuntu) supports installing an FS to the root with a provided,
supported kernel module. This is a huge deal for me; I will probably install a
new FS on my main file server and move from ext4 to zfs.

~~~
xenophonf
In case you ever feel less conservative, here's how I set up my MacBook Pro to
dual boot Mac OS X/FileVault and Ubuntu/ZFS/LUKS:

[https://gist.github.com/xenophonf/2e2d1a1550b0fb8dae98](https://gist.github.com/xenophonf/2e2d1a1550b0fb8dae98)

------
acd
ZFS is great, happy its included in Ubuntu 16.04 by default.

------
obelisk_
Anyone know if this applies for Lubuntu as well? I use Lubuntu on my rencently
bought new desktop for the default of LXDE. I intend to upgrade my desktop on
which I put Lubuntu 15.10 (I bought a computer without OS so as not to pay the
Windows tax) to Lubuntu 16.04 because I understand it'll be based on LXQt, the
successor to LXDE (and it's not a matter of newer is better -- I am a fan of
Qt) and also because I think Lubuntu 16.04 will be an LTS release and I've
been very happy with the stability of Lubuntu 14.04 LTS which my previous main
computer was and is running.

~~~
jcastro
Yes, all Ubuntu derivatives use the same repository so you should be set.

~~~
obelisk_
Thanks :)

------
jgworks
Does this include a fix for this bug?

[https://github.com/zfsonlinux/zfs/issues/1548](https://github.com/zfsonlinux/zfs/issues/1548)

~~~
faragon
Check if this works for your case:

echo > myfile

P.S. not the "solution", but may help in case you fill the disk by accident,
and need to make some room before a remount cycle. P.S.2. this could help also
when your disk is already 100% full, without enough space for deleting files
-not enough space for new inodes- (I tested that case on ZFS NAS with no left
space at all, and worked).

------
ansible
Huh, well that's interesting news.

I'm currently setting up a couple servers using LXC with btrfs.

I ending up choosing LXC (as opposed to LXD, docker, rkt, etc.) because I
wanted something relatively straight-forward. I just wanted some containers I
could create, log in to and configure.

If this was a bigger deployment, I'd take the time to use docker or something
else. But for now, just being able to get going quickly is nice. For backup /
failover, I can btrfs send / receive the containers to another host and start
them there.

~~~
stubish
Yeah, thats all fine and good. Nothing to announce with lxc/lxd + btrfs
because it already works fine :) I do like the wizard to easily setup the zfs
backend, rather than you needing to manually replace /var/lib/lxc with a btrfs
partition or however you are doing it.

I've been using lxc + btrfs daily for quite a while, setting up and tearing
down hundreds of containers on a busy day. I stopped using lxc snapshots after
I had a btfs subvolume that would crash the system when I mounted it. After
that, no problems.

~~~
ansible
_I stopped using lxc snapshots after I had a btfs subvolume that would crash
the system when I mounted it. After that, no problems._

That's unfortunate. What operating system version were you using at the time?

I've actually switched to using Ubuntu 15.10 for the container hosts so that I
can get a more recent version of the btrfs tools. The intention is to upgrade
to 16.04 as soon as is reasonable, and leave them there for a long time.

------
wmf
This post only mentions LXD. Will ZFS be used for Docker?

~~~
awalton
Docker switched to Alpine Linux by default, so there's that hurdle... Not to
mention the giant legal question mark of loading CDDL code into a GPLv2
kernel.

~~~
mverwijs
One of the ideas of containers / virtualization is that the host operating
system (Ubuntu, in this case) has as little to do as possible with the VM /
container (Alpine, in this case).

Running an Alpine container using Docker on an Ubuntu host will work just
fine.

------
sengork
I wonder how this would work under shared storage model for container clusters
(eg. Mesos). NFS does not cut it for all use cases (eg. DBs).

------
satbyy
For those who missed it, Debian Project Leader Neil McGovern gives details [1]
about how licensing issues were resolved so that ZFS can be in Debian now. It
is distributed as source-only dkms module.

[1]: [http://blog.halon.org.uk/2016/01/on-zfs-in-
debian/](http://blog.halon.org.uk/2016/01/on-zfs-in-debian/)

~~~
cjbprime
Though that's not relevant to this news -- Ubuntu isn't using DKMS, they put
it directly into their kernel tree, violating GPLv2.

~~~
mrb
No, Ubuntu may have put the ZFS source in the kernel tree, but they still ship
it to endusers as a separate kernel module and separate Ubuntu package (edit:
the separate package is "zfsutils-linux" for the userspace code)

AFAIK to violate the GPL they would have to ship ZFS compiled code in the
kernel image, but this is not what they are doing.

~~~
mjg59
Whether something is a derivative work or not is incredibly unlikely to be
determined by whether it's distributed as a module or linked in.

~~~
ryao
If I take a chapter of a textbook, modify it to be a standalone volume in a
collection of books and start distributing it, I am distributing a derived
work of the original book, not a derived work of the collection of books. The
latter constitutes an aggregation and unless there is some license
(superseding doctrine of first sale in the case of books) that prevents it
from being redistributed with such things, it is perfectly okay to do that.

Similarly, the original code was taken from OpenSolaris and was adapted for
Linux. No matter how we change it, it is a derived work of Solaris.
Furthermore, it is distributed as part of a mere aggregation, which is okay
with OSS under the OSD and also okay with the GPL under the GPL FAQ. The only
time you can claim a combined work is formed is when the module is loaded into
a running kernel, but the GPL does not restrict non-distribution and the
kernel with the module loaded into it is not being distributed.

You can argue that GPL advocates did not intend to support a license that
allows any of this. However, I expect that you would have trouble finding an
attorney that will interpret what the copyright holder thought the terms said
to supersede the legal meaning of the terms unless explicitly stated.

If you make a license for the kernel that does not allow derived works of
other platforms' software to be distributed as ports, you would violate #9 of
the OSD and could not call it an open source license:

[https://opensource.org/osd-annotated](https://opensource.org/osd-annotated)

~~~
mjg59
If you take the plot from an episode of Star Trek and modify it such that it
fits into the Dr Who storyline, you've created a work that's derivative of
both Star Trek and Dr Who. Similarly, if you take code from Solaris and modify
it such that it tightly integrates with Linux, you've created a work that's
derivative of both Solaris and Linux. Since ZFS can only be distributed under
the CDDL and since GPLv2 requires all derived works to be distributed under
the GPL, you can't satisfy the license.

~~~
ryao
> If you take the plot from an episode of Star Trek and modify it such that it
> fits into the Dr Who storyline, you've created a work that's derivative of
> both Star Trek and Dr Who. Similarly, if you take code from Solaris and
> modify it such that it tightly integrates with Linux, you've created a work
> that's derivative of both Solaris and Linux. Since ZFS can only be
> distributed under the CDDL and since GPLv2 requires all derived works to be
> distributed under the GPL, you can't satisfy the license.

That is analogous to writing a new piece of software intended to be similar to
an existing piece of software rather than a port of software under license.
Examples of the former include the Linux kernel (meant to be similar to UNIX
SVR4) and the wine project (meant to be similar to Windows). If that argument
is valid:

1\. Oracle is in an excellent position to sue every Linux user not using
Oracle Linux, because they own rights to UNIX SVR4, which they inherited from
Sun.

2\. Microsoft is in an excellent position to sue wine users.

3\. James Cameron and 20th century Fox would also be in trouble with Disney
for Avatar's similarities to Pocahontas.

4\. Probably plenty of other bad things.

However, this argument does not apply to ZoL because the code originated in
OpenSolaris and is under license and exists as a discrete module, rather than
a whole program.

So far, the only thing that you have concretely stated is that you met some
attorneys who were unwilling to make a decision on legality. You are not an
attorney (unless you have obtained a bar number since I last asked) and I have
yet to hear that anyone with a bar number that agrees with you.

If you want to prohibit people from using software you write with things that
you consider to be derivatives when the law does not recognize them as such,
you need a license that makes that explicit. Such a license could not be
called an open source license under clause #9 of the open source definition:

[https://opensource.org/osd-annotated](https://opensource.org/osd-annotated)

Consequently, the GPL is definitely the wrong license for that.

~~~
mjg59
> That is analogous to writing a new piece of software intended to be similar
> to an existing piece of software rather than a port of software under
> license.

I take ZFS from Solaris. I rewrite it to work with Linux. In which sense is
this not equivalent to my analogy? The examples you're giving are not
equivalent, because in each case the work was written without deriving from
the other copyrighted work.

> However, this argument does not apply to ZoL because the code originated in
> OpenSolaris and is under license and exists as a discrete module, rather
> than a whole program.

That's an entirely arbitrary distinction.

> So far, the only thing that you have concretely stated is that you met some
> attorneys who were unwilling to make a decision on legality.

No, I said that lawyers had told me that ZoL was an infringing work but that
we wouldn't know for sure unless a court decided on it:
[http://www.phoronix.com/forums/forum/software/general-
linux-...](http://www.phoronix.com/forums/forum/software/general-linux-open-
source/827623-ubuntu-is-planning-to-make-the-zfs-file-system-a-standard-
offering?p=827785#post827785)

> If you want to prohibit people from using software you write with things
> that you consider to be derivatives when the law does not recognize them as
> such

Nobody wants that.

~~~
ryao
> I take ZFS from Solaris. I rewrite it to work with Linux. In which sense is
> this not equivalent to my analogy? The examples you're giving are not
> equivalent, because in each case the work was written without deriving from
> the other copyrighted work.

I take it that you never actually read the ZFSOnLinux source code.

It is not really rewritten. There is a compatibility layer in place to prevent
the need to rewrite much of the code and a very small percentage of the
original kernel code actually changed to support Linux, but what did change
was meant to use for interfaces that are provided by the kernel to allow
proprietary modules to be ported, which suggests any license is fine.

However, the claim that writing a brand new TV show script inspired by another
forms a derivative work is to claim that writing things from scratch forms a
derivative work.

> That's an entirely arbitrary distinction.

It is the distinction lawyers are making.

> No, I said that lawyers had told me that ZoL was an infringing work but that
> we wouldn't know for sure unless a court decided on it:
> [http://www.phoronix.com/forums/forum/software/general-
> linux-...](http://www.phoronix.com/forums/forum/software/general-linux-..).

Do you have bar numbers of these lawyers? Is there any reason to think that
they were thinking that zfs.ko somehow used GPL exported symbols or some other
thing that is not actually true that does not involve taking your word for it?
I did have one person going to law school tell me that it was a derivative
work because of that. He did not think he could claim otherwise after an
explanation that the code does not do that.

Given that your legal views are so incredibly divorced from those of actual
lawyers with whom I have talked, I am not inclined to believe you when you say
that they had no misunderstanding, especially when it seems that you have
never actually read the code to be able to be sure of that.

> Nobody wants that.

Your claims are inconsistent with that.

~~~
mjg59
> It is not really rewritten.

It has several direct calls into Linux functionality that don't go via SPL,
but it's also unclear that simply adding an abstraction layer is a meaningful
mechanism to avoid derivation.

> what did change was meant to use for interfaces that are provided by the
> kernel to allow proprietary modules to be ported

There are no such interfaces in Linux.

> the claim that writing a brand new TV show script inspired by another forms
> a derivative work is to claim that writing things from scratch forms a
> derivative work.

I didn't make that claim. The analogy in question involves taking an existing
work and modifying it such that it includes components of another work.

> It is the distinction lawyers are making.

It's the distinction a lawyer that you've spoken to is making.

> Do you have bar numbers of these lawyers?

Yes.

> Is there any reason to think that they were thinking that zfs.ko somehow
> used GPL exported symbols or some other thing that is not actually true that
> does not involve taking your word for it?

No.

> Your claims are inconsistent with that.

My claim is that I have reason to believe that, under copyright law, ZoL is a
derivative work of Linux and as such is subject to the terms of the GPL. If
the final legal determination is that it's not a derivative work then the GPL
is irrelevant.

------
jpgvm
Shame how most of the conversation devolved into licensing rubbish. Almost
none of us are qualified to speak on that, leave it to the lawyers - which I
assure you Canonical did too.

With that out of the way, ZFS is by far and away the best filesystem for
container workloads. Hopefully we will get deeper quota and I/O throttling
support soon.

I have been using ZoL in production for many years now thanks mostly to the
work of Brian Behelendorf and Richard Yao. So if you find yourselves here
thanks for all the work you have put into making ZoL awesome.

~~~
2trill2spill
> Shame how most of the conversation devolved into licensing rubbish. Almost
> none of us are qualified to speak on that, leave it to the lawyers - which I
> assure you Canonical did too.

This a million times, It will be nice to have the illumos community, the
FreeBSD community and now the Linux community contributing to one piece of
core software. It's especially amazing considering most open source operating
system projects don't share major kernel subsystems.

------
anilgulecha
This can be game-changing for the NAS/SAN industry.

I'm surprised their lawyers gave an OK, where FSF, SFLC and friends have given
a thumbs down. If their interpretation is good, suddenly the large AIX/Solaris
dominated storage boxes open up to a _LOT_ of ubuntu-based/ubuntu-derived
competition.

Exciting times..

~~~
anilgulecha
BTW, if ZFS is used for the root filesystem, say hello to insanely simple OS
upgrades and roll-backs :)

~~~
Filligree
You could also use NixOS, which is even better at this. :)

And also has native support for ZFS, btw.

~~~
JdeBP
Or indeed PC-BSD. (-:

PC-BSD's installer understands ZFS. It creates an all-ZFS system, including
the root volume. The boot manager uses ZFS for "boot environments".

* [http://download.pcbsd.org/iso/10.2-RELEASE/amd64/docs/html/i...](http://download.pcbsd.org/iso/10.2-RELEASE/amd64/docs/html/install.html#zfs-layout)

* [https://www.ixsystems.com/whats-new/2013/10/31/the-revamped-...](https://www.ixsystems.com/whats-new/2013/10/31/the-revamped-life-preserver/)

* [https://blog.pcbsd.org/2013/06/pc-bsd-status-update/](https://blog.pcbsd.org/2013/06/pc-bsd-status-update/)

------
robbyt
How is this possible, legally? Based on my basic understanding of the ZFS
license, it's not possible to legally distribute ZFS and GPL code (linux
kernel) together.

~~~
henesy
It says that it is OpenZFS specifically.

~~~
davexunit
Which is still CDDL licensed and thus GPLv2 incompatible.

Here's the copyright file from the relevant commit to the Xenial kernel tree:

[http://kernel.ubuntu.com/git/ubuntu/ubuntu-
xenial.git/diff/z...](http://kernel.ubuntu.com/git/ubuntu/ubuntu-
xenial.git/diff/zfs/COPYRIGHT?id=b65ce14d3445e8fe69b5d59aaaf2699319714179)

    
    
        The majority of the code in the ZFS on Linux port comes from
        OpenSolaris which has been released under the terms of the CDDL
        open source license.  This includes the core ZFS code, libavl,
        libnvpair, libefi, libunicode, and libutil.

------
davexunit
Sorry to be a downer folks, but Canonical is violating GPLv2 by merging ZFS
into their kernel tree. The CDDL and GPL are _incompatible_.

[https://twitter.com/mjg59/status/700073945064611841](https://twitter.com/mjg59/status/700073945064611841)

~~~
lobster_johnson
Comment by Richard Yao, Gentoo dev and ZFS On Linux contributor [1]:

    
    
        ... there is no legal issue preventing the sources
        from being combined because neither the CDDL nor
        the GPL place restrictions on aggregations of
        source code, which is what putting ZFS into the
        same tree as Linux would be. Binary modules built
        from such a tree could be distributed with the
        kernel's GPL modules under what the GPL considers
        to be an aggregate. These concepts have passed
        legal review by many parties.
    

[1] [http://www.phoronix.com/forums/forum/software/general-
linux-...](http://www.phoronix.com/forums/forum/software/general-linux-open-
source/827623-ubuntu-is-planning-to-make-the-zfs-file-system-a-standard-
offering?p=827628#post827628)

~~~
jamiesonbecker
"However, there is nothing in either license that prevents distributing it in
the form of a binary module or in the form of source code."

[http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue](http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue)

Whether _any_ loadable kernel module can be licensed under a different license
than the GPL is still up for debate (even Linus says "sometimes"):

[http://www.tldp.org/HOWTO/Module-
HOWTO/copyright.html](http://www.tldp.org/HOWTO/Module-HOWTO/copyright.html)

(Of course, Ubuntu et al have been doing this for years e.g. with Nvidia or
HBA binary blob drivers, so it does appear to be a mostly-settled issue..)

~~~
ryao
This was settled with the Andrew filesystem, which is a matter that I do not
believe Linus considers up for debate. ZFS being a port from another system
just like the Andrew filesystem is naturally in the same boat.

------
mikegerwitz
There's a number of licensing comments here. Linking with CDDL modules is
explicitly mentioned by the FSF:

[https://www.gnu.org/licenses/license-
list.html#CDDL](https://www.gnu.org/licenses/license-list.html#CDDL)

I'm not going to comment further on any implications of Debian/Ubuntu's
decisions, since IANAL.

~~~
AstralStorm
In case of Linux specifically, Linus voted on the practice to provide so-
called GPL entrypoints and symbols. (MODULE_SYMBOL vs MODULE_SYMBOL_GPL) This
is explicitly allowed in GPL as a clause about relaxing provisions. (exception
clauses such as one in Glibc or GCC)

If ZFS kernel module uses only those permitted symbols, it's likely fine from
legal standpoint. If not, there is a problem.

~~~
ryao
The current ZFSOnLinux codebase does not touch any GPL exported symbols when
built as a loadable kernel module.

------
dkarapetyan
Ubuntu is always slightly behind and slightly ahead of the curve. Why are they
digging their heels in with lxd when docker and rkt make so much more sense?

