…Being "upstream" is more a matter of degree than an absolute ordering. In the past, most ZFS features have originated in illumos, and those changes have been ported to the other platforms (notably FreeBSD and ZoL). Looking forward, we see more features originating in ZoL. The OpenZFS community (including FreeBSD) is working to put in place technical processes to ensure that new ZFS features are available on all platforms. The original post in this thread was a proposal for how to do that.
…The way that things have evolved over the last few years means it would be much more difficult to just import changes from ZoL. ZoL was originally far behind the OpenZFS repo, but as they were catching up, they were also fixing bugs and adding features. So there is not really a common point in the source we could start from to import ZoL commit by commit.… The biggest thing to remember is that this is still OpenZFS, and still run by the same developers as it has been. We are just commonizing on the repo that has the most features integrated into it.
So, basically, Delphix was a (the only?) commercial user of Illumos and now they're abandoning it for Linux, which means Illumos will starve for manpower going forward?
It also sounds like FreeBSD also lacks the resources to maintain an implementation of something like ZFS, which is ironic since not too long ago it seemed like one of the marquee features they had that Linux didn't.
There's no contradiction here. FreeBSD was never the primary contributor to (Open)ZFS (it was an Illumos thing), but we had the resources to make it a first class citizen on FreeBSD at a time when Linux wasn't doing that (mostly for licensing reasons).
It seems like the Linux world has gotten over the licensing problem one way or another, and at the end of the day Linux has something like 100x-1000x more developers to bring to bear at the problem. So between that, and illumos' imminent death, ZoL is the active upstream. FreeBSD is happy to benefit from the ZFS improvements in ZoL, especially given the collaborative spirit we've seen from the ZoL team so far. If anything, kudos to the ZoL folks for the huge amount of effort they've put in over the last handful of years.
Rumours of our death are, of course, greatly exaggerated.
If I understand correctly, the developers were saying there are specific parts of Linux where anything using them is considered a derivative work, and therefore even distributing source would be a violation of GPL. An instance of this is the crypto layer that was added in recently with the native dataset encryption. SPL actually reimplements this logic because the ZoL maintainers claim they can't use the Linux crypto subsystem without violating GPL.
I believe that comment by Matt/Kip reflects more of a personal opinion about how things should be (he's a stalwart FreeBSD contributor), rather than how they actually are in practice.
They continue to port code to Illumos (i.e. KVM, continued work on Pluribus port of bhyve, etc.).
SmartOS is what their data centre orchestration solution gets built on-top of.
You can see their changelog for SmartOS they are quite active:
Development of SmartOS and Triton Datacentre occur under their joyent organisation on GitHub.
On the flip side, I'm glad I saw this when I did. I really don't want to have to know all about ZFS tooling, and hoping that they get the UI/UX in place for a clean NAS experience, and may just buy a prebuilt from iX later next year.
guess I have some additionally learning to do :-)
That can still be somewhat true due to the licensing mess. ZoL will never be in the tree for linux. The best you can hope for is the Canonical solution which at least ensures a supported version for the distro kernel.
My theory on how this might happen is an independent implementation of BtrFS that extends from ZFS code, sharing it whenever possible.
Should such a thing emerge and surpass current BtrFS, what else could Oracle do but throw in the towel?
From my understanding both Linus and ZoL team prefer to be separate.
But it's unlikely they'll do it. In recent years they've relicensed-under-GPL some things (like the DTrace userspace utilities) but I have doubts they'd ever do it for something like ZFS.
I sincerely doubt that. If Oracle would relicense/dual license ZFS as GPLv2 compatible then I feel convinced that the ZoL contributors would happily do so as well.
I can't think of a single reason why they would not want to be in mainline Linux, and I have seen nothing indicating that the Linux kernel devs would be anything but happy to have it 'mainlined' as it would only make Linux better.
Of course this all hinges on Oracle, which makes it extremely unlikely in my opinion.
From my personal look, there could be significant mess due to attempts to "remove duplication" or similar moves to make the code more linux-like. While ZFS is not the rampant layering violation some people accused it of, there's significant amount of code there (and funnily enough, there's actually overlap with some less-known linux features)
Yes, all the parties that contributed code would have to agree, not just Oracle. However, I doubt anyone would object relicensing ZFS to some license that both FreeBSD and Linux would find acceptable.
A bit of a leap - while your theory could be correct, alternatively, if the filesystem is complicated and can be ifdefed to work for many OS's, it makes sense to combine dev efforts into one codebase rather than having multiple independently maintained forks..
not sure which is the case here.. but in any event.
... it's that Solaris is effectively finished as an OS. It's not even its own upstream anymore for arguably the greatest modern contribution to any OS.
This is tragic, and needless. Oracle effectively shot itself in its own foot with the pointless and deliberate choice of licenses, the on-again, off-again open sourcing of Solaris, and the complete lack of commitment. Solaris had a real shot of remaining a fine alternative UNIX operating system, at least for the server side of things, but now this shot is gone.
They killed Solaris like they killed Java.
They bought Java, because they were using it everywhere in their own product offering, and didn't want it under someone else's control (esp. if IBM would have been Sun's buyer rather than them).
Anything else which came with the purchase (Solaris, OpenOffice, VirtualBox, MySQL, the hardware business, 30K+ employees) amounted to, in their eyes, literally strings attached, that they had to pretend to care about for a while.
Even Java itself is not getting treated too well, but that was never the point - the point is not to lose control, and to monetize what they can.
The free version is basically useless: "Stay connected for 2 hours while inactive (permanent for first 7 days)" because of this.
(Debian also has a policy that you may not publicly log its IRC channels, and I also think it's wrong.)
(I'm freenode staff but currently mostly on hiatus due to lack of tuits)
Why would that be? Lack of interest on sending them upstream or them being sent but never merged back?
 is the monthly meeting notes, and while I haven't had a need to visit one and confirm, they certainly appear to be open to interested parties, not just an invite-only group.
(I recently proposed something controversial  in an attempt to try and get predictable consistent behavior on a certain thing across all platforms, which is when I learned about this from someone telling me they discussed it there.)
ZoL also maintains their own list of upstream (e.g. openzfs) commits and what their status is of landing in ZoL. 
 - https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HB...
 - https://openzfs.topicbox.com/groups/developer/T04955dd2e8aa4...
 - http://build.zfsonlinux.org/openzfs-tracking.html
I suppose you could argue for having a single, explicitly OS-independent "upstream repository" rather than treating either Illumos or ZoL as upstream? I doubt there are enough developers who would want to work on ZFS generically rather than ZFS in the context of a particular OS. ZoL maintains a clear separation from Linux proper for licensing reasons so I'd think they're "independent enough" to serve as upstream for non-Linux versions.
That would neatly solve the issue of ZoL patches never making it back to Illumos.
The heartening moment, is that linux and BSD should share code when they can, because its sensiblet to avoid pointless fork. Pointful fork, (and I exclude licence terms mostly) is something we have to deal with. Pointless forking is just a royal pain.
An outcome of this should be that I get more guarantee a ZVOL export/import cycle across Linux/BSD (which btw I have now done about 10 times on 10TB ZFS filesystems) will work with no awkward binary consequence to flag/version mismatch which came about because of two codebases: version mismatch because one was "old" and one is "new" is a different thing.
Could you explain? FreeBSD was not the reference implementation of ZFS, illumos was. ZoL is still tied up in the OpenZFS project (as are illumos and FreeBSD), this is more a matter of rebasing onto the subproject where more development is occurring. As it was, FreeBSD's implementation was already using a Solaris/illumos compatibility shim, and it sounds like switching isn't that much work.
What does the solaris compatibility shim do in the end, which couldn't be done direct in the codebase? I don't like shims, it feels like the linux compatibility ABI moment where something gets added and the shim drifts, and then you can't do eg DRI any more because "they changed how it worked and we didn't catch up"
But, thanks for the cluestick hit. If this is just flavour of ice-cream from the same truck, I'm less worried.
On a separate note, illumos is a really cool project. I hope this transition (and apparent reduced activity in their development on ZFS) is not a symptom of the project losing momentum. My first experiences with a computer were on SunOS and early Solaris, and illumos is the the [open source] heir to that throne.
"[§]3.1 … Any Covered Software that You distribute or otherwise make available in Executable form must also be made available in Source Code form and that Source Code form must be distributed only under the terms of this License. …
"[§] 3.4 … You may not offer or impose any terms on any Covered Software in Source Code form that alters or restricts the applicable version of this License."
CDDL is per-file license, does not automatically extend itself to the whole product. But it also has wording about patents et al, which puts it beyond direct inclusion in kernel just like MPL.
Out of tree with one degree of separation (Solaris Porting Layer) is based on the precedent of AFS implementation for linux, where the argument was that the code is not derivative of Linux kernel code, as in no way was VFS unique to linux or any other crucial part involved.
Well, MPLv1 was incompatible with the GPL. MPLv2 has an explicit clause to make it compatible (namely to allow relicensing within limits). I would imagine the lack of "relicensing within limits" would be the key problem with CDDL and GPL -- since CDDL was based on MPLv1.
> Out of tree with one degree of separation (Solaris Porting Layer) is based on the precedent of AFS implementation for linux, where the argument was that the code is not derivative of Linux kernel code, as in no way was VFS unique to linux or any other crucial part involved.
Right, especially since the ZFS source code was developed completely separately to Linux and the SPL is licensed GPL+CDDL. This is the same thing that NVIDIA does with their drivers (they even build their driver binaries on FreeBSD to show that it "couldn't possibly" be a derived work of Linux and have a GPL shim).
Though FSF doesn't seem to support it.
And a few HN posts regarding the Ubuntu's announcement.
Having spoken to some Canonical folks involved in this discussion, basically it boils down to the fact that the SPL (Solaris Porting Layer, which converts Linux VFS APIs to Solaris ones for ZFS) is GPL licensed and that ZFS was developed entirely separately from Linux (and Torvalds has said he agrees with the latter point). It also helps that ZFS is free software.
Honestly I think their strategy is that it's very unlikely they'll get sued (not to mention the past 5-8 years of OpenZFS development was done without an Oracle CLA). IMHO the FSF is primarily against it because they think it might water down the GPL's effects.
Yes, but it's shipped as a separate kernel module (as in separate file) rather than built into the Linux kernel. As for the legality of this, it can really only be determined in court, which hopefully won't happen since I can't think of anyone who would want to prevent Linux from using ZFS now that Sun is gone.
It would be great if it could eventually be part of Linux mainline which would then allow it to easily be used on boot volumes as well. That would require dual/relicensing though, since Linus Torvalds have said that CDDL code will not be merged with mainline (most likey a result of consultation with Linux Foundation lawyers).
Oracle? I doubt they would, but I mean, it's Oracle.
The problem we have is Oracle also don't seem at all interested in maximizing ZFS either. So they'd quite happily see both die.
One can be aggressively litigating using ones IP while also letting the technology behind that IP die.
Considering btrfs isn't really going anywhere why would they not rather take the effort to relicense zfs than put any effort on btrfs if they have no bad intention with zfs?
Fixing zfs' situation surely can buy some mindshare for Oracle.
Sounds like edge cases for performance and stability could show up?
Because of GPL incompatibility with the CDDL, Linux cannot do this. A compliant installer must put root on something else, usually ext4 or xfs. A Linux installer iso putting root in a zfs dataset opens a terrible legal door.
% zfs list rpool/ROOT/default
NAME USED AVAIL REFER MOUNTPOINT
rpool/ROOT/default 14.8G 60.1G 11.9G /
The GPL is not an EULA. It's a distribution licence. There is absolutely no problem in using ZFS on your system, and there's no legal restriction on what an installer may or may not do.
If said installer downloaded and compiled the code (nvidia-drivers style), you'd not have an issue. Only once the code is combined does it "violate" the license. You can't distribute something in violation of GPL2, but you can keep it for yourself. As long as the installer doesn't distribute offending binaries, you're fine.
Encryption support should be on 0.8 (next) release:
1: Issue #674
Eh? The community port and support already happened, years ago. I made the move myself from ZEVO once Spotlight support got mainlined but even that was a while ago and it already had a ton of great features before that. It is a small scale effort compared to Linux or FreeBSD of course, but it has kept up pretty darn well and is pretty solidly developed at this point. I push it pretty hard on my Macs and it's been pretty reliable, though there are definitely edge case oddities that can be found. You can also see the smaller scale of the effort in areas like the wiki not being fully up to date, or that booting off of it still is rough. Even so it's pretty solid at this point and hits all the core features including encryption.
Long term the big risk of course is that Apple will shut off all user low level access to their drives via T-series chips in Macs, since they don't actually give hardware owners the ability to use their own keys there. Ideally this would be illegal but I doubt any movement will be made on that in the US at least in the near future. Even so given that Apple up until recent has still been selling brand new Macs without it macOS itself will likely support running on those systems for a long time to come (Apple still officially supports running the existing version even on 8 year old MPs, granted with compromises but those had pretty fundamental divergences from modern firmware too). O3X is still pretty nice in the mean time.
But what if Apple simply built APFS into one of the next generations of T-series chips so it can physically deny block-level access to software? Not going to happen: Apple needs APFS to work not only with built-in storage but also with external USB drives, RAM disks, DMGs, sparse bundles, and so on. This means APFS needs to remain in kernel space, and kernel code will continue to have block-level access, and so will O3X and all of its features.
There is no cause for pride in newly-acquired skills that rest on the backs of giants.
Considering one of thse comes default the base system and the installer can install to it, has jail(8) integration, etc etc
and one of them is an addon which will never be merged to the main kernel (which is only a kernel), and because of this, will likely never be directly supported by the main linux distribution vendors, I'd say the jury is still out on this piece..
lack of base system integration for linuces also leaves the 'maintained' somewhat up in the air too.. if I'm running $linux_flavor and Zfs crashes, and the ZoL devs blame this particular distros particular configuration/build options, toolchain, etc
where do I go for a fix?
migrate to another linux with totally different package mgmt, etc?
but yes, it looks like the zfs core code itself is more actively updated for that portion directly.
Zfs is front and center for lxd on Ubuntu - and about as "supported" as can be?
I just can't wait for crypto support to really land in open zfs - would make it even easier to use for external drives (now encryption needs separate support, geli on bsd, dm-crypt/luks on Linux). And using zfs as a volume manager would make things in general easier and less complicated.
To the distro of course. How is this even a point of contention?
Kind of ironic given DragonflyBSD roots in FreeBSD
Why? Existing FreeBSD ZFS users can and in all probability will continue to use it. Why does a less mature FS on a less widely used BSD derivative suddenly look appealing to you?