
Native encryption added to ZFS on Linux - turrini
https://github.com/zfsonlinux/zfs/commit/0b04990a5de594659d2cf20458965277dd6efeb1
======
jlgaddis
Slightly off-topic but if anyone has any resources on performing a clean
(preferably, Ubuntu or Arch) Linux "root on ZFS" installation, please share.

I followed the instructions for Ubuntu 16.04 on the github.com/zfsonlinux wiki
[0] a while back and (encountered a few little issues along the way but) got
it working, although I experienced some MAJOR performance problems so
_something_ wasn't quite right (exact same hardware is blazing fast when
running FreeBSD). I can't imagine it was just "how things are" with regard to
the current state of ZFS on Linux (or Ubuntu specifically) -- it was like
someone hit the laptop's "pause" button a couple of times per minute.

[0]: [https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-
on-...](https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS)

~~~
imsofuture
This sounds crazy, and is, but following those instructions verbatim with a
16.04 install I managed to _brick_ a new laptop. Yeah...

~~~
iso-8859-1
Define "brick". You flashed the wrong BIOS?

~~~
pantalaimon
Possibly deleted the efi partition

~~~
vmp
Or "rm -Rf /" with a writable /sys/firmware/efi/efivars/ mount.

[https://github.com/systemd/systemd/issues/2402](https://github.com/systemd/systemd/issues/2402)

~~~
the_why_of_y
This problem was fixed half a year ago, surely Ubuntu 16.04 should have a
backport of the relevant commits?

[https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....](https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0389075ecfb6231818de9b0225d3a5a21a661171)

------
mrsteveman1
Related pull request with more details & discussion

[https://github.com/zfsonlinux/zfs/pull/4329](https://github.com/zfsonlinux/zfs/pull/4329)

~~~
wspeirs
tcaputi works with me at Datto. We use ZFS as our main file system for some
200+ PB of storage. It is a truly amazing file system, and this change, once
merged, will add another layer of security for our customers. Great stuff!

~~~
okket
At this scale, why ZoL and not a more 'native' ZFS with FreeBSD (or even
Illumos)? Which also perform better and have more features?

Is it the lack of drivers? Better availability of other 3rd party software for
Linux? The package manager? Or just "more brains in Linux"/"our CTO grew up
with Linux"?

~~~
abrookewood
I thought that ZoL and ZFS+FreeBSD were largely feature comparable?

~~~
justincormack
Other than orders of magnitude more testing and experience and usage on
FreeBSD. Much code is the same, but the glue may have bugs.

~~~
pantalaimon
Well Linux receives orders of magnitude more testing and usage than FreeBSD,
so I don't see what your argument is

~~~
fyolnish
His argument refers to the current state of this specific component.

~~~
pantalaimon
ZFS on Linux has been around for 3 years now and is integrated in Ubuntu 16.04
- I'd argue that by now it has received more usage than the BSD version.

~~~
mioelnir
You can argue that the earth is shaped like a torus. I can argue that the moon
is violet. That is not the point, and has nothing to do with the merit of the
argument.

Setting this aside, all usage hours are not equal. How do you compare an hour
of enduser desktop usage on Ubuntu 23.57.something to a usage hour of a
commercial NAS storage solution internally based on FreeBSD and ZFS? Do you
think the enduser performed comparable product testing cycles?

I've seen ZOL systems where the zpool claimed it was online, but contained no
vdevs. `zfs list` had no datasets, but datasets where mounted and trying to
read from them got your process stuck in-kernel. It just lost/forgot its
devices.

Up until one of the latest releases on every boot you could roll the dice by
which name your pool would import the devices. Behaving differently on
identical machines and setups. Personally, I am still not trusting that
problem to not reappear again.

ZOL is bolted on. With a large nailgun. Simple as that. At times, it feels
about as integrated as pjd's original patches distributed on the FreeBSD
mailinglists. And since this division is not technical, but legal, based on
the license choice made 30 something years ago for Linux with regards to where
the code could be exported to and what could be imported into, this situation
will not resolve itself.

------
ryao
This is not native encryption that was committed. It is just the kernel
cryptography framework required for native encryption. Native encryption comes
next.

------
makmanalp
What's the stability of ZFS on linux like these days? Anyone have any positive
/ negative experiences to share?

~~~
aeorgnoieang
I've been running two very-lightly-used sets of pools on two old desktops for
... about a year and a half now actually and everything's gone swimmingly.

Nor have I read of anything bad happening to anyone that didn't involve not
having backups.

All in all, lots of people are running it, lots of people seem to be having
almost entirely positive experiences with it, so it seems like it's very
stable.

~~~
agumonkey
Could you give machine specs ?

~~~
barrkel

        $ head -1 /proc/meminfo
        MemTotal:        4046880 kB
    
        $ grep model.name /proc/cpuinfo | head -1
        model name      : Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
    
        $ sudo zpool list
        NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
        tank  36.2T  27.2T  9.09T         -    10%    74%  1.00x  ONLINE  -
    

That's 10x 4TB WD Red drives in raidz2 configuration.

I've been running ZFS for the past 6 years, a little over 2 years on ZoL, and
4 years before that on Nexenta (Ubuntu userland with OpenSolaris kernel).

I've had 5 HDD failures over the past few years, never lost any data.

I've found ZoL to have less jitter when streaming video files in particular
over SMB than the Solaris implementation did. ZoL is tighter on memory though.

raidz2 isn't particularly fast for random I/O (since it acts like it has a
single spindle), but I don't do much random I/O. It's mostly video and local
backup cache.

I'll be thinking about expanding my current setup soon enough. I'll probably
be aiming for a raid10 setup for better random access, and of course the new
system will have more RAM, and ECC.

~~~
agumonkey
Thanks. Beefier cpu than the beefier I got here, but RAM is a great hint that
for personal needs ZFS can run on normal machines.

Interesting that you had no issues without ECC in the first place. Some ZFS
boards are pretty against trying ZFS without.

~~~
Elhana
ZFS don't really need ECC that much, just FUD.

Most likely someone got bitten by faulty RAM, got corrupted data, which
stopped after getting ECC RAM. => "ZFS wants ECC" myth, but in reality you
just need stable hardware.

At the same time, if you are running like 30-40 TB pool, you probably want
that ECC RAM, it will cost only small fraction of total storage box price and
will save you from rebuilding a pool if shit happens.

~~~
__jal
I don't know about anyone else, but when I'm seeking advice about software
from people on the internet whom I don't know personally, I tend to weight the
words of the authors of the software in question more highly than that of
people who didn't write it. There are probably exceptions to that I'm
currently forgetting.

And the authors of ZFS, from the Sun days on, have been consistently and
repeatedly saying that _if you care about your data, use ECC_.

I personally don't get why people seem so resistant to using ECC. It isn't
that much more expensive and we know these errors happen in around 8% of
DIMMS[1]. It reminds me of people I know who refuse to buy a decent power
supply and then complain about hardware failures because of shoddy power.

Personally, paying a slight premium to avoid an almost one in 10 likelihood of
silent data corruption seems like a no-brainer to me. But then, I care about
my data.

Edit: to be clear, ZFS will perform less dangerously with non-ECC ram than a
filesystem that doesn't checksum with non-ECC, because it will detect Bad
Things happening and tell you about it. ECC helps avoid the problems in the
first place.

[1]
[http://research.google.com/pubs/pub35162.html](http://research.google.com/pubs/pub35162.html)

~~~
wtallis
ECC isn't that much more expensive, _if_ you already have a processor and
motherboard that give you that option. If you're trying to use ZFS on surplus
consumer Intel equipment, ECC is a major expense.

And aside from the cost issue, I don't think people are _resistant_ to using
ECC. You have probably just misinterpreted people who are justifiably shooting
down ZFS/ECC scaremongering: Advising people to never use ZFS without ECC RAM
is _bad advice_ , because the advice should simply be to use ECC RAM if you
want enterprise grade reliability, whether or not ZFS is part of the picture.
Some people _aren 't_ in need of that level of reliability but can still
benefit from ZFS, and they shouldn't be misled into thinking that ZFS has some
particular need for ECC RAM.

~~~
__jal
I agree on the last part, and I think the particular weirdness here is a
result of ZFS historically mentioning the value of ECC a lot more than other
FSes. I think this probably makes people think that ZFS in some way depends on
it more than others, rather than simply pointing out that there's a
disturbingly high chance of encountering a problem without it that applies to
everything.

And sure, if you're building a frankenbox, ECC is probably not an option, or
the sort of expense that takes it out of the frankenbox category. I do hope
that folks wouldn't store important data on dodgy hardware, but that is about
more than RAM, and also none of my business.

I don't believe I'm misinterpreting people's reactions - I've witnessed people
assert that ECC is a scam, waste of money and similar. Even after I point them
to that link I posted above, they still seem to believe that It Won't Happen
To Them. ("I don't have millions of machines.")

In any case, I still find it frankly bizarre that people run the risk with
important data. If you asked people if they wanted to buy a CPU that had an
~8% chance of undetectably lying to them, I'm pretty sure the vast majority
would at least want to spring for the premium one that allows at the least
detecting the lie.

~~~
wtallis
> I've witnessed people assert that ECC is a scam, waste of money and similar.
> Even after I point them to that link I posted above, they still seem to
> believe that It Won't Happen To Them. ("I don't have millions of machines.")

If by your own citation 92% of DIMMs operate with zero errors per year, and a
consumer machine has at most four DIMMs, then it _is_ actually pretty likely
that any given consumer machine will operate without RAM errors even without
ECC. And when you multiply by the low probability that a DRAM error will cause
catastrophic data loss, then it is very easy to come to a reasonable
conclusion that ECC is not worth the expense.

~~~
__jal
Hey, it is your data. I personally don't like gambling with mine, but yours is
none of my business.

Again, I don't know how many people out there buy other products with a nearly
1 in 10 chance of undetectably not performing the function they are supposed
to perform, but it seems nutty to me.

Although that line of thought does go some way towards explaining the vitamin
business...

~~~
wtallis
It's disingenuous of you to keep putting the error probabilities in terms like
"nearly 1 in 10" without acknowledging the context that you're talking about
the probability of a _transient_ error occurring at any time over the course
of a full year of continuous operation. Most people actually _are_ comfortable
with the idea that their equipment will have occasional downtime or faults,
but you're trying to paint a very different picture.

~~~
__jal
Buddy, I would heartily encourage you to believe whatever you makes you happy.
Your insult is false and petty, and I think you're pretty wrong about the rest
of that.

I personally consider it pretty disingenuous to call a fault _that damages
data on disk_ 'transient'. The root cause may have been transient, but the
damage doesn't go away if/when a stuck bit functions normally again. Or do you
consider a stroke leading to paralysis a transient injury?

I'd also like to see a cite that "most people actually are comfortable with
the idea that..." a 'transient' fault that scrambles random data they've
chosen to keep. Where, exactly, are you getting this?

Finally, assuming you have some actual basis for that claim, how many of those
people run ZFS? You wouldn't conflate nontechnical people who buy the cheapest
box at Best Buy with folks who take the time to configure software RAID across
several disks using a nonstandard filesystem, would you?

------
black_knight
Why would each file system need a "native encryption"? What gains are there
fromt this?

Encryption seems it would be more cleanly implemented transparently underneath
the file system level.

~~~
johncolanduoni
There's a few things.

First off, ZFS takes advantage of doing its own volume management on bare
metal. Some aspects of data recovery/resilience work better when it can
interact directly with the device, particularly for resistance to bit rot
(which is one of ZFS's biggest advantages). Putting something like LUKS in
between isn't as bad as LVM or hardware RAID, but is not great.

Second, it enables the use of different ways of using a particular block
cipher like AES (namely GCM instead of say XTS) that have some advantages such
as authentication of data. I'm not sure if it's an option for this particular
implementation, but nothing about the way ZFS encryption works would preclude
using XTS, while GCM doesn't really map well to block device encryption (there
is no good way to store the extra IV and authentication code, while ZFS can
put it in the metadata).

There are of course disadvantages. Some information about the structure of the
data on your drive is accessible that would not be if you used block
encryption. Also, unlike LUKS or LUKS w/ LVM you can't easily mix filesystem
types on the same drive set.

~~~
therealmarv
I wonder how/what is better with direct hard drive access here, if there is
bit rot on LUKS I expect it is also fixed with ZFS, what should be different?
"Some aspects of data recovery/resilience work better when it can interact
directly with the device, particularly for resistance to bit rot (which is one
of ZFS's biggest advantages)."

------
knz42
So now thanks to this there are two implementations of AES in the linux
kernel. Who's responsible for ensuring they are both correct?

~~~
nine_k
The other AES implementation is not licensed and thus can not be exported for
use with ZFS.

(And no, it's not the USA crypto export restrictions circa 1994, it's two
pieces of freely available code with different notions of freedom coded in.)

------
emaste
Note that the commit linked here is a port of the Illumos Crypto Framework
(ICF), which is a dependency but is not the change that actually brings native
encryption.

------
jvehent
That is a terrifying amount of crypto code. Has anyone audited this, or plan
to?

~~~
jlgaddis
It was _Signed-off-by_ two devs @llnl.gov so there's no need to worry, the
U.S. Government is looking out for us. /s

------
nixomose
Is there really a need to make zfs your root volume? You can reinstall your
root volume in a few minutes from a flash drive. What you really want is your
home directory to be zfs, and just do all your work in your home directory.
Saves all the grief of trying to make it your boot volume and it works just as
well.

~~~
doublerebel
This is basically what I do. A copy of the root volume can be kept in a ZFS
filesystem since the zpool can be easily mounted during any recovery.

------
mei0Iesh
I think this is most useful for cross-compatibility. Right now if a client
uses FreeBSD ZFS, and you need to mount it to access project files on your
Linux desktop, you can't if they used encryption. But after this is standard,
you should be able to mount the same ZFS filesystem anywhere.

~~~
mioelnir
Do you create zpools on memory sticks? Or how do you export the pool and move
the device with the exported pool on it to your linux desktop?

~~~
mei0Iesh
Yes, you can use ZFS on memory sticks. You can also "zfs send | zfs recv" to
copy a single projects dataset snapshot. You don't need to use actual disks
though, if someone wants to send you project files you can have them either
send you a file from "zfs send > snapshot.zfs", or send it over SSH, "ssh
friend zfs send | zfs recv".

Encryption could be an issue if for example someone uses a FreeBSD based NAS
for large data files, and you want to skip the network and just access them
directly from your Linux box. You can "zfs export; zfs import", but not if
they used encryption. That's where I think this will be useful, because then
we have one standard filesystem we can use everywhere.

~~~
mioelnir
I am well aware of all of that. But all the zfs send/recv options can by their
very definition not have the full disk encryption problem hinted at in the
comment I replied to.

Also, if it is easier to take your NAS offline and apart to chuck the disks
into your desktop (compared exporting it over the network), then your NAS is
too small. What you were looking for is a Laptop.

Which leaves abusing the zpool on a memory stick as data interchange format.
Most likely with copies=1, so you have to add some par2 files anyway, at which
point you could simply put them on figuratively any other filesystem out
there. And encrypt them with gpg/openssl etc. That way I would also not have
to run a potentially maliciously crafted filesystem within my kernel.

------
mrsirduke
> We cannot use the Linux kernel's built in crypto api because it is only
> exported to GPL-licensed modules.

What a strange choice by the Linux kernel.

~~~
cyphar
The crypto code is all very internal to the kernel, so of course it will be
GPL. Besides, I personally think that all kernel drivers should be GPL because
they are essentially all derivative works of the kernel. The ability of module
loading to get around the GPL is one of the more worrying decisions made by
Torvalds.

------
Infernal
As someone who just last night set up an Ubuntu 16.04 server with the
intention of using ZFS, should I wait until this hits the Ubuntu repos? Is it
possible to enable this encryption on an existing filesystem?

~~~
thehnguy
Similar questions here. It's unclear to me how I can integrate this into my
system and how to go about it. I'd hate to have to blow away my zpool to do
this.

My zpool isn't very full. Maybe I can make a new dataset that's encrypted and
move everything from the unencrypted dataset to the encrypted dataset. Of
course, being able to encrypt in place would be most ideal, but I'll settle
for it being a painless operation that doesn't require blowing away a zpool.

------
DashRattlesnake
So what's the relation of this to OpenZFS? Is this currently just for the
Linux port, and not yet pulled into OpenZFS for other platforms?

~~~
emaste
This work is being done in ZFS on Linux but will make its way back into
OpenZFS/Illumos and FreeBSD.

------
lmm
So does ZFS on FreeBSD support native encryption? Can I switch my existing
pool?

------
zxv
native encryption with AES-NI support sounds awesome. When is likely to make
it into ubuntu repos?

