
Initial Hammer2 filesystem implementation - joeschmoe3
http://lists.dragonflybsd.org/pipermail/users/2017-August/313558.html
======
jitl
Very exciting to see implementation progress on HAMMER2. Some basics about the
design:

\- This is DragonflyBSD's next-gen filesystem.

\- copy-on-write, implying snapshots and such, like ZFS, but snapshots are
writable.

\- compression and de-duplication, like ZFS

\- a clustering system

\- extra care to reduce RAM needs, in contrast to ZFS

\- extra care to allow pre-allocation of files by writing zeros, something
that will make SQL databases easier to run performantly on HAMMER2 than on ZFS

And much more. The design doc is an interesting read, take a look:

[https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEA...](https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEAD:/sys/vfs/hammer2/DESIGN)

~~~
chongli
_snapshots are writable_

Isn't that an oxymoron? I thought the entire point of a snapshot was that it's
an immutable record of the filesystem at a moment in time.

 _extra care to reduce RAM needs, in contrast to ZFS_

ZFS utilizes a lot of RAM for caching. That's not the same thing as needing a
lot of RAM. I've seen this same complaint about modern operating systems.
People will buy a lot of RAM and then get annoyed when they see the OS making
use of it. As long as the memory is available to be allocated to applications,
why should we care whether or not the operating system makes use of it?

~~~
tedunangst
Is it better if we call a snapshot a branch point?

I believe the complaint about ZFS and RAM is that the caching cannot be
considered optional. Performance is substantially worse than other filesystems
with more modestly sized caches, so the memory isn't really available to
applications.

~~~
chongli
_Performance is substantially worse than other filesystems with more modestly
sized caches, so the memory isn 't really available to applications._

Is that in comparison to other filesystems that provide the same level of data
integrity as ZFS? Otherwise you're comparing apples and oranges.

~~~
tedunangst
You didn't say the memory use was justified by providing better integrity. You
said it was optional.

~~~
chongli
Yes, the memory use _is optional_. Pick two:

1) data integrity

2) low memory use

3) performance

~~~
dozzie
With ZFS I hear it's "pick one, and you cannot pick `low memory'".

~~~
gigatexal
Lol this is FUD. ZFS is easily tunable to not blow up and use all your ram.

~~~
benchaney
Does "FUD" just mean "something I disagree with" now? How is this FUD?

~~~
gigatexal
Spreading misinformation is what I was taking issue with not just disagreeing
and making a stink about it like a troll might.

~~~
JdeBP
You're not appreciating nor applying the _fear_ , _uncertainty_ , and _doubt_
parts of "Fear, Uncertainty, and Doubt".

------
gigatexal
Dillon, I had always thought was a hack when he forked FreeBSD at 4.x but he’s
proven to have some novel ideas when it comes to these things and I’m looking
forward to trying out the production ready Hammer2 FS

~~~
jacobush
I have great respect for his ability to get shit done ever since Amiga days,
so my thought was not "a hack" but rather - "how can he have this much
patience and stamina?!"

~~~
laumars
I'm not aware of this work on the Amiga. What kind of projects was he involved
in?

~~~
rodgerd
The DICE C compiler/IDE was his best-known bit of work.

------
TallGuyShort
Other than the design doc (which, being BSD, is bound to be the primary source
of truth), does anyone know of any tech talks or more visual presentations
about the design of Hammer FS? I sure do love being available, but for just
starting to wrap your head around an FS architecture, talking through some
slides would sure be neat. I'm not immediately seeing much on YouTube...

~~~
JdeBP
Well, there's BSD Now #53, although that's primarily verbal.

------
alberth
It's amazing how much work the Dfly has acompkksjed given how few developers
there are.

I really hope Dfly gets more adoption and broader use.

------
blue1
Does H2 feature data integrity (checksums etc)? For me that is one of the best
features of ZFS

~~~
mjevans
It's a short email post. Yes, they're build in to the inode structure and
would be used for de-duplication. (However I'd hope that for a production
system there's still a /verify/ pass on the data being de-duped to confirm it
actually __is__ a dupe.)

~~~
mrighele
Interestingly, this is not what happens in ZFS [1] (unless the defaults have
changed since a few years ago):

    
    
        If you accept the mathematical claim that a secure hash like SHA256 has
        only a 2\^-256 probability of producing the same output given two different
        inputs, then it is reasonable to assume that when two blocks have the
        same checksum, they are in fact the same block. You can trust the hash.
        An enormous amount of the world's commerce operates on this assumption,
        including your daily credit card transactions. However, if this makes
        you uneasy, that's OK: ZFS provies a 'verify' option that performs
        a full comparison of every incoming block with any alleged duplicate to
        ensure that they really are the same, and ZFS resolves the conflict if not.
        To enable this variant of dedup, just specify 'verify' instead of 'on':
    

[1] [https://blogs.oracle.com/bonwick/zfs-
deduplication-v2](https://blogs.oracle.com/bonwick/zfs-deduplication-v2)

~~~
loeg
And HAMMER2 will use even larger 512 bit hashes for blind dedupe, making the
possibility of a collision causing data loss vanishingly small.

~~~
Dylan16807
Bah, 256 bits is already so vanishingly small it's hard to comprehend. If you
put a thousand disks into a pool you might reach 2^40 blocks, leaving you with
256-80=176 bits of margin. That is never going to collide. You could make such
a filesystem for each atom on Earth and the odds of a single collision would
be less than 0.1% You could put a _billion_ disks in a pool and still have 136
bits of margin.

512 bit hashing is basically placebo.

~~~
loeg
I agree that 256 bit hashing is totally sufficient. I wouldn't call 512 bits
"placebo," though. Just overkill.

~~~
Dylan16807
You hit a point where the longer runtime increases your chance of error by a
very small amount that is nevertheless larger than the protection you gain.

------
pmoriarty
Is there any work to get HAMMER2 on Linux?

~~~
rurban
The better question is when are you ready to switch to a proper kernel,
DragonflyBSD? When you think about HAMMER2 you should really think about all
the other improvements also.

~~~
zsd
I use Linux just because it has a large community and good driver support. But
I like hearing about innovations in OS development.

What would you say to me or show to me to sell me on the idea of using a BSD?

~~~
zsd
I rest my case.

------
beastman82
This sounds like a weapon system designed by Stark Industries.

~~~
qbrass
Not Hammer Industries?

~~~
beastman82
bah! yes that's much better.

