
LCFS: A New Container Filesystem for Modern Datacenters - old-gregg
https://portworx.com/lcfs-new-filesystem-modern-datacenter/
======
davexunit
How many more file systems are we going to see before we realize that the file
system is the wrong layer of abstraction to be solving the deduplication
problem?

~~~
LeoPanthera
I think the filesystem is a fine place to do it, I just wish more filesystems
would give us the option to do deduplication offline.

ZFS's dedup uses grotesque amounts of RAM since it insists on holding the
dedup table in it and only doing dedup in realtime.

btrfs however allows you to scan the filesystem and do dedup of existing data
later (even while it is mounted), using no RAM at all. This is one of a couple
of places where I think btrfs is actually becoming a better-designed
filesystem than ZFS.

~~~
jjobi
Again, the focus of this blog is not about a de-duplication story, but about a
new docker storage driver.

~~~
LeoPanthera
Well thankfully open discussion is a feature of Hacker News, not a bug.

------
gourao
Hi, I am one of the contributors on this project, and happy to answer any
questions... LCFS will also be supporting the snapshot driver discussed here -
[https://github.com/docker/containerd/pull/484](https://github.com/docker/containerd/pull/484).

Currently, the install experience is not the greatest... We rely on the Docker
v2 plugin interface and that is still in beta. We are working with the Docker
team to help us smooth out some of these rough edges.

LCFS will also support other container formats and any help from the community
is much appreciated.

~~~
akanet
This looks amazing. One of the biggest pains of using docker is that the
storage driver inevitably explodes the system.

Is anyone using LCFS in production? Why did you get started building it?

~~~
gourao
LCFS is not yet being used in production as far as I know

We started building this because there are a lot of issues running containers
in production, specifically around image space and inode management and also
with page cash usage.

This file system was specifically designed to address memory and storage
management of container images

------
CSDude
Really nice to see the new file systems addressing this issue, but, as many
including me hurt by AUFS, I am being cautious. Do you think it is battle
tested enough? As it is different than other merging ones, it is a little more
dangerous than them as well since it works on block level? Not trying to be
negative, I really liked it, and I am not even happy about even a docker load
command performance over my overlay2, and will defintely be trying LCFS.

~~~
jjobi
It is a mini-file system, working directly on top of a block device. Having
something else in between is not a good idea. Much of the data handled by
docker storage driver is temporary and does not even have to hit disk before
thrown away.

This project is currently in experimental mode. We would appreciate any
feedback you may have.

------
softwarelimits
Does anybody know of a good deduplication script / tool that runs on every
filesystem and can be used as a nightly cronjob?

~~~
kruhft
Here's a quick one if your filesystem supports hard links:

    
    
        #!/bin/ksh
    
        typeset -A hashes
    
        find . -type f  -exec shasum -a 512 {} \; |
            while read hash filename; do
                if [[ -z ${hashes[$hash]} ]]; then
                    hashes[$hash]="$filename"
                else
                    echo ln -f "'${hashes[$hash]}'"  "'$filename'"
                fi
            done
    

Pipe to /bin/sh if you're happy with the results.

~~~
rsanders
Good answer, but unless you mark the files read-only/immutable, this ought to
have big red flags all over it. Userland just can't do this safely in the
general case.

~~~
kruhft
Good point. This is for data that's not in use.

------
jpgvm
It's fuse based however, so that has some caveats. Interesting project though,
could see a native port one day maybe.

~~~
jjobi
This was done using _fuse_ to make it portable across different distros. We
have not seen any major issues testing it as a docker storage driver, compared
to the graphdrivers in use today.

