
Show HN: Compressed directories on Linux - jamesbowman
http://www.excamera.com/sphinx/article-btrfs.html
======
shmerl
Is there a way to mount loop device without sudo? Some kind of FUSE option?

It's useful as a workaround for 32-bit applications which aren't built with
LFS (Large File Support) enabled, and because of that fail on large XFS
partitions. It's a very common bug in games which come out in 32-bit only.
Making a loop device is a workaround, but using sudo for it all the time is
annoying.

Also, there is probably no need to explicitly create a loop device, unless you
already used up all existing ones (see man mount):

    
    
        if no explicit loop device is mentioned (but just an
        option `-o loop' is given), then mount will try to find 
        some unused loop device and use that, for example
    
              mount /tmp/disk.img /mnt -o loop
    

I.e. in such case I simply create an empty file of certain size, then format
it with some filesystem, and then mount it as a loop.

For example:

    
    
        dd if=/dev/zero of="$image" bs=1024K count="$img_size"
        mkfs.xfs "$image"
        sudo mount -o loop "$image" "$mount_path"
        sudo chown $USER:$USER "$mount_path"

~~~
hashhar
For iso images you can use fuseiso
([https://wiki.archlinux.org/index.php/Fuseiso](https://wiki.archlinux.org/index.php/Fuseiso)).
While for all other purposes you really should take the time to learn about
udisks. ArchWiki would be a great place to start.

~~~
shmerl
Interesting, thanks. Control with Polkit makes it flexible.

------
barrkel
Note that it's not recommended to use journalled filesystems on file-backed
loopback devices: [http://loop-aes.sourceforge.net/loop-
AES.README](http://loop-aes.sourceforge.net/loop-AES.README)

This approach is not a useful path to lower overall disk usage IMO because you
need to thick provision the directory up front, with an estimate of overall
compressed disk usage. That's a recipe for usability pain. Thin provisioning
with sparse files won't work well with a COW tree structured backing store
like btrfs either.

~~~
chungy
btrfs isn't journalled, but all the same, the issue is data integrity in a
crash scenario. That will affect btrfs, and frankly, any other filesystem (for
example ext2 or ext4-without-journal); imagine if an application depended on
syncing files at certain points and the underlying storage stack decided to
write out a "sync2" before "sync1" was written out.

------
wmu
What about data safety? Say, a hard drive got a bad sector, then a whole file
is broken. This always stopped me from using system-wide compression. But
maybe it's not as bad as I suppose.

~~~
hultner
I've personally been running ZFS with LZ4, where checksums handle corruption.
Another upside is that LZ4 is fast enough to actually improve I/O throughput,
since overhead is negligible. On my jail/container/vm datasets I've seen space
reduction upwards 80%, with backup-datasets I've seen around 30% reduction.

Another possibility is data deduplication, which is useful if you have plenty
of RAM. You can deduplicate across files on block level. However I have only
used it sporadically due to the high ram usage.

------
crypto5
Will this cause double-buffering (one buffer for host FS and another for
BTRFS) with doubled RAM consumption?

~~~
loeg
The host FS will only buffer compressed pages (so not _double_ ), but yeah.

------
doomrobo
I like this article style a lot. A very cool result, presented clearly and
step-by-step, in less than a full screen of text.

~~~
vortico
Yes, he had a real world problem, discovered that there's not really a great
solution in existence, made a new tool to solve it, and presented it in its
entirety, all in a single page.

~~~
im3w1l
Only thing missing is how to put it in /etc/fstab

~~~
LeoPanthera
If you're going to be using it regularly enough to want to put it in fstab,
you should probably just allocate a regular partition for it.

------
Zardoz84
Or you can simply use directly BTRFS with LZO compression enabled.

------
Filligree
Or you could use a base filesystem that supports compression, such as ZFS.
_shrug_

~~~
polygot
But if you just want _one_ directory on an existing filesystem to be
compressed, this might be an easier option

~~~
barrkel
The article's suggestion means eating up 50G of allocation straight away. I
don't think needing to estimate your compressed allocation up front is an
easier solution at all. Normally with directories you use thin provisioning
pulling from the whole device's free space; thick provisioning for a directory
is not a path for easier reduce space consumption.

------
fsiefken
With ZFS and LZ4 or LZ4HC you might get better compression, BTRFS is more
lightweight though, so it's a great solution. I wonder if encryption would be
possible in a similar way.

~~~
pmlnr
[https://en.wikipedia.org/wiki/EncFS](https://en.wikipedia.org/wiki/EncFS)

------
jwilk
From
[https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)
:

 _Show HN is for something you 've made that other people can play with. HN
users can try it out, give you feedback, and ask questions in the thread._

This blog post is not a "Show HN" meterial.

Plase change the title to "Using BTRFS with loopback for compressed
directories".

------
pmlnr
Use ZFS, create a dataset for the dir and turn compression on per dataset.

------
aeroevan
How is this different than a chattr +c on the directory? I suppose the
compress option doesn't force compression (so it will try to recompress
compressed files).

------
fuzzyfizz
I don't get it. How does this produce 1TB of storage with a 50GB backing
store?

~~~
rovr138
It creates a 50GB store. And because of compression, he can write 1TB of data
onto it.

He/She is hoping the 50GB's are enough since there are no calculations on the
article.

