Hacker News new | past | comments | ask | show | jobs | submit login

I'm not sure I understand your use case well, but have you tried lvm's snapshots? That could be the simplest solution.

If you're going to try btrfs, check if your system/tools handle the space overcommit correctly. Some ways to check the available space don't really play well with snapshots. (As in, they report less space available)

The use case: A build system runs 100's of compile/test jobs that use the same underlying git repo. The git repo is massive so giving each process its own copy is not practical. Instead each subprocess gets a unique overlay to do it's work. After a subprocess finishes if it has any artifacts that will be used by a later stage of the build they are copied to a "common" overlay which will get mounted on top of the original lower file system used in the previous phase. So the system needs to be able to use multiple lower overlays (RO) as it moves through each phase. After its all over all the data needs to be quickly deleted such that the system is free to perform the next job.

You can use a btrfs subvolume for the build location, and take snapshots at all rollback points. This is fast. Basically, you create an r/w snapshot at first that will evolve the current batch's "common" overlay.

Proceed by making another r/w snapshot for the/each subprocess.

Just work inside that folder (which behaves ~like a CoW bind-mount) and copy the artifacts into the "common" snapshot.

btrfs sub del -c path/to/temp/build/snapshot

will clean it up and ensure it won't reappear after a crash. You can skip the -c if you don't care about it reappearing.

If you have enough space to handle up to maybe a couple minutes delay in garbage collecting, you don't need to force any special syncing or such, as it just happens in the background.

As for the phases, I don't think this is a problem. btrfs snapshots are like git branches that don't allow merging, leaving explicit copy as the only recourse.

I'd definitely try lvm snapshots in that case. They free you from the specific choice of the filesystem. But I don't know your exact workload, so... testing and profiling some options. Make sure you try larger lvm snapshot blocks to lower the overhead - 4KB may still be the default and it kills performance.

> The git repo is massive so giving each process its own copy is not practical.

Just to make sure: This is per point-in-time, and not just because of history? You can usually make things somewhat cheaper just by doing a shallow clone. But of course that doesn't help if at any given point-in-time the repo is still too big.

Even then, you can actually use multiple external worktrees, saving on the .git folder (or at least any significant size contributions of it).

I don't think git uses copy_file_range yet (at least I can't find it looking through its source), so it'll make expensive copies for each worktree.

And he said even deleting the diffs in an overlay is too expensive, which normally should be cheaper than deleting a worktree.

correct its per point in time. Either way the overlays are a TON faster. :)

It is likely a lot of work, but sounds like working towards out-of-tree builds will pay great dividends going forward in your codebase.

Another way to do this with just userspace tools could be git clone -l, combined with committing artifacts to a temporary branch to be used by later stages. But maybe the space savings would still be suboptimal because there would be copies of the artifacts both in the workspace and the git db?

I have done something similar using a ramfs for the work directories. That way when you are finished you simply save what you want and delete it. Building is much faster than to disk. If your data does not fit in RAM, just use an image file in the same way.

I think there is a limit on how many layers Docker can handle in an image and if I recall correctly the number is like 63 layers. I don’t have any idea why it is like this but it could be chosen for performance reasons.

> The use case: A build system runs 100's of compile/test jobs that use the same underlying git repo.

Very interesting. Thanks for sharing!

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact