
Persistent “pipes” in Linux - an_ko
https://gist.github.com/CAFxX/571a1558db9a7b393579
======
otterley
See also Varnish's log implementation, which employs a similar technique but
uses a shared memory region instead: [http://varnish-
cache.readthedocs.org/en/latest/users-guide/o...](http://varnish-
cache.readthedocs.org/en/latest/users-guide/operation-logging.html)

------
zimbatm
I wonder how it behaves in terms of performance and disk fragmentation.
Potentially log-oriented databases could adopt that technique instead of
having lots of files on disk.

------
teddyh
I would like to know what the file system implementors have to say about this
technique. Is their reaction “ _Aargh, why would you do that, it wrecks all
our optimizations!_ ” or is it “ _Huh, OK, that’s kind of odd but I guess it
would technically work_ ” or even “ _Yes, finally someone who takes advantage
of all our nice features like we always wanted!_ ”?

~~~
planckscnst
Or maybe, "Huh. That's interesting. Maybe I'll make a new filesystem object
type or add an optional disk buffer for pipes."

------
ksherlock
Another approach (used by qmail and I'm sure other programs) is to use the
file system as a database. Process 1 creates a file into a temp directory.
Once writing is finished, the file is atomically moved to the working
directory where Process 2 deals with it. That wouldn't work so well if you
need them to be processed in a specific order, though.

------
porlw
Soooo...

What happens when an errant grep wanders in to the directory holding these
persistent pipes?

~~~
_yy
Nothing, since those "persistent pipes" are just regular sparse files (just
like VM images). Clever idea.

------
joshbaptiste
Some previous thoughts:
[https://news.ycombinator.com/item?id=10068591](https://news.ycombinator.com/item?id=10068591)

------
djcapelis
Hmm, sparse files get a little annoying after the offset gets high enough.
I've had weirdness with sparse files that had offsets in the 64bit range.
So... eventually if you don't compact and start from a low offset ring buffer
style, this solution will stop working after you pass enough data through the
"pipe".

Yes. That means a lot of data, but it's a boundary case worth knowing about.

