Hacker News new | past | comments | ask | show | jobs | submit login
A transcoding FUSE filesystem for music (khenriks.github.io)
96 points by jm3 on Sept 15, 2014 | hide | past | favorite | 34 comments

I'm hoping that this type of 'filesystem as a utility' becomes a common method of chaining programs together, simmilar to the pipe.

On a related note, another FUSE filesystem I have used is ciopfs [1] which makes a case insensitive filesystem from a given folder. Very useful for running windows programs that are incidentally platform independent (as in written in a language like Java) and break on case sensitive filesystems.

[1] http://www.brain-dump.org/projects/ciopfs/

Whoa, my mind is more than a little blown at this idea. Like Dropbox, but for actions or operations. Just drop a file and now its being converted or manipulated. I hope this becomes a reality!

It isn't efficient. We've had compositional tools for ages with good old unix pipes and similar; you can build those into a kind of hierarchical namespace with virtual filesystems, like a bunch of tools similar to this one, and view things through some chain of filesystem interfaces to implicitly perform operations on them. But these methods of composition are very opaque to the kernel that connects their components (FUSE is even slower than this design contraint implies because of how many times you go back and forth between the kernel and userspace), and as such you have lots of redundant copies of buffers, serializations/deserializations, and so on, which destroy your performance compared to a big monolithic "blob" program with built-in specialized processing pipelines.

The real direction to look is for systems where your tools are composable but also which have novel means of performing whole-pipeline optimization--like stream fusion in haskell, or a JIT'd Lisp machine. Of course, those examples are far from perfect, but they show some attributes of what partial solutions to the age-old problem might look like. They do have their own problems as well, like ad-hoc data processing in haskell being more thought-intensive (though, you know, correct in processing edge cases) compared to composed string filters in a shell.

You could take a job composition system like unix pipes, and add a buffer discipline that allows for zero-copy, or provide a way to declare buffering behavior, but at that point you're essentially building a typesystem on top of some real-world semantics, and it's a better idea to start from what you want to achieve in the first place and then design upward from there. If you start from the bottom up, you can also merge the operations performed inside each (memoizing pure work or fusing stream operations) or describe what kind of data you accept and produce, both of which can help avoid redundant computation. You'd probably end up with something with affine or linear types, a module system a la ML, and some novel combination of other handy features for ergonomics (which is largely what has kept unix pipes relevant to this day).

Presumably, if this were to catch on, it would start in toy projects and projects with a major usability advantage, where the performance loss is acceptable, and as a secondary means of using other programs. Then, we might see optimizations to make the performance of this method not horrible.

In concept, it should be relatively simple to recognize where a FUSE filesystem is mounting another FUSE filesystem and bypass the kernel to let the two processes talk directly to each other. You still have the problem that any operation would need to travel through N processes. You might be able to improve on this with a mechanism for a filesystem to say "I have to effect on operation X" and have it be bypassed when X occurs.

Correct me if I'm wrong, but I would've assumed that most methods of IPC (unix sockets, pipes...everything except shared memory) would need to be copied into kernel space (via a write(2) then copied back out (via a read(2)) at some point. Are you suggesting that FUSE add a mechanism to open some shared memory communication path between FUSE processes?

Another way of tackling the inefficiency is to provide a composable filesystem-like abstraction in userspace.

Like the scheme handlers in Polymorphic Identifiers: http://objective.st/URIs/

As a final step, you can export such a composition via FUSE, but that's just for legacy support.

I've written a few data processing systems over the years and one of my foundational principles is to eschew complex command line parameters and instead just use the filesystem to infer things.

For example: building a comic book library? Where do I store the publisher, title and issue metadata? Why not just infer it from the organization of the file system?

D.C.Comics/Batman/The Batman #123.cbz Marvel/Avengers/Avengers - Ultimate #12.cbz

I once built a text processing and conversion system that used a different scheme.

You mount or copy your text files into a folder "source" and it picks up whatever's in there processes it and dumps the converted result out to "sink".

Later I set it up to have some filters on the data. So "sink" turned into a hierarchy of different pre-filtered results.

sink/Car_Makers sink/countries/France sink/countries/England sink/Truck_Makers

And in the end, only documents that fell into one of those classes appeared in the relevant sink folder. At a glance you'd know if there was something relevant to that class of documents or not just by running "ls" or "dir" on it.

It cut hundreds of hours off of analyzing the results since you'd know a head of time if it was likely you'd find something productive related to one of the output classes.

This work with fuse is really exciting.

Imagine dumping in your text files wherever, fuse mounting it and then just exploring the results in all kinds of useful ways. Maybe you write an "entity extraction" filesystem.

So "Mobydick.txt"

Gets turned into

/mnt/mobydick/characters/ishmael/chapters/1 /mnt/mobydick/characters/ishmael/chapters/4 /mnt/mobydick/characters/ahab/chapters/4 /mnt/mobydick/characters/queequeg/chapters/3 /mnt/mobydick/characters/queequeg/chapters/4

/mnt/mobydick/chapters/4/characters/ishmael /mnt/mobydick/chapters/4/characters/ahab /mnt/mobydick/chapters/4/characters/queequeg

/mnt/mobydick/locations/Nantucket/chapters/1 /mnt/mobydick/locations/Nantucket/chapters/4 /mnt/mobydick/locations/Pequod/chapters/4

/mnt/mobydick/chapters/4/locations/Nantucket /mnt/mobydick/chapters/4/locations/Pequod

/mnt/mobydick/chapters/1/text.txt /mnt/mobydick/chapters/2/text.txt /mnt/mobydick/chapters/3/text.txt /mnt/mobydick/chapters/4/text.txt

And now you can use the fuse filesystem as an analytic tool

I've been a little obsessed over these same lines. It seems ultimately to boil down to a tag-based fs (or at least that's been the most productive way to search for people trying to do it.) Here are a couple attempts I know of:

http://www.tagsistant.net/ http://web.mit.edu/6.033/2011/wwwdocs/writing-samples/sbezek...

What would be also interesting is to store all book files in a database. Then you add tags with author, year and so on and then you can create several FUSE filesystem like author/year/title or just author/title or author/series/title. So you can browse you collection in different ways. The same would work for images or music. Genre/artist/album/title or year/artist/album

I'm using Plex for comics. I don't worry about where the metadata is stored. There's an interface to enter it manually, and an API to write an agent script that grabs the metadata for you.

Trying to shoehorn it all into the filesystem as directory names seems... destined for disappointment.

Out of curiosity, how are you using plex for comics? The photo gallery feature? I keep asking the developers to add cbz and cbr support but it doesn't seem to be coming.

Yeh, the photos. The developers are assholes, it's a closed source project... but everything else is so bad it's tough to not go with it.

I've also figured out how to do audio streams (radio stations).

that's called a semantic file system

I guess kinda. Except the semantic "files" are just artifacts of some process on the real files that transforms them into the output FS.

My understanding of Semantic file systems is that the links between files approximate directory traversal but the files or more or less that normal files you'd have on your file system.

This reminds me of a very fun assignment we had in college: we had to implement a FUSE filesystem to extract ID3 tags from MP3s. This was part of our OS class to learn about the filesystem (while at the same time we were diving into the Linux kernel).

I'd really recommend to anyone who wants to understand more about filesystems to write your own FUSE filesystem. It isn't difficult (assuming you have a basic understanding of C), they run in userspace so they can't mess things up too badly, and it is really gratifying to drag/drop files into a folder and have magic happen.

Actually, you don't even have to know C; you can do it in Python. Here's an example: http://www.stavros.io/posts/python-fuse-filesystem/

Huh, I had used MP3fs or something like it years ago. I see this project's code dates back to 2006, so maybe this was it? Glad to see it lives on :)

Yeah, mp3fs has been around a long time. It's really useful.

Very cool. I had this same idea (and of course secretly hoped I was the first to have thought of it, but that's rarely the case) just the other day, as I found myself having to re-encode a bunch of FLAC files purchased at HDtracks.com to M4A, since Spotify for some absurd reason doesn't support FLAC. It's straightforward enough with ffmpeg, but it's yet another annoying step required before you can listen to the music.

Alternative: This script synchronizes a music collection recursively one way from a lossless format source directory to a lossy file format target directory.


Nice! I did this a few years ago as a class project but didn't take it as far as this project is doing. Impressive.

Been using this for years - very useful and nice to see it being maintained again.

A while back I saw a similar project that used Samba to transcode on the fly.

Why specifically MP3? Something more up to date like Opus could be more interesting.

Probably because it's the most commonly supported format by portable players?

There is more to music than just portable players.

My car supports only MP3s, and only up to ID3v1 tags (will not play files with ID3v2 tags). That's the reason, why I convert everything into MP3. I'm not going to change my car or it's audio system for few more years.

Hard to say, for portable players I only use something that can run Rockbox and it supports Opus just fine. But the general trend now is decline of their usage altogether. With the rise of handsets and tablets portable players became more exotic, like PDAs of the past. I barely see them being used anywhere. And with general purpose mobile OSes and software media players, supporting more codecs shouldn't be an issue.

When I get some music in lossless format like FLAC I encode it in Opus for regular usage.

I guess because it's better to use a format everyone has heard of rather than one no one has heard of ?

Not really. It's like proposing to use IE6 because people "have heard of it".

I wonder what people here like outdated patented codecs. It's like a lose lose in all aspects.

My Rockboxed Sansa Clip crashes when I try anything in Opus. I keep FLACs in my music collection, but I encode to Vorbis for the Clip. I used to put the FLACs on there directly, but with Vorbis I can put 4x more music on it. I also need to have better hardware for FLAC to make a perceptual difference.

It works well for me on Sansa Fuze. Are you using the latest release or Rockbox?

Vorbis is OK, but Opus allows using lower bitrates for comparable quality, so it can save space and also should be more optimal for CPU utilization.

An interesting test: http://listening-test.coresv.net/results.htm

I keep FLAC for originals too, but encode in Opus for regular usage. FLAC allows reencoding in the future if anything improves, new codecs come out and so on.

As someone who is thinking about moving to opus for non-archival music playback, what sort of bitrate targets do you use for encoding?

I know opusenc uses VBR, but as far as I can tell there is no V0 equivalent that is accepted for high quality lame (MP3) encoding.

Until now I used to encode it with --bitrate 160 which is 160 Kbps variable bitrate. But from the test I linked above it seems to be an overkill, since 96 is already approaching encoding transparency for many people. So the optimal value can be somewhere above 96 and below 160. You can experiment for yourself, but I'd give a bit more room than your own transparency level to make it a safer bet for more people.

Yeah, latest release (3.13). I'm not too worried about saving space, since I have a 32 GB microSD card in it, and my Kickstarted 64 GB PonoPlayer should arrive before Christmas.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact