Steve's blog

Now that disk space is cheap, it makes more and more sense to rip music and store it in FLAC format. I used to use abcde to encode directly to Ogg Vorbis, but a while back I decided to re-rip. The disks in the server at home now have lots of hours of music in a nice lossless format ready for easy playback on various machines on the network.

However, that's not all I need. I also want to be able to create Ogg Vorbis files for my own music player with limited space. And Jo's Ipod won't play either .ogg or .flac (yay for closed devices!)... At least I now have readily-available original quality music already on hard disk, so I don't need to go and find CDs again. But I really don't want to have to work through my directories of music by hand, re-encoding individually.

My solution: write a filesystem that will read in the .flac files and convert to .ogg or .mp3 on the fly. Now I can simply read the files I want off the server (via sftp/rsync/whatever) and it will create them for me!

So, fuse-music is my first attempt to do this. It's a simple perl program that works using the perl FUSE bindings to generate a pseudo-filesystem where a tree of .flac files will appear as a tree of .ogg or .mp3 files instead. It's not wonderfully fast (as encoding takes time), but it works. The only major difficulty I have at this point is calculating the sizes of the output files: if I want rsync to work reliably then the reported sizes need to be static and correct. But it's not possible (AFAICS) to predict the size of an MP3 or Ogg Vorbis file without actually doing the encoding and measuring the output size. So (ick!) that's what the code currently does when necessary. It caches the output size in a perl hash DB so the work is only needed once per file; ideally I'd like to not to have to do this at all!

If this sounds interesting at all, feel free to have a look at and play with the code: http://git.einval.com/cgi-bin/gitweb.cgi?p=fuse-music.git;a=summary and I'd love to hear your feedback.

TODO:

Make it work with NFS?
Cache output for a while, rather than re-encode every time
Go multi-threaded, but may need to move to C for that, multi-threaded perl is nasty!
Add support for more output formats (wav/aac?)

18:53 :: # :: /software/fuse-music :: 9 comments

Comments

Re: Fuse-music, a transcoding filesystem
James Vega wrote on Fri, 02 Jul 2010 19:35

That's awesome! I once started a similar endeavor, which I was going to call transcodefs, but got bogged down in trying to support any <-> any conversion and figuring out how to pass data among the various audio libraries (should've just learned gstreamer).

Re: Fuse-music, a transcoding filesystem
jbd wrote on Fri, 02 Jul 2010 19:54

Hello,

it seems quite nice. mp3fs does that (mp3 from flac) using libflac and libmp3lame. It works very well and perform conversion on the fly : the file is never written on the disk and you can seek all over it.

I've tried to do something similar as James Vega (called transcodefs too :p) using gstreamer to deal with any<->any conversion. But it seems quite difficult to predict the size of the final file on the fly, which is necessary for having a nice behaviour (rsync like you mentionned).

I'll keep an eye on your work !

Re: Re: Fuse-music, a transcoding filesystem
jbd wrote on Fri, 02 Jul 2010 20:25

mp3fs calculates the final size of the mp3 file from the flac file and it seems quite correct since i'm using mp3fs on a daily basis without problems.

When i said that it's difficult to predict the size of the final file, i meant in a format agnostic way.

There is also two others projects based on gstreamer to achieve what you want : gstfs and gstfs-ng. The last time i looked, nothing was really on the fly but maybe it's the only way to be format agnostic ? In that case, full transcoding + a Last Recently Used cache seems the way to go, at least for me :)

Re: Re: Re: Fuse-music, a transcoding filesystem
James Vega wrote on Fri, 02 Jul 2010 20:51

gstfs looks a lot like what I was going for, except I wanted to hide the details of gstreamer from the user. It does do full transcoding + an LRU cache as you suggest.

QuFS
roguelazer wrote on Fri, 02 Jul 2010 21:10

There was a paper at FAST this year on a somewhat-related topic which you might find interesting:

http://www.usenix.org/events/fast10/tech/full_papers/veeraraghavan.pdf

Re: QuFS
Steve wrote on Sun, 04 Jul 2010 14:57

Cool, thanks for the pointer! :-)

Rockbox
David wrote on Sat, 03 Jul 2010 16:05

Or you can install Rockbox on your portable player devices to be able to play OGG and Flac media.

Re: Rockbox
Steve wrote on Sun, 04 Jul 2010 15:02

You can, but only if the device is supported. Last time I looked, Rockbox didn't support new enough ipod versions to cover the one that Jo has.

Re: Fuse-music, a transcoding filesystem > DLNA / UPnP
Simon wrote on Mon, 05 Jul 2010 10:23

You may use DLNA systems like python-coherence which transcodes media on the fly, though not through a POSIX filesystem.

Your Comment