How long until someone posts that the "future of CMS" is actually a centralized index, with the data existing wherever it makes sense? Kind of a personal card catalog.
I'll answer your question and then engage in some shameless self-promotion. I wrote about it after posting my comment. Here's the chunk which answers your question:
Where am I now?
Heh. I mentioned that looking for duplicate files is harder than I thought it would be. I'm actually on my third try. The first one was when I thought "I can do this with a script", the second was with .NET, where I aimed bigger, but found not nearly big enough.
So, I've just completed the work on the file crawler, and the next bit is submitting the crawl results to the index. I've done this part before, and I don't expect it to be particularly hard, but I have to find the time for it. After that, something resembling a UI (I am trying to solve a problem), then put the whole thing out there with a big fat "alpha" disclaimer (probably Apache license, since I'm using so much of their stuff).
Along the lines of what you're working on, I think what you really want is a form of network browser capable of locating files stored redundantly across a completely decentralized network.
Done correctly, this would make notions like "backups" completely obsolete, and "my files" would exist on any computer that allowed me to log in to the appropriate service.
Fast, decentralized, redundant network systems are a beast to engineer though, but it's do-able.
Bonus points if you figure out how to let users store private data on such a network, and keep it private, while also allowing them to share specific files with specific individuals or groups of individuals.
Pull that off, and you'll have the holy grail of data storage.
I've only got some rough notes on this; I don't think I'd start trying to actually implement it for another two years yet.
FWIW, that's exactly what I'm working on.