
Hyperdrive – a novel file sharing network - sgrytoyr
https://github.com/mafintosh/hyperdrive
======
nine_k
«A file sharing network based on rabin file chunking and append only feeds of
data verified by merkle trees.»

As far as I understand, this is about efficient verified replication of files,
so that only smallest changed chunks are actually replicated (unlike e.g.
rsync).

~~~
mindslight
rsync and Rabin chunking are both based on rolling hashes determining where
exactly to cryptohash. rsync operates "in place" and benefits from chunks
being in consistent order, whereas Rabin chunking uses precomputation to draw
from unordered basis chunks (eg cross-file chunking).

~~~
zbjornson
Pardon my ignorance, but I'm still not quite understanding the applications
and benefits. Distributed file hosting? High-concurrency reads? Torrent-like?
"file sharing network" is a bit ambiguous to me.

(I seem to have broke the demo with a 100 MB file, btw.)

~~~
mindslight
(Not my project. I'm just commenting on the general techniques)

Rabin chunking should get you quick cross-file partial-file deduplication. So
if you're already storing a file and then download a modified version (from
someone else) with changed metadata in the header, only the differing data
should be transferred/stored.

------
Fice
Looks similar to IPFS ([https://ipfs.io/](https://ipfs.io/)) but with less
ambitious goals. And the IPFS team is also working on JS implementation that
will run in browser: [https://github.com/ipfs/js-
ipfs](https://github.com/ipfs/js-ipfs).

------
mankash666
I'm unable to get the demo [1] to work. I'm using Firefox 42.0 on Mac OS
10.11.12

[1]:
[http://mafintosh.github.io/hyperdrive/](http://mafintosh.github.io/hyperdrive/)

~~~
mafintosh
yes i haven't tested the demo in firefox yet. there is an open issue for that
on the hyperdrive repo. it should work in chrome though and there is no reason
it shouldn't work in firefox once i find the bug.

------
out_of_protocol
Any idea how peer discovering work? Can't find any mention of it

~~~
mafintosh
it's in the README examples. you can use discovery-channel,
[https://github.com/maxogden/discovery-
channel](https://github.com/maxogden/discovery-channel) to do peer discovery
which in turn uses multicast-dns and the bittorrent dht to find peers.

