
BTFS – mount any .torrent file or magnet link as directory - cab404
https://github.com/johang/btfs
======
saurik
This is effectively what IPFS is, but IPFS takes it to the next level by using
the equivalent of torrent files for subdirectories within the mounted torrent
files, so you can "traverse" entire networks of the pretty-much-a-torrent
files without first having to download and mount them. And like, you could
implement similar functionality here by having torrent files that contain
files named .torrent have those files be treated as directories, but at that
point IPFS is going to be much more feature complete; even like, if you have
some giant file, on IPFS that might be represented by trees of hashes as
opposed to merely a giant array of hashes so you can incrementally and
efficiently access the parts you need. So like, while this is awesome work, if
you find it _inspiring_ , maybe channel that inspiration into learning about
IPFS and extending that ecosystem, rather than doubling down on trying to
build on torrent files.

~~~
trevyn
Are there any design changes that could create organic adoption of IPFS?

Or, conversely, what design decisions do you think have hindered adoption?

~~~
dafoex
I've had a breif look at IPFS, but not enough to say anything with any amount
of confidence. That said, the thing that immediately put me off was that
everything seemed to want to be browser based, like starting some service that
makes a local web server that acts as a GUI - there was even a chrome
extension that tried to make this easy for you.

A feature that might exist, but I haven't discovered, that would make IPFS
more appealing to me, would be a way to mount the IPFS like it were an FTP
server, and navigate it with whatever file manager you prefer and not have to
worry about managing the backend stuff.

That is to say, I'd like BTFS for IPFS.

~~~
AgentME
It's built into ipfs already: ipfs mount

------
manjalyc
The source code is far smaller that I would've expected, tiny in fact - no
seriously, its probably <2000 lines of actual code and looks extremely simple
to audit and fix bugs. It seems the only real external dependency is
libtorrent.

Honestly, if your not already amazed if not at the simplicity of this project,
than you should be by the simplicity of fuse filesystems and the capabilities
it exposes.

~~~
calvinmorrison
If you think that's excellent and are interested in fuse, you're probably
intereted in 9front which takes the concept to another level. Ftpfs does what
it says, but webfs handles http and forms and the like, upas/fs can show your
imap inbox, it's splendid and simple and allows your tooling to be the
familiar nix-like utilities like awk and sed. Even capture your screenshot by
catting /dev/screen to a file

~~~
so_tired
Which lib/approach would you recommend if i need run a remote batch process
using local files?

For example, on the local system:

remote-execute gcc localfile.c -o localfile.o

And on the remote system, there is a gcc and a remote agent which receives and
retransmits the files

~~~
calvinmorrison
You would use rcpu(1) which essentially imports the remote cpu. In other words
the remote processor "sees" the local file structure/namespace.

9front is full of "free carrots". If you want a remote vpn, just import
another servers /net ontop of your process namespace and you're all set.

~~~
so_tired
9front looks so very interesting. Thanks

~~~
chx
It also stays at "looks very interesting", it's not a usable OS on a normal
machine for everyday tasks.

------
toomuchtodo
This is cool. I can mount most of the Internet Archive as a directory with
this. Would love this to grow into what your consumer cloud storage services
are but for torrents (a global file system you can mount locally, using
torrent data as inodes, with visual representations for availability status
similar to the Dropbox green check boxes). Perhaps even with support for web
seeding from a storage system of last resort (Backblaze, S3, etc).

~~~
rakoo
I know it's not good spirit, but mounting torrents into the filesystem has
been done a few times already, there are many projects that do that already.

Also, it's not really advertised, but S3 can act as a torrent node:
[https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.ht...](https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html)

~~~
toomuchtodo
The cool part is you can have S3 serve a torrent, and then disable access to
the object with permissions once it reaches an adequate seed level. You’re no
longer charged for outbound data, but S3 will continue to act as a tracker for
the swarm.

~~~
MR4D
Just checked Amazon's S3 page linked to by the PP.

Question - how do the following "Notes" that Amazon has on it's page affect
that?

"Amazon S3 does not support the BitTorrent protocol in AWS Regions launched
after May 30, 2016."

"You can only get a torrent file for objects that are less than 5 GBs in
size."

~~~
res0nat0r
Likely just can only seed/torrent via older s3 regions is all.

~~~
toomuchtodo
Blessed be thy us-east-1.

------
Randor
Well,

I wrote a device driver for Windows 8 that would mount a .torrent file as a
virtual disk way back in 2011. We pitched the intellectual property around to
several large companies. The only interest we had in the technology was video
game companies... where we could mark file pieces as 'high priority' and allow
the player to play the game even when the game disk was partially downloaded.

The IP was sold to a company developing their next-generation console video
game system as part of a 24 million dollar package and never saw the light of
day. I suspect what they really wanted was the associated patents.

------
gruez
I wonder how intelligently this handles piece download order. Torrent clients
usually use rarest-first download algorithm, but if you've mounted a movie and
you're playing it with VLC, you'd probably want it to download sequentially so
there's as little time spent buffering as possible. You could download the
pieces on demand (whatever piece the application is trying to read), but I
suspect that will provide a suboptimal experience, especially if your download
speed is barely above the bitrate of the movie.

~~~
calvinmorrison
libtorrent has repeatedly ignored requests for sequential download because of
the silly spec. There's a patch out there that quite easily enables it. It's a
bit absurd to me because there's a very good use case for sequential download
and it's not clear there are any actual side effects.

~~~
Hamuko
Doesn't sequential downloading degrade the swarm performance?

~~~
calvinmorrison
Maybe, but probably less relevant for seeds that are not new. Also if people
are streaming via torrents it's a net benefit if all peers have early bits not
just random data. In short I do not have empirical evicidence but I do believe
the decision to do so should not be made by a library and I also believe the
intial assumption that random is better holds true today.

------
fuzzybear3965
Can someone explain why I might use this instead of downloading a select group
of files using a torrent file/magnet link? Is the advantage that it
selectively downloads files on an as-needed basis? Or, is the product niche
somehow related to the fact that the torrent file/magnet link is mounted as a
filesystem, instead of only as a collection of files and folders?

~~~
kchr
The latter - it lets you navigate the file structure of a torrent as well as
read (stream) file content using any of your usual applications, without
having to download the files first.

~~~
fuzzybear3965
Streaming. I thought about this in the interfile sense, but not the intrafile
sense. This makes sense.

------
jtl999
Okay. I'm afraid to ask. What happens in the case of a) a torrent with no
seeders or b) Has seeders with enough data to get a directory listing but
either the seeders "drop off" afterwards or are incomplete

Other experiences with network filesystems (CIFS/NFS) on multiple platforms
sometimes have things getting hairy if the connection to the server drops
while the filesystem is mounted or in use

~~~
Randor
Hi,

A quick glance at
[https://github.com/johang/btfs/blob/master/src/btfs.cc](https://github.com/johang/btfs/blob/master/src/btfs.cc)
on line 190 reveals that no seeders would result in an infinite wait state:
pthread_cond_wait(&signal_cond, &lock);

The author could mitigate this by automatically failing any reads where there
are zero seeders and/or if the peer piece map does not contain the block.

------
yunruse
This is a brilliant concept of a tool. I’ve never worked with monorepos
before, but something like this with rw capacity would be an absolutely
amazing pipeline, if every developer in the office and beyond are full nodes
in the system.

~~~
david_draco
Torrents can't write though, or are you referring to a shared filesystem like
NFS?

~~~
adtac
Let's create a chain of immutable blocks with back pointers on top of this to
create new versions of the data. Maybe even call it a... blockchain.

------
anacrolix
See
[https://github.com/anacrolix/torrent/tree/master/fs](https://github.com/anacrolix/torrent/tree/master/fs)
and
[https://github.com/anacrolix/torrent/blob/master/cmd/torrent...](https://github.com/anacrolix/torrent/blob/master/cmd/torrentfs/main.go)
for a Go FUSE implementation of this. (Looks like it was started in early
2014.)

------
shric
Neat idea, but it seems like abuse of the torrent protocol. If peers don't
grab chunks randomly then it partially defeats the purpose of bittorrent.

~~~
oh_sigh
Every torrent client I've used recently has an "enable sequential download"
option. The only torrents this would realistically affect are maybe ones where
the seeder may go offline when a bunch of clients have been doing sequential
downloads and no one has the final parts of the content.

~~~
keyserzose
Which would be the same as no one would have those random parts.

------
eqtn
BTFS with mutable torrents would make a nice piracy suite

~~~
rakoo
Mutable "torrents" with fuse access exists, it's called hyperdrive
([https://github.com/hypercore-
protocol/hyperdrive](https://github.com/hypercore-protocol/hyperdrive)). See
the daemon in particular ([https://github.com/hypercore-protocol/hyperdrive-
daemon](https://github.com/hypercore-protocol/hyperdrive-daemon))

~~~
namibj
No. Hyperdrive is not like the official BEP-conform "mutable DHT item"-based
updatable torrents.

They essentially say "download the most recent version signed by this pubkey",
which ia notably distinct from hyperdrive's append-only log.

~~~
rakoo
"download the most recent version signed by this pubkey" is literally how
hyperdrive works.

The append-only log is hypercore, one of the building blocks of hyperdrive.
Hyperdrive is a layer above and gives you a view of how the archive is at any
point in time, with preferential access to the latest version.

TBF what I understood from the request of "mutable torrents" wasn't the
literal feature, but a way to have decentralized nodes exchanging updatable
content. Bittorent with the Mutable Torrents BEP does work, but it's nowhere
near as developed as what hyperdrive is _today_. And as sad as it may sound I
don't think it will ever be.

~~~
namibj
Yeah. My point was, that BEP 46[0] doesn't have any archive/log functionality.

[0]:
[https://www.bittorrent.org/beps/bep_0046.html](https://www.bittorrent.org/beps/bep_0046.html)

------
inyorgroove
Tools[1] like this have existed for at least 6 years. Don't know if the older
tools still work though.

[1] [https://github.com/mafintosh/torrent-
mount](https://github.com/mafintosh/torrent-mount)

------
ngcc_hk
If you make your own torrent files, how to prevent it accidentally share with
the world? Is there anything in the torrent files that can prevent this?
(Program and configuration can make mistake.)

~~~
anacrolix
Yes, there are BEPs that describe not announcing or accepting connections in
certain circumstances. There are also peer discovery mechanisms that are
designed to work on local networks. Of course none of this matters if you mess
up (and someone externally cares enough to take your data), or if your client
doesn't support any of this stuff.

------
torrmentor
Why not torrentfs? What are the needs for obfuscation or association with
btrfs ('better try remaking fs'? )

~~~
w4hirelife99
“Why this project updated yesterday and not the project last updated 3 years
ago?”

Unless you mean a different one since you didn’t link source

~~~
JoshuaRLi
Maybe
[https://github.com/anacrolix/torrent/tree/master/cmd/torrent...](https://github.com/anacrolix/torrent/tree/master/cmd/torrentfs)?

------
grawprog
This looks pretty awesome. Does it work with private trackers?

~~~
pxx
Err. Why would it not?

edit: more constructively, this seems to just use libtorrent in the backend,
which supports the 'private' flag
[https://www.libtorrent.org/features.html](https://www.libtorrent.org/features.html)
. Though even if you didn't, it would still 'work' in the sense of being able
to download chunks.

~~~
dewey
It also depends on which client ID it exposes and if that one is whitelisted
on trackers. But that's not really a technology problem.

------
Giorgi
Cool but can't really see any practical use of this...

------
diericx
This is freaking awesome!!

------
waynesonfire
Wow, what a great concept.

------
6510
Great stuff, thanks!

------
yjftsjthsd-h
Well that is neat, albeit an unfortunate name (1-char off from BTRFS). Wonder
what performance is like - does it try to preemptively download? Cache? I
would expect it to take a while to run `cp ~/mnt_btfs/big_file ~/Downloads/`,
but how about `find ~/mnt_btfs`? If it's performant, it could be really
nifty:)

~~~
shuringai
very unfortunate indeed as there's another project with this name:
[https://github.com/TRON-US/go-btfs](https://github.com/TRON-US/go-btfs)

I dont know whether the maintainer is a supporter of crypto (as in tokens) but
having a common name with a Tron product (or cryptocoins in general) can
backfire.

~~~
heavyset_go
It looks like the BTFS in the OP was the "original", with its first commit in
July of 2015. Could be wrong, because the GitHub interface doesn't make it
easy to get to the beginning of the 13k commits for go-btfs.

~~~
jrimbault
9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac 2014-06-26 Big things have small
beginnings.

(I have an alias specifically for finding that out)

Personnally, I don't care about any of the crytpo-mining-coining and I think
the two are sufficiently distinct.

~~~
lgierth
That's when go-ipfs was started - see my other comment for more info

