
Using BitTorrent with Amazon S3 - folz
http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html
======
RKearney
I'm fairly certain this has been a feature of S3 since its launch in 2006.
Here's as far back as I could find on archive.org[0] it's May 24th 2006 and
the URL indicates the documentation was published on March 2006.

[0]
[http://web.archive.org/web/20060421112025/http://docs.amazon...](http://web.archive.org/web/20060421112025/http://docs.amazonwebservices.com/AmazonS3/2006-03-01/)

~~~
cfaulkingham
It was there since the inception 03/14/2006 to be exact.

[https://noisemore.wordpress.com/2006/03/14/amazon-s3-has-
bit...](https://noisemore.wordpress.com/2006/03/14/amazon-s3-has-bittorrent-
support/)

~~~
rmetzler
Yes, and you'll find this date in the top left corner of the linked AWS API
documentation.

------
__Joker
Anyone downloading software or any content over a slow and flaky network would
question why people don't use torrents for distribution with some "permanent"
seeder.

I really had a horrible experience sometime back, while downloading a software
of around 2 GB. Where a network would die down and chrome would discard the
partial download.

~~~
estsauver
Chrome's downloads can actually be resumed using the wget --continue flag I
believe.

~~~
petemc_
That depends on a few things, such as the file still being at the previous
url, you still having access to the file, etc.

------
imrehg
Just moved a bunch of our company's large downloads over to S3 recently
(mostly BSP releases and drivers), and this would be awesome for that 100MB -
4GB package size range that they have!

Tried it with a few clients: Transmission does not seem to work, though (both
web client, and GTK): "Tracker gave HTTP response code 404..."; rtorrent seems
to be able to download (but looks like no upload?); deluge downloads well and
also kickstarts the transmission clients.

I wonder what's unacceptable in the Amazon infrastructure for transmission...
And the 80kb/s max seed rate mentioned in the comments might be a showstopper
already.

~~~
e40
A couple of months ago I tested with Transmission (on a Mac) and it worked
fine. The problem from my point of view was that the S3 side only seeded at
50-100 KB/sec, but if I downloaded directly from S3 I could pull 2MB/sec.

Unless you have a bunch of seeds, that are essentially permanent, it's pretty
useless.

~~~
Havoc
Might still be useful as a seeder of last resort to borrow done finance
terminology. Ie back a large number of say community nix isos with it.

------
jberkel
I'm currently using this to build a decentralized distribution system to ship
relatively big database files to iOS clients without having to go through S3
all the time. It works quite well but some providers throttle the bittorrent
traffic (or worse, throw in RST packets) so I'm not sure how well this will
work in practice. Also note that the S3 default seeders only upload at around
80kb/s, so you'll always need at least one external seed to get good
performance.

~~~
fit2rule
May I ask how you're doing torrent on iOS? The reason for my interest is that
I recently ported ipfs ([http://ipfs.io](http://ipfs.io)) to iOS and am
interested in any experience you've had with running these styles of protocols
on iOS clients in the field ..

~~~
jberkel
It's just libtorrent packaged up as iOS8 framework + a convenience API,
available here:
[https://bitbucket.org/jberkel/torrada](https://bitbucket.org/jberkel/torrada).
Still WIP and not used in a production app, but hopefully soon.

It works well if you're on a non-throttled network and has support for local
peer discovery which is great for testing (install from a local torrent seed,
or another device). Apart from bandwidth saving it's also very convenient to
let libtorrent handle consistency checks and resuming of transfers.

ipfs looks interesting, have you published your port?

~~~
fit2rule
Thanks for the details about torrada, I'll check it out!

Seems that for peer apps on mobile, we need to make it easy for the user to
understand their bandwidth-usage - indeed in my testing I never run the app
where I don't have solid, unlimited Wifi. For these classes of apps, I think
the presentation of this problem to the user is going to be important.

>ipfs port

Its more of a harness than a port - basically, its possible to use Go on iOS,
and so I have a project set up to include Go, cross the bridge between Obj-C
and Go runtimes, and so on .. and, then I've built a Go app with ipfs, running
on iOS. There aren't any features yet - but the harness is done, so, TODO:
write Go, use ipfs framework, interface with iOS-GUI normatives, etc.

When I get actual ipfs functionality coded, I'll put it up on a repo .. too
much futzing around is required right now to see anything actually
functioning.

~~~
jberkel
Ah, I had a look at goios, I can imagine that it is a lot of work, the
integration looks quite complicated and far from being finished, good luck.

About sharing bandwidth, was thinking about only enabling when the user is on
wifi, but I think I'll use torrent more as a distributed mirror network than a
"real" P2P system, which is difficult in a mobile context. People can easily
"donate" bandwidth by seeding the data files needed for the app (using a
separate torrent client though, not the app itself).

~~~
fit2rule
Actually, I got everything working, and can build and include an ipfs project
in the iOS bundle .. I just don't have a UI to wire it up to the Go-side
functionality. But in case you're interested you can have a look here:

[https://github.com/seclorum/ios-go-ipfs.git](https://github.com/seclorum/ios-
go-ipfs.git)

Might have to change a few paths in the build-script, but it'll give you an
idea (if you're interested) how to go about building Go apps on iOS. Turns out
its not that hard! :)

>Sharing bandwidth

Yes, I see the same sort of conclusion from my perspective of wanting to put
ipfs on iOS - its really only as a way of committing content to the mesh when
the User wants to serve - i.e. idle times connected/charging on local Wifi ..
I think my next step will be to wire up the camera and a few fields to be used
to create a publishable set of content ..

Anyway, pull requests welcome!

------
byaruhaf
You can get torrent only for objects that are less than 5 GB in size. That's a
bit limited.

~~~
Jedd
I wouldn't think it's hugely limiting. In practice I'd speculate that the
majority of torrents out there are either around 180MB, 370MB, or 1800MB

If (for whatever reason) you choose to use S3 as your tracker, then you just
need to commit to breaking your >5GB content lumps up into multiple pieces.

~~~
reitzensteinm
You're probably right that the majority of torrents on the internet line up
with the common TV show sizes, but it doesn't at all follow that users of this
service will follow that distribution.

I have 60gb files in S3, and saturating available bandwidth to download them
in a reasonable time frame if they're not in the same country is actually a
bit of a challenge.

Bit torrent would be one of the fastest and fault tolerant ways to retrieve
the files, so it not being available stings a bit.

While I'm trying to download large files on consumer internet, I'd imagine
that huge files between geographic locations and server grade connections
would face a similar problem.

------
userbinator
A bit OT but...

 _Javascript is disabled or is unavailable in your browser. To use the AWS
Documentation, Javascript must be enabled._

...really? Their documentation is perfectly readable without it. All the links
are real, bookmarkable links; even the buttons for the PDF, forums, and Kindle
version work.

~~~
hvm
I suspect there's a default 'javascript required' message in a file that's
included with all pages.

------
zurn
Are there easy to use multiplartform BT libs around that would let you use
this just for high bandwidth, nearby peers? Eg filter peers by latency as a
first pass.

~~~
colechristensen
Why would you care to exclude peers slow or far? What benefit would you
expect?

The torrent library to use is libtorrent.

[https://github.com/rakshasa/libtorrent](https://github.com/rakshasa/libtorrent)

~~~
zurn
S3 will normally provide ample bandwidth, but in some situations there's a
local shared downlink that is the bottleneck. For example in asset heavy
multiuser VR or game apps used by many people in same location. By only
including the nearby peers I get all the benefit but minimize the risk of
people accusing me of clogging / mooching off their uplink.

------
gwern
I've used this in the past to distribute scrapes of black-markets (eg
[https://www.reddit.com/r/DarkNetMarkets/comments/2zps7q/evol...](https://www.reddit.com/r/DarkNetMarkets/comments/2zps7q/evolution_forums_mirrorscrapes_torrent_released/)
&
[https://www.reddit.com/r/DarkNetMarkets/comments/2zllmv/evol...](https://www.reddit.com/r/DarkNetMarkets/comments/2zllmv/evolution_market_mirrorscrapes_torrent_released/)
). It saves a lot of bandwidth (which is not all _that_ cheap on S3) and so
far the <5gb restriction hasn't been an issue.

------
ShirsenduK
Wow! This may lower the entry barrier for a new movie/TV streaming company.
They can lower their costs by offsetting the bandwidth to users. Legal torrent
streaming might have just got the boost it requires.

~~~
corin_
This isn't a new feature, it was available at least in 2012, possibly earlier.

~~~
rmc
I read about it in 2010.

~~~
hrrsn
I used it in '08-09.

------
NKCSS
Could be cool to get better speeds, if they have multiple seeders for the
bucket.

~~~
derefr
Honestly, I'm surprised this is built into S3, rather than CloudFront.
Ideally, you want your object replicated to a bunch of edge nodes, which then
all simultaneously join into a single swarm as seeds.

~~~
ianlevesque
It predates CloudFront.

~~~
rossk
Pretty sure it was part of the initial, day one S3 release.

------
dimino
I once read an article or email about why web browsers can't/won't implement
BitTorrent clients -- one of the biggest barriers IMO to making BitTorrent
more ubiquitous for file download/upload is the fact that I can't just click
on a link and have my browser handle the rest, like I can with an HTTP/FTP
link.

Does anyone have a link that explains why browsers won't do this, or at least
a brief explanation?

~~~
ianlevesque
Opera used to do this, so there's no particular reason specific to BitTorrent
that the others do not. Probably the usual questions of priority and bloat.

[http://help.opera.com/Windows/9.00/en/bittorrent.html](http://help.opera.com/Windows/9.00/en/bittorrent.html)

------
rctay89
Apart from myself, anyone else hoping to see _downloading_ straight into an S3
bucket? (Seeding from a bucket has been around for some time, you could
imagine my disappointment when clicking the link)

------
Sami_Lehtinen
It still needs BitTorrent client. How about doing exactly the same but using
WebRTC which would work without additional client application.

------
tedchs
Uh, this feature has been available for years, hardly "news".

------
fiatjaf
I'm dreaming about using IPFS on S3. Will it be possible?

------
cosql
awesome

