
Show HN: Bt – BitTorrent library in Java 8 - atomashpolskiy
https://github.com/atomashpolskiy/bt/blob/master/README.md#----bt
======
atomashpolskiy
There are many great BitTorrent clients out there, yet I've made my own and
willing to share and attract collaborators!

~~~
yodsanklai
Just curious, how many hours do you think you've spent on this project?

~~~
atomashpolskiy
I'd say something in the range of 500-1000 hours.

~~~
olalonde
That's less than I was expecting. Did you study any existing library to get up
to speed?

~~~
atomashpolskiy
No, after I had studied BEP-3, wiki and and a few papers (most importantly
"BitTorrent economics"), I had a very clear understanding of what needs to be
done.

------
jph
Thank you, your work is terrific! Good docs, good code, and I especially like
the ability to create custom code to write to alternative storage.

~~~
atomashpolskiy
Thanks!! Yeah, I was pursuing such design that would allow to
replace/customize most of the parts, including the internals. As a bonus, it
makes it easier to continue working on the library as the codebase grows (now
nearing 25K lines of code)

------
gigatexal
Man this looks slick, going to give it a shot. Love the ability to be able to
download a magnet link from the CLI and not worry about what some fancy GUI is
doing in the background cuz I am paranoid.

~~~
atomashpolskiy
Thank you! Make sure to install
[http://www.oracle.com/technetwork/java/javase/downloads/jce8...](http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html)
to allow 160-bit cryptography and use -e flag to be completely safe!

I also have a simple shell script locally, that wraps the java command, so I
have to just type `btmagnet AF0D9AA01A9AE123A73802CFA58CCAF355EB19F8` to
download to a pre-determined location (hardcoded in the script). I wonder if I
should share it on the CLI README page

~~~
sarnowski
> Make sure to install
> [http://www.oracle.com/technetwork/java/javase/downloads/jce8...](http://www.oracle.com/technetwork/java/javase/downloads/jce8..).
> to allow 160-bit cryptography

Or don't use the Oracle proprietary distribution but the OpenJDK of your
operating system which normally doesn't exclude proper crypto algorithms.

(be aware that the JDKs downgrade silently if certain libraries are missing in
your host)

~~~
atomashpolskiy
That's some really useful information, thanks! Need to add this to the readme

------
zitterbewegung
It looks great but the top GIF makes it seem like you are offering a command
line BT client. If I were you I would have it slowly scroll through the code
of the client in the GIF or put a few frames of code of the client and or some
compiling and then show the current top GIF.

~~~
atomashpolskiy
Oh, I didn't expect this :) Sounds good, but I'm not sure I have patience do
this right, it's such a pain to record gifs - I always misspell or misclick
something, and by take #47 go completely nuts

~~~
deft
Try using asciinema. It records your terminal session and works so well. Of
course typos and such are still an issue...

------
yodsanklai
Nice work. I'm wondering, how would you test/benchmark such an application.
Besides manual testing on a variety of torrents, how would you do regression
test there? same question for benchmarking. How to compare this work with
another client?

~~~
atomashpolskiy
I have several hundred unit tests and also a bunch of integration tests. While
UTs usually test API of individual classes, each of the ITs creates a swarm of
local peers and launches a torrenting session with certain conditions: seeders
to leechers ratio, downloading from .torrent file vs using a magnet link, PEX
enabled/disabled, encrypted vs raw message streams, etc.

Benchmarks (like in libtorrent) is something I'm looking forward to, but the
project is still in the early stage -- there's a lot to be done with regards
to optimization, i.e. switching to using NIO selectors, adding a caching
layer, etc. That's a lot of work, and that's why I'm looking for
collaborators, before investing more time into performance-tuning.

~~~
silotis
I would advise against implementing a custom cache for piece data. Caching
file data in-app is counterproductive because it wastes memory duplicating
data which the OS already holds in its page cache. Instead memory mapped I/O
should be preferred.

~~~
atomashpolskiy
It makes sense, what would you advise in case of big files that don't fit into
memory (or when it's undesirable to use big amounts of RAM, and speed can be
sacrificed)? Maybe caching should be disabled by default but with the option
to turn it on if the user wants it?

~~~
the8472
you can memory-map files in chunks and keep references to the mapped buffers
or evict them once a certain limit is reached.

note that mapped buffers are not necessarily backed by RAM, they act more like
swap space, except it goes straight to the original file instead of a swap
partition. the downside is that anything reading from the buffer can
unpredictably incur IO overhead if there's memory pressure and the OS decides
to not back them with memory. java doesn't have an api to check residency of
buffers. a possible workaround is to first queue them up on a separate thread
and forcing them in before using. if they're already there it should have low
latency. if they aren't then only the victim thread eats the IO penalty.

~~~
atomashpolskiy
So, something like LRU cache of MappedByteBuffers, one buffer per chunk (i.e.
"piece")? I wonder what queueing strategy would look like, just enqueue a
couple of subsequent pieces when one is requested and loaded?

~~~
the8472
Start simple, then heap on optimizations if there are latency or throughput
issues.

------
djhworld
Really like the approach around using Guice as a central point for hooking
your own code into the system.

Will look forward to reading through the code later!

~~~
atomashpolskiy
Thanks! God bless IoC, it saves my ass each time I make some silly design
choice

------
styfle
I'm curious, it looks like the screenshot shows the author is using a Mac. But
Mac OS hasn't shipped with Java in several years. So why Java? Is it simply
familiarity, or are there other reasons?

~~~
pvg
Most OS's don't ship with a C compiler, go, python 3, rust, arc, etc, what
does that have to do with anything?

~~~
weberc2
You're conflating developer tools with a language runtime. Every popular OS
ships with everything necessary to run a C, Go, or Rust binary. Python 3 and
Java both require a separate runtime to be installed, but I will say that
every Java CLI I've used has had terrible startup performance.

~~~
atomashpolskiy
I'm using Lanterna, a Java library for creating text-based terminal GUIs
([https://github.com/mabe02/lanterna](https://github.com/mabe02/lanterna)),
and the startup is 1-2 seconds. Can't recommend it enough

~~~
weberc2
Good to know, but 1-2 seconds is still very slow compared to a native app. I
have an 8MB Go CLI application that can startup and print its help in .05
seconds, and even that is probably slow compared to a C or Rust equivalent.

~~~
pjmlp
Any savy Java developer knows how to AOT compile to native code, if that
really matters.

There are plenty of JDKs that support it, and even Oracle is finally adding
support for it with Java 9, initially only for Linux x64.

By Java 10 timeframe no one that only knows about OpenJDK and not the several
other JDKs can state that Java starts slow vs C, because both will be AOT
compiled to native code, loading native .so files.

~~~
readittwice
From what I've seen Java/JVM AOT won't produce binary executables but shared
libraries that will be loaded on startup by the JVM. When I first read about
Java AOT-compilation I expected it to produce real executables that's why I
think this is important to mention. But please correct me if I am wrong. So if
this is true even with AOT compilation you still need the JVM as dependency
(although I think you can get a minimal JVM in under 10MB). It also seems that
the AOT compiled code needs to be recompiled on Java updates. Although I quite
like Java and the JVM, I still wouldn't choose Java for writing simple command
line tools.

Nevertheless it is quite useful for improving startup for tools like Gradle or
server applications. But sure, I also don't expect Java startup performance to
be a problem in the mid- to long-run.

~~~
pjmlp
Correct, but that is no different than having to ship glibc.so or MSVCRT.dll
alongside your application, specially if you want to be certain which version
gets used.

This is only relevant to Oracle's implementation though.

Other JDKs do support static linking, as long as you restrain yourself from
features that could require dynamic compilation.

There was a talk at JVM Languages Summit about the current state of AOT
compilation regarding Java 10.

[https://www.youtube.com/watch?v=n5DCg6M2MDM&index=11&list=PL...](https://www.youtube.com/watch?v=n5DCg6M2MDM&index=11&list=PLX8CzqL3ArzXJ2EGftrmz4SzS6NRr6p2n)

~~~
weberc2
I would rather ship glibc (8MB) or musl (.5MB) than the JVM (100s of MBs?),
but it's not necessary given that nearly every target already has a libc.
Also, Go programs don't depend on libc, and C and Rust can probably statically
link against some libc in a pinch.

~~~
steveklabnik
Rust can use musl, yes.

------
segmondy
Amazing work, how long did it take to build? Where you working on it full-time
or part-time?

~~~
atomashpolskiy
Thank you! Calendar year, in the evenings and on weekends, with sprints of
several weeks and a month or two of rest/AFK after each sprint to keep sanity
(I have a full-time job, a wife and a 3 year old kid). Happy that I had the
persistence to keep working on the project

~~~
segmondy
Very impressive! Thanks for sharing, it's encouraging to me and I hope to
others to know that with proper time management we can put out amazing work.

------
intellix
I remember using Java based client Azureus and the memory usage was huge.
Overnight it would leak to about 1Gb. Later uTorrent appeared and it used
2mb... Forever

I guess nothing to do with this but was triggered by two combined words:
Torrent and Java

------
amzans
It's great to see projects like this on HN. Congrats man, well done!

------
nunobrito
Hello, thank you for writing the library.

On my context it is useful for downloading large files without depending on a
single point of failure or fixed location.

Have a question in regards to the example syntax:

client.startAsync(state -> { if (state.getPiecesRemaining() == 0) {
client.stop(); } }, 1000).join();

This is a feature from newer Java syntax that I'm not yet familiar enough with
lambdas to understand in full what is happening.

Perhaps I can ask for an example that replaces the "->" into the older Java
syntax? I know this would likely come at the cost of a few extra lines, it
would certainly ease a lot for many developers looking at the example. Many
thanks.

~~~
train_robber
In the older syntax, something like this:

    
    
      client.startAsync(new Consumer<TorrentSessionState> {
    	void accept(TorrentSessionState state) {
    		if (state.getPiecesRemaining() == 0) { 
    			client.stop(); 
    		}
    	}
      }, 1000).join();

~~~
nunobrito
Thank you. Really useful.

------
wodencafe
Wow thanks man! This can be useful in a variety of circumstances!

------
ikurei
Not very familiar with the Java world yet. I've heard Android now supports at
least some features from Java 8, but would this library work in an Android
app?

~~~
atomashpolskiy
I guess not. There is a ticket for backporting this project to Java 7 or
RxJava
([https://github.com/atomashpolskiy/bt/issues/7](https://github.com/atomashpolskiy/bt/issues/7)),
but I personally think it's not worth it.

------
tonyrice
It's amazing how this made it to the front page of HN and I had no idea until
now. I actually discovered this by googling a couple hours ago :)

------
flarco
Wonderful Lib!

------
alexanderbisc
It's possible in Azureus, but it might be a lot of effort to turn the
functionality into a library.

