
Show HN: Magnetissimo – Self-hosted torrent search engine written in Elixir - sergiotapia
https://github.com/sergiotapia/magnetissimo?utm=hn
======
nikcub
I'll be trying this out, it could work well with Cloud Torrent. I've largely
moved away from Torrents tho and switched to using NZB's and apps that
automatically download tv shows and films[0]

Here is my stack:

1\. NZB downloader - NZBGet - [http://nzbget.net/](http://nzbget.net/)

2\. TV Shows - Sonarr - [https://sonarr.tv/](https://sonarr.tv/)

3\. Films - CouchPotato - [https://couchpota.to/](https://couchpota.to/)

4\. NZB Search - NZBHydra -
[https://github.com/theotherp/nzbhydra](https://github.com/theotherp/nzbhydra)
(I contribute to this project)

5\. Plex - [https://www.plex.tv/](https://www.plex.tv/)

Add Torrent support:

6\. Client - rtorrent -
[https://rakshasa.github.io/rtorrent/](https://rakshasa.github.io/rtorrent/)

7\. Web Interface - ruTorrent (warning: PHP) -
[https://github.com/Novik/ruTorrent](https://github.com/Novik/ruTorrent)

8\. Proxy private trackers to XML-RPC - Jackett -
[https://github.com/Jackett/Jackett](https://github.com/Jackett/Jackett)

9\. Cloud Torrent - web interface - [https://github.com/jpillora/cloud-
torrent](https://github.com/jpillora/cloud-torrent)

Additional:

10\. Plex stats, analytics and user management - PlexPy -
[https://github.com/JonnyWong16/plexpy](https://github.com/JonnyWong16/plexpy)

11\. Plex request - allow users to request content -
[https://github.com/ngovil21/PlexRequestChannel.bundle](https://github.com/ngovil21/PlexRequestChannel.bundle)

Hardware is a pair of HP Proliant Gen8 microservers, Ubuntu 14, Docker, nginx
and LetsEncrypt. There is no real easy way to set this all up, you have to do
each part of the stack yourself (a docker-compose file would go a long way to
simplifying it)

[0] I spend over $200 a month on content subscriptions so I don't feel bad
about utilizing the conveniance of NZB downloads + Plex

~~~
problems
I do the same, very similar setup actually, but I still wind up using torrents
when someone wants older stuff.

Unfortunately you get mixed results in terms of how things are labeled on
public trackers and I haven't had a private tracker account since what.cd
died.

I'll have to try out a few of these torrent search tools and see how it works
out, but most private trackers didn't want to just take cash and seemed to
want you to jump through hoops instead, so unless I can buy accounts via other
means I'll probably just stick to public.

~~~
ValentineC
> I'll have to try out a few of these torrent search tools and see how it
> works out, but most private trackers didn't want to just take cash and
> seemed to want you to jump through hoops instead, so unless I can buy
> accounts via other means I'll probably just stick to public.

I think most private trackers are trying to avoid developing a reputation like
that of IPTorrents [1], which is notorious for allowing people to "donate" for
accounts.

Most private trackers also explicitly forbid members from selling invites for
money, and, more recently, giving them to strangers.

[1]
[https://www.reddit.com/r/trackers/comments/4xta11/so_lets_ta...](https://www.reddit.com/r/trackers/comments/4xta11/so_lets_talk_about_iptorrents/)

~~~
problems
Yeah, looks like IPT and ultrahdclub might be worth a purchase though.

~~~
rapfaria
That is until they disable your account, and offer to enable it for another
$30. Visit /r/trackers and there's probably a thread about IPT.

------
sergiotapia
Hope you guys like this and use it! If it's hard to use, tell me. My goal is
to make this simple to run anywhere. Part of that will be to use Distillery to
create compiled executables.

I rewrote this after an initial version that used Redis as a queue to process.
Asking end users to install Redis was a step too far I think, so I stepped
back and thought about how to solve this without external dependencies.

I ended up with a simple Elixir and Erlang queue implementation. It works much
simpler, just as fast, and no freaky deps. GenServer starts up a worker for
each crawler, and it schedules work by itself. It's really strange to use a
language so complete, it kind of feels like cheating. Programming in easy-
mode.

Pull requests very welcome to the crawler folder, it's super easy to write a
crawler for your favorite site!

[https://github.com/sergiotapia/magnetissimo/tree/master/lib/...](https://github.com/sergiotapia/magnetissimo/tree/master/lib/crawler)

~~~
gyrgtyn
you could make it a nerves project too, maybe after you get rid of some of
those database dependencies.

It'll make you a ~100Mb linux distro that will boot straight into your app in
~10 seconds on a raspberri pi.

[http://nerves-project.org/](http://nerves-project.org/)

~~~
sauvage255
very useful project, thanks for sharing this!

------
fenollp
You should look for sitemaps. They are often listed in robots.txt. Will save
you lots of time!

These sites have sitemaps:
[http://torrentproject.se/robots.txt](http://torrentproject.se/robots.txt)
[https://www.torrentdownloads.me/robots.txt](https://www.torrentdownloads.me/robots.txt)
[https://thepiratebay.org/robots.txt](https://thepiratebay.org/robots.txt)
[http://goldtorrents.com/robots.txt](http://goldtorrents.com/robots.txt)
[https://bitsnoop.com/robots.txt](https://bitsnoop.com/robots.txt)

Enough with the hype: "high performance" What?? Your program is not
concurrent: you have one process per website and go through URLs one at a
time. You don't even use Bloom filters.

------
tmalsburg2
After many years, Bittorrent doesn't cease to amaze me. Such a fantastic
technology. Sad, though, that it is primarily known for illegitimate uses.
This gives it a bad reputation and makes it hard to promote Bittorrent for
more worthwhile applications. For instance, Bittorrent would be a perfect
platform for sharing scientific data but when I suggest that to colleagues,
they don't take it seriously because Bittorrent is perceived to be that tool
for downloading pirated movies. I wonder how this barrier for adoption in
academia could be overcome. Perhaps, with a separate network that is targeted
at academic uses (think alt-coins). Call this network ResearchTorrent and
researchers could use the technology without ever having to make contact with
the whole ecosystem around pirated material. Universities could support this
network without risking bad PR and getting in conflict with the music/film
industry.

~~~
jzelinskie
I work on a BitTorrent tracker on the side[0] that's used for deploying
software at a few companies, but has the unique feature of a middleware layer.
Right now, there are two different middleware that require a signature of the
infohash (files being shared) before allowing content to be shared. One is for
JWTs[1] and upstreamed in the project and the other is for a blockchain and
hasn't been upstreamed yet. Using features like this, they can run their
tracker on the public internet and not worry about people sharing illegal
content. I had never considered it, but this sounds like a great use case for
academia, too.

[0]: [https://chihaya.io](https://chihaya.io)

[1]: [https://jwt.io](https://jwt.io)

~~~
tmalsburg2
Very interesting. Does that mean that there would have to be a central
authority that has to sign new torrents? For the present purpose that may not
be ideal because this system wouldn't be decentralized, and one institution
(or a small set of institutions) would have all control. Ideally, there would
be some consensus mechanism that determines which torrents are bad and
shouldn't be distributed. Also, what happens when the key for singing torrents
is stolen? In the case of a company using such a system, that situation can be
handled but in a more decentralized system with an emphasis on long-term
storage the loss of a key may be a bigger problem.

~~~
jzelinskie
>Does that mean that there would have to be a central authority that has to
sign new torrents?

In this scenario, the source of truth is the location where your users are
acquiring their .torrent files.

>For the present purpose that may not be ideal because this system wouldn't be
decentralized, and one institution (or a small set of institutions) would have
all control. Ideally, there would be some consensus mechanism that determines
which torrents are bad and shouldn't be distributed.

Trackers are the centralized alternative to using the Mainline BitTorrent DHT.
The performance advantage significant enough they're still in common use after
the ubiquity of DHT support in client software. You could build a tracker that
serves peer data that it has cached from crawling the DHT, which is an idea I
don't think has been done yet. As for a consensus mechanism, that should
probably be done out of band and client software could be configured to query
it and react appropriately.

>Also, what happens when the key for singing torrents is stolen? In the case
of a company using such a system, that situation can be handled but in a more
decentralized system with an emphasis on long-term storage the loss of a key
may be a bigger problem.

With the JWT system, you can rotate keys, but it will break clients announcing
with the old key. Users would have to manually update all their torrents to
participate in the swarm again. This is a shame, but there is no official
signing scheme in BitTorrent. If the JWT system as chihaya has implemented
gathered traction, it could be updated such that clients could actually fetch
JWTs regularly from an auth endpoint and then it'd be possible to handle
rotation the same way as most OIDC implementations.

~~~
tmalsburg2
I like the idea of a tracker crawling the DHT. Research institutions could
sign their torrents with their private keys and the tracker only indexes those
signed with approved keys. Institutions register their public keys with the
tracker and when an institution loses their keys, only their torrents are
affected. Anyone can run such a tracker and there is no central authority.
Nice. Is it possible to store signatures in torrent metadata?

------
mrmondo
Looks really interesting, I love goal 1: no JavaScript - YES! Thank you! And
elixir is a very exciting language, especially to an ops person where when I
read elixir code it actually makes sense to me and seems elegant. Ignoring my
technical praise the objective of the product is solid, this is something I've
been interested in for a while, I have a solution for usenet but not so much
for torrents.

~~~
marvel_boy
Absolutetly. +1 for the "no javascript" goal.

~~~
mrmondo
I'm so sick of 'designers' thinking they're programmers and smashing out some
NodeJS or JavaScript heavy crap with no regards to performance or security.
Elixir seems to attract some smart minds that think 'big picture' \- obviously
I'm generalising here and I'm not stating that this particular project is
brilliant or anything but it is a trend I've observed.

~~~
huduo
NodeJS is a dependency in the usage instructions for Debian 7 and CentOS.

~~~
swiley
Ew, what for?

~~~
luckystartup
The creators of Phoenix didn't want to write their own code to manage
Javascript and CSS (e.g. rewriting the Rails "asset pipeline" in Elixir), so
they just integrated Brunch [1], which is built on Node.js and NPM. I think
this was an excellent decision. (Node.js is only needed during development and
deployment of your assets. It is not used in production.)

[1] [http://brunch.io](http://brunch.io)

~~~
dingdingdang
That is still pretty awful imo - think quite a few of the people considering
Elixir and Phoenix are doing so explicitly to get away from having to Node.js
close to their computers in the first place.

~~~
true_religion
I don't see the problem. Why not use Javascript toolkits to package, and
develop in Javascript?

I can understand languages not self-hosting their compilers, but I'd at least
like it to self-host its toolkits---bundlers, packers, preprocessors, etc.

------
brian-armstrong
When someone posts about a GPL violation, people are quick to decry it, and
rightly so. But this repo demonstrates, right in the first image, this tool
being used to violate copyright, and we applaud that. Why do we have this
double standard?

~~~
arianvanp
They're the exact opposite from my ethic point of view.

Gpl violation is a violation where the person didn't share information.

Torrentting 'illegally' is the act of sharing information.

If you're all for sharing an free flow of information.. I don't see how these
two conflict I my ethical point of view.

~~~
dibstern
In both scenarios the person's property is being stolen.

Their property might be in the form of information, sure.

But one product being more widely distributed does not make any ethical
difference. The fundamental is that the information/property has been stolen
in both cases.

~~~
cyborgx7
Copyright infringement has as much in common with stealing as it does with
sharing. Both of them are ideological terms to paint the action in a certain
light and neither give a complete picture.

------
yoavm
Looks like a useful tool, but I kinda wish this kind of programs had a CLI. It
seems to be doing one main thing (searching), and it could be super useful to
be able to pipe the results to a downloading program. Some programs can
benefit vastly from a GUI, this... I'm just not sure what's the point. Kudos
to the developer nevertheless for the tool itself and for the choice to go
with Elixir.

~~~
smcl
Sounds like you found yourself a wee evening task - hack a command line
interface for this tool :D

(edit: only messing - I was just thinking that if I wanted to learn Elixir
hacking on some existing project is a good way)

------
luckystartup
I might have to start an anonymous GitHub account so that I can contribute to
this. It would be fantastic if there was a way to start a download directly
from the web UI.

Many torrent clients can monitor an RSS feed, so that would probably be the
best solution. I would add the ability to create a user account, and each user
could set up a few RSS feeds. In my case, I would like to have one RSS feed
with the label "Movies", and one for "TV Shows". Then the main search page
would have buttons to add the magnet link to either RSS feed.

I already use [http://showrss.info](http://showrss.info), which provides an
RSS feed for all of my favorite shows. So it would be great to manage my own
RSS feed for movies and specific episodes.

~~~
the8472
> It would be fantastic if there was a way to start a download directly from
> the web UI.

There are browser addons that can send magnets and torrents straight to the
remote APIs of torrent clients.

------
listic
Can anyone suggest a good torrent tracker suitable to run a torrent site?
Opentracker [0] seems to be abandoned and the author doesn't reply.

[0]
[http://erdgeist.org/arts/software/opentracker/](http://erdgeist.org/arts/software/opentracker/)

~~~
the8472
opentracker is what is used by the high-throughput trackers. the protocol is
fairly simple and doesn't change a lot and the implementation is fairly
minimal, so they don't need to update the codebase often, it does it's job,
it's mature.

for smaller sites there are more specialized trackers that allow more control,
e.g. chihaya and gazelle.

------
kenning
Has anyone hosted this somewhere yet? I was hoping I could just run it locally
but you have to crawl all those sites every time, and it's not concurrent...

~~~
uhryks
> leveraging Elixir's GenServer and Erlang's BEAM VM.

> it's not concurrent

(。´･ω･)?

~~~
sergiotapia
It's concurrent in that multiple crawlers run at the same time. But I do not
concurrently crawl single websites. An initial version had that, and
Magnetissimo easily crawled 600 torrents every page 3 seconds (not hyperbole).
It just pounded websites and I didn't want that. It was bad karma.

------
dopkew
Are there other programs that offer this same function?

~~~
bharani_m
Not a self-hosted program. But, Skytorrents [1] was featured on HN recently
[2].

[1] [https://www.skytorrents.in/](https://www.skytorrents.in/)

[2]
[https://news.ycombinator.com/item?id=13423629](https://news.ycombinator.com/item?id=13423629)

~~~
patrickk
Another "inspirational" list thanks to the US government, from a HN comment[1]

[1]
[https://news.ycombinator.com/item?id=13237667](https://news.ycombinator.com/item?id=13237667)

[2] [https://ustr.gov/sites/default/files/2016-Out-of-Cycle-
Revie...](https://ustr.gov/sites/default/files/2016-Out-of-Cycle-Review-
Notorious-Markets.pdf)

------
giridhar50
very useful tool

