Finding this sort of thing always makes me stop short and realize, "Oh yeah, no matter how smart I think I am, there are people a million times more accomplished than me on HN." Not to make me despair so much as take care to avoid blathering the first thing on my mind before reading and considering what others have said.
The 'magnet URI' was merely the right suggestion, for a tiny sliver of shared convention, at the right time.
You can look at the original writeup – http://magnet-uri.sourceforge.net/magnet-draft-overview.txt – and find a goofy ad-hoc mechanism where local software interested in 'magnet' links listened for localhost web hits, in a certain port range, rather than fighting over the 'magnet' URI scheme handler. That could be seen as either a blathering kludge or a very early attempt at 'web intents'.
I don't know if any code other than my demo ever implemented that localhost-handler trick. But the main idea – of a generically-named URI that was descriptive rather than location-centric – had its own legs. It spread from software to software, even without any formal spec stewardship/revisions. A key step for its longevity, when 'magnet' was adopted to wrap the 'btih' identifiers for DHT-based torrent tracking, happened long after my attention moved elsewhere.
I'd like to think the continuing adoption was because the original draft, even though crude and incomplete, had the right mix of a familiar/defensible URI syntax, wide-open descriptor semantics, and a catchy name/icon.
It's basically a URI that contains the torrent hash enabling a client to find peers through a decentralized network such as DHT instead of requiring a tracker. I believe you can append trackers and other info to the URI as well, but I'm not sure about that.
The other two comments are (correctly) describing what a magnet link is in the sense of its implementation. On the off chance that you wanted a more abstract/basic description, a magnet link is defined by what you're linking to, as opposed to where it is.
This is kind of cool, because a file can change locations (how many times have you seen a dead link/File Not Found?), and with a magnet link, if you have a file that matches the description, you can be reasonably certain that you've found the right file (unlike the URIs you're familiar with, which can change the content stored at each location). On the flip side, actually finding the file from the description is a bit more complicated, but as you can see, it's still pretty easy for the user when implemented correctly.
It's better for TPB, because they're no longer telling you 'where' the file is, just what it 'looks' like. Legally, I'm sure that makes some difference.
AFAIK, it's a torrent for a torrent; a DHT implementation. The URI for the magnet link has all the information it needs to find peers and start downloading the data, the first datum of which is the torrent file.
Ups to you! I think (and hope) that we haven't seen the end of magnet yet. I wish the scientific community would start using magnet links for paper references, instead of archaic bibtex layouts combined with online publishers who charge 30 dollars without adding value.
When multiple downloads of the same file are available it's usually beneficial to see and select the one with the highest number of seeds. If DHTs become the norm, will it still be possible to obtain and use such data?
I think that when you use a magnet link it either includes the tracker (which is the part of BitTorrent that tracks the seeds) in the text of the link or uses a more sophisticated technology called a Distributed Hash Table.
> Perhaps even better, without the torrent files everyone can soon host a full copy of The Pirate Bay on a USB thumb drive, which may come in handy in the future.
I've been saying this for years: governments playing an arms race with hackers is like playing Whack-a-mole or cutting of heads of hydras. Every time you block one means of communication (starting in this case with Napster), a new, more decentralized, harder-to-combat protocol is going to emerge. Even if you're the MPAA and don't like copyright violations, you have to face that fact. That doesn't mean you (necessarily) have to throw in the towel and abandon the idea of copyright infringement altogether, but it does mean you need to start being creative instead of engaging in a direct legal-technological battle - an arms race.
I wonder how long it will take before the big players realize this and try to figure out a way to use this new playing field (the Internet) to their advantage, instead of trying to squelch any technological development so that they can cling to old models of payment and distribution. It was nice while it happened, but let's face it - we're past the point of no return. Even if they everyone hosted their own copies of TPB on their thumb drives and then they found some way to shut that down, I'm certain there'd be some hacker smart enough (like gojomo above) to come up with something that just makes things even less centralized, more difficult to track, and more difficult to shut down.
The RIAA and other major music industry players may have no future, but there is definitely a profitable future for a "music industry" populated by entities that are willing to adjust to the modification in distribution channels and consumption habits.
As it stands, the old-fashioned fogies are simply in denial and doing their best to prevent the marketplace from changing because they don't know how to use the new technology and they don't know how to succeed in the new marketplace. That doesn't mean that there's no future for the industry as a whole, it's simply the way entrenched players eventually die.
I'm with you on this one. It's less that the RIAA and major labels 'don't know how to use the new technology and...succeed in the new marketplace' so much as we're currently (still) in an in-between stage of market control.
Take for example Radiohead or Louis CK. Their previous successes allowed them to produce, release and distribute independently of big entertainment — but I don't think we're at the point where a small independent artist can effectively take their own music to market in a manner comparable to using a label. I don't have data to back this opinion, but high profile successes in independent distribution have been biased toward artists who were well-known and previously successful.
I don't think it is fair to compare Radiohead or Louis CK to that of a lesser known artist in this regard, because the bias is all in the media coverage.
Bands have been self releasing (outside of "big entertainment") since well before digital distribution, because signing to a label isn't always feasible (financially or creatively). Digital distribution makes just makes this even easier.
You only know about Louis CK & Radiohead because that is that the media covering. Digital distribution & artists self releasing material is happening and has been for a while, but no one knows because unless they are already into these lesser known bands & record labels. You have to think of it like this: "Lesser known artists can be more successful by releasing their own material digitally than they would by signing with a label." Or "This independent record label can lower their overhead and make things easier for their bands by embracing digital distribution."
" Internet is tightly controlled (which it cannot be.)"
Oh, it can be controlled alright. SOPA, PIPA today, tomorrow FIPA, DIPA and FuckInternetPA. One of them will pass, then more and more.
And then you're sitting on cable tv. Clicked that link? Invoice has been mailed to you. Clicked that link? House search. Clicked that link? Prison cell prepared. Thank you and good bye.
Oh, you will watch what you do online, you will not click on links without knowing before hand what is behind it. Good luck with that. You will not read just about anything, you will not write whatever you want, certainly not where ever. You will censor yourself before they come to censor you. They will produce and you will consume.
It is still fascinating how resilient not only Piratebay but also others such as Demonoid have become. A few years ago it looked like they were close to getting shut down. It is impressive how they are now able to withstand all the "forces" and Governments had to shift their focus towards the ISPs.
There are three main methods (in order of preference):
1. Use a cached list from a previous run. This is ideal in terms of load balancing, and also helps with bootstrap time if your IP hasn't changed, so all major torrent clients have such a cache.
2. Ask peers you're connected to via a traditional tracker and .torrent file. This is the next best thing, but can only be used if the first torrent you download is via a traditional tracker.
3. Use some bootstrap peers hardcoded into the torrent client. Has a single point of failure, but it works.
Each torrent client can use a different set of bootstrap peers, note - they're nothing special, just a DHT peer that has the bandwidth and CPU power to deal with bootstrap requests (which isn't a very high hurdle - you only get hit once, on the initial install, or if the client's been shut down for so long it no longer has any valid peers). Or the user might even supply their own, if need be.
Thanks for the information, this part was always a mystery to me as well.
To clarify, if someone has never downloaded a torrent before and starts with a rarely downloaded file, would the use of only magnet links mean they will probably not be able to find another peer? It sounds like unless you already have torrents downloaded, your list of possible peers to check for the file is very short.
If your client connects to just one peer you essentially gain access to the entire bittorrent DHT network. As a dramatic simplification, your request for the rare torrent would propagate until it found the peers seeding or downloading it.
But what if the users sharing the file are also connecting for the first time? It seems like if my friend wants to seed a file and I want to download it, it only works if we knows peers (who know peers who know peers who...) know peers of me. Is this correct?
In theory (assuming that there are links to everyone, and no way to have separate networks) (it's of course a theory only, but seems very likely that there are "massive users", who have files of "every kind", connecting the different groups), as long as you both connect to one who is connected, it will work.
Although this is quite the extreme case, and not really the purpose for Bittorrent to begin with, I guess. The main interest is load balancing of popular downloads, not finding the one person with the one file for you.
I've actually found torrents to be a great way to share large files with a friend. There's no invasive upload dialog or locking, no practical filesize limitations other than the free space on my friend's computer, you don't have to upload to a public-facing server before your associate can start downloading, port forwarding is handled transparently by UPnP, client software is user-friendly, easily available, and often already running anyway, and it's easy to scale if another friend wants a copy at some point in the future.
Of course, TPB switching to magnets doesn't really affect this. Torrent files are still used to provide information on the download, they're just distributed from within the P2P network instead of TPB itself. If you have a torrent file to share with an individual or small group instead of the whole world, there's no reason to stop using torrent files directly.
It is a good point, however, I think it would be celebrated by the target audience. People, or the "average person" who wouldn't generally visit TPB, or know what it is, will probably not really care about it's support, or even what SOPA is for that matter.
For those who visit/use TPB, I think it would be akin to a rally call.
So, what if you don't run the client on the same machine you're browsing TPB with, like people who have torrent enabled routers or VPS/seedboxes? I suppose you can copy-paste the link to your client, but clients which had automatic pick-up of .torrent files (like rtorrent) were nice because you could just drop them on a remote directory and have them be downloaded.
I wonder if I could write a small application just to download the torrent file from the magnet link/DHT to copy it to the remote server afterwards.
They announced that they were planning on doing this right when magnet links came out. I haven't used .torrent files since I discovered magnet links and I actually find they more convenient than downloading a torrent file.
I've been using mostly magnet links for the past year and haven't experienced any issues. If people understand that they work just like a normal link/torrent file, this won't make the process any more complicated. Hopefully this is another win in the long run, as links are harder to stop than files.
Only slightly as I understand it, since from a legal perspective the intent of someone providing providing links instead of files doesn't change, and that's what counts.
From a technical perspective, and as someone who doesn't understand how magnet works under the hood, I'm slightly concerned that DHTs might be easier to attack in an underhanded manner than an HTTP server would have been.
I can see this leading to a block on many sites on all links starting with the "magnet"-protocol, if it's not in place already. It also highlights the issue with SOPA further. Had that series not been allowed to be shared, Hacker News would be in big trouble for hosting the link.
Wow, I didn't see this before. I can understand filtering certain terms for court requests, but voluntarily? It looks like Google is attacking distribution methods of content with such broad filtering. Probably to make compromises with the anti-Internet activists. Now judges can interpret it to mean that Google can happily censor any results for certain words as it has been done in other anti-Internet regimes. What scumbags all round.
Note that this isn't filtering the terms for search, just filtering them for auto-complete. As in, if I type "lord of the rings" it won't suggest "lord of the rings torrent" even if that is the most common or one of the most common searches beginning with "lord of the rings."
If you actually type "lord of the rings torrent" and hit return, Google will still do that search and return relevant results.
They are doing this because for many pieces of popular media, the torrent search was fairly high in the list of most popular searches, and they don't want to be seen as suggesting that you download the torrent. Likewise, they won't autocomplete obscene or porn related terms, but will still do searches for them.
There is a difference between banning a term entirely from their search engine, and deciding that it's not something that you want to suggest to people who haven't requested it, to avoid suggesting illegal acts, avoid legal trouble, avoid offending people, or the like.
"This is something we looked at and thought we could make some narrow and relatively easy changes to our Autocomplete algorithm that could make a positive difference, Cano added."
They believe in filtering broad innocent terms in them as a solution to copyright infringement rather than saying "No no, we can't filter broad terms, especially innocent uses of it. Not only does the language evolve, but the infringers can adopt innocent names like congress:<link to song>. We need to intelligently handle copyright infringement without hurting access to legitimate websites."
It looks like part of doing SEO now is to be aware of avoiding broad innocent terms. I hope the unaware people don't get caught in Google's broad autocomplete filter.
Talking about magnet links I remembered KAD with ed2k and eMule. Its almost same. Bittorrent with magnet links just have different chunk size possibility to make faster download with smaller piece size.
From what I understand, DHT has a big trade-off vs BitTorrent: DHTs are crawlable and copyright holders can more easily track who holds copies of what yet it's easier to duplicate search engines like TPB within hours for the same reason.
So the effect seems to be that the RIAA, MPAA, etc. will likely not be able to take down trackers; they'll have to revert back to suing their "customers" (or lobbying to pass absurd legislation for that matter).
It seems that TBP just taken another step to push judge and jury's confusion during next trials even further.
"Y'know, we're trying to shut down pirates' secret base that... doesn't serve a single file!"
DHTs/Torrents are great for static data (like a movie) but bad for dynamic data, like a website with a list of movies, their ratings, user comments, etc.
There are, however, other systems that are designed to combat this, like freenet. They tend to be overwhelmingly slow, because you need to pass lots of data around to make it consistent.
Then again, magnet links, titles, and a little html are probably not a lot of information, so it probably could be done. I just haven't heard of any attempts yet. It's tempting to go and write one. Could you make a decentralized, P2P version of reddit with distributed trust? I think it's possible but hasn't been tried.
I think the easiest way is to not make it really "dynamic" but an append-only structure of static content. For example a Reddit thread might be representable as a static list of actions (add reply, upvote, downvote, etc), which the client would process to get the current state.
That way, you could immediately download an earlier version of a resource and then get the updates as they spread through the network.
They have a big problem in their design where currently it works exactly as you described, and at "some unknown point in the future" when the database of all transactions EVER in the history of bitcoin gets too big for each person to have to have a copy of in order to add another transaction, they'll "figure out a way" to make it unnecessary.
Since they are doing this big change, is there a way to make it more secure on the user side, too? Like encrypt the traffic and make it impossible for RIAA to track IP's?
Also since they say that it's like every user would have the TPB site on their computer, does it mean blocking the site would be completely useless? And since they are just links, and links are pretty much speech, I figure it would be impossible to turn it into law as well, to specifically target magnet based sites like TPB.
The entire design of Bittorrent is predicated on clients advertising to each other which torrents they're seeding and which pieces are available. You can encrypt your traffic to get it past your ISP's deep packet inspection, but the peer at the other end has to be able to decrypt it. And you have no way of knowing what nefarious organization controls that peer.
Don't the torrent files contain all the hashes for each piece? Doesn't that mean if a single piece is bad, the entire torrent can't be verified? Torrent files contain a lot of useful data that isn't found in a magnet URI. Is this just being ignored?
The hash in a magnet URI is of the info dictionary in the .torrent file. It's used to query the BitTorrent DHT for peers running the same torrent. Once another peer is found, your client will download the full metadata file from it, and from there it works just like any other BitTorrent download.
This is definitely a step in the right direction. Not saying this is a good step to increase piracy, but there really is no need for trackers when we have magnets, and this will lead to a healthier bittorent community.