The sqlite db contains a imdb column but it seems like the author forgot to include it in the fts4 index, meaning one cannot search for "tt9916362" even though it's right there in the database. It's a shame because this curated mapping is the most useful aspect of RARBG.
FYI, this dump has 826'201 magnets (torrents) with an associated imdb ID and 2'017'490 without, including lots of porn but also random music and software.
Category breakdown (careful though, most items aren't categorized or use a category code I couldn't interpret):
XXX ( 1): 2,255
XXX ( 4): 607
Movies (14): 3,206
Movies (17): 117,440
TV (18): 198,314
Music (23): 11,621
Music (24): 471,161
Music (25): 1,339,739
I have a question for you as even playing with this or creating oss project using this sounds like inviting trouble. So if any programmer wants to create an open source project using this data (just for kicks) then apart from using a VPN and throwaway email do you need to be careful about anything else? Any tips?
How does this compare to the ipfs dump that someone claimed has over 2 million entries? Your DB dump has only around 1.6 million... so does ipfs have duplicates? or is there something substantial missing in your dump?
I was curious how this works and then I saw the sqlite requests in the network tab. It's amazing to see what we have access to these days -- SQLite over HTTP over IPFS to provide a giant, censorship-resistant database!
In practice it's very rare for me to see a direct ipfs protocol link, almost all the traffic goes through HTTP gateways (which frequently cache the content as well). Hard to imagine that they don't become a target for "hosting" pirated content if / when IPFS becomes more than a negligible platform for piracy. (A significant amount of Library Genesis traffic is already using IPFS via these same gateways.)
As you mention, there's a DMCA process for some of the gateways, but that might not be enough to ward off attention.
Can you explain bit more? Isn't the dump of last RARBG magnet links on the order of MBs? We can just download it and grep in plain text. I don't get what is the role of SQLite or IPFS here.
Disclaimer: I wrote that article and was somewhat involved in the other sqlite over ipfs project this is forked from.
Yes, for MB size files just downloading the whole thing will be faster and much easier - even if running in the browser. I'd say the boundary is somewhere around 10MB compressed (~20-200MB uncompressed). Looks like the sqlite dump used here is ~400MB in size, ~180MB compressed. Loading that in the browser would probably work but it wouldn't be too great.
torrents work fine without having a monetary incentive, this misconception of the blockchain crowd really has to die its killing real decentral solutions
I said "incentive". You made it "monetary incentive".
What do you mean with "torrents work fine"? Do you have any statistics on the lifetime of torrents?
Everything ever written to the Bitcoin blockchain is still available and massively distributed. It seems very unlikely anything will ever get lost during our lifetime. So I would say that is an example that monetary incentives do work?
>Everything ever written to the Bitcoin blockchain is still available
This isn't useful for 99% of stuff and pushing every video and music file on earth through a global state machine with the performance characteristics of an Atari from the 80s would render the thing inoperable. The entire Bitcoin network has a bandwidth of like 1mb every 10 minutes, the total size of the blockchain is half a TB in total, people torrent more porn in the time it took me to type this.
Maintaining a global, complete history of transactions only ever made sense for one problem, double spending, and is utterly useless for sharing files.
Aaah private trackers! I remember once I pondered to enter one of those "special groups" but bailed when I read that you had to study for some sort of interview that was required to try to join .
Lol no... I dont like interviewing for a new job, why would i agree to get tested for something as stupid as piracy? (Unless... mention me ONE private tracker that doesn't have piracy).
I remember what.cd had interviews like that. Tried and failed as a kid, that was a pain
There are definitely private trackers that don't have anything like that though (iirc they're called "semi-private trackers"). One i remember using is t411 when it was still running
it's content addressed, if an ISP was so keen it would be simple to blackhole requests for a particular file (of course, one need only to change 1 bit to get a new content hash, but you have to redistribute the new file from scratch
I don't know about how IPFS is implemented, but you could use content-addressed blocks underneath every file too. This way flipping a bit means that only one underlying block changes, and that the new file would share N-1 blocks with the previous one, making redistribution only require sharing a single block.
A note of caution for those unfamiliar with how IPFS works.
It's very similar to BitTorrent with how content distribution happens. Your local node will broadcast which content it has available (downloaded).
If you access a piece of content you automatically become a host for it so you still need to use a VPN if you live in a country where you can get sued.
However, is using a proxy like ipfs.io really using ipfs?
If everyone did that, there's no point to using this protocol. The strength of the network comes from the fact that the content gets replicated/distributed when accessed. That doesn't happen when accessed through a proxy.
The implementation works well, but has limitations. It only works from browser to node and not browser to browser. Depending on what you are looking for it is more or less fast
Using Brave it makes this very clear, the above link won't open by default without the user choosing to run a local node or use a public gateway instead. It explains the implications of both options.
That would hold true if IPFS would be a specific application.
But it is a protocol.
I would not expect that the protocol specifies that a complient client shares what it downloaded. That sounds more like a choice the client developers make on their own.
But I'm happy to be corrected if someone has a link to the protocol definition and it says different.
By default, what the GP says is true. If you can point to specific popular clients that do not have that behavior by default then I think that would be more convincing than saying "Well technically there's no requirement" when it's impossible to avoid in practice without writing your own custom implementation from scratch
AIUI Lassie uses indexers to locate content instead of using a distributed hash table (DHT). So a content identifier (CID) might be present on IPFS (and downloadable by a regular IPFS client), yet 'invisible' to Lassie.
If my understanding is correct, it can't be considered a complete IPFS client.
I think it uses indexer and saturn (which ought to fall back to the DHT if it hasn’t cached the content of a CID already).
That said they have a GitHub issue tracking CIDs that are retrievable via Kubo (fka go-ipfs) that aren’t yet retrievable via Lassie - it suggests to your point that there must be some difference; it also suggests they intend for retrievability parity though, so fingers crossed!!
Yes and the many IPFS implementations (I've tried them all) consume tremendous amounts of bandwidth and require shocking levels of system resources (considering what it does fundamentally).
There's an entire cottage industry of IPFS pinning and gateway providers that exist largely because of the challenges of running your own IPFS node for anything beyond casual use.
Ipfs is is a neat project with a lot of interesting ideas. but, yeah, the deamon is often way too resource hungry. I would like to see a lite/embedded mode that would better run in resource constrained or casual setups.
If you access a piece of content over ipfs.io, and you don't have your browser set up to actually do those requests over a local IPFS daemon, you are not using IPFS. You are just using a normal centralized website.
Is it recommended to run my own proxy then, and is there any boilerplate project out there? I could also use OpenVPN, but seems like I just want to proxy ipfs, not my whole connection.
But now, I dont know why, theres a lot of IPFS pinning service I got HUGO hosting in GH and deploying in FTPS from fleek.com. It has nothing on it but it works like a charm.
G'day Why! This is operating through the gateway. So you have to trust the gateway currently. Yes, you can move the gateway to your machine and then the trust requirement doesn't extend over the network.
However with local ipfs, bitswap doesn't support range requests, so you're at least downloading the enclosing blocks which could be 2 MiB for 1 KiB requested data, or 2000X more data than you need.
Is there an IPFS dedicated to training data? A mirror of input datasets and fully open models resident on HuggingFace could endeavor to cut out onerous license agreements when possible.
IPNS kind of sucks but it’s “worked” for me for years. (sucks = very slow name resolution and slow update propagation, at least when I was using it a couple years ago)
Do you have any advice on how to work out if an IPFS project is using IPNS? I can't see anything on this page that suggests it is, but maybe there's a better, more intentional technique over searching "ipns" in the inspect source and network tab : )
I'm not the person you replied to, but I'll answer anyway :)
IPNS is essentially DNS over the IPFS network. IPNS domains point to a specific IPFS file (or a set of files, like we see here). IPNS domains are signed with a private key; when you want to update your IPNS entry, you add your new content to a new IPFS file
and then you update the IPNS entry by signing it with a public key.
This seems to be derived from a dump that surfaced the internetz around June 1[0], even before the GitHub repo[1], which is probably the result of a random person/team's archiving/scraping effort. We won't be able to know the cutoff or the coverage percentage without that person's insights.
I'm in Germany as well. You're fine. You didn't download anything, and more importantly, didn't UPLOAD anything. Accessing this index site is not fundamentally different from using any other search engine.
How do IPFS HTTP mirrors deal with javascript origins? The origin here is always https://ipfs.io/, right? Or can a http server send a custom origin in the response headers?
Web Browser use cases without native ipfs:// and ipns:// should use subdomain HTTP gateways like localhost in Kubo, or public dweb.link, cf-ipfs.com, which provide Origin per root CID, sandboxing web apps from each other.
I recently received a phishing email, hosted on IPFS. Seeing torrents on there too, now, makes me wonder how they deal with content that's illegal in whatever jurisdiction.
I think the best arguments I heard about it was, unlike torrents, the availability is longer. Because the "nodes" store them generally for years at minimum UpTo even decades.Downloads are as fast as your broadband connection, just regular SSL communication, hence no VPN. Don't have to join private trackers, and prove you are worthy enough to be allowed to download.
You dont have to join private trackers, but you do have to join a private usenet provider. The only companies holding years/decades long retention of files are the paid private ones.
Breadth on Usenet is a lot worse though. One can find a private tracker to cover pretty much every -non mainstream- niche one can think of, and that also comes with a community attached.
Pure DHT peer-discovery alone isn't that great but keep in mind that the magnet points to a .torrent file, which usually does contain a list of traditional trackers for the file.
Actually, a minimal magnet link only contains the infohash which "points" at just the info section of the torrent file. The info section contains the name, list of files and SHA1 hashes for all pieces, and the private flag.
The trackers are outside of info section, I assume so the list of trackers can be modified without effecting the info hash.
Though, some magnet links include an optional xs parameter which points at an http url containing the .torrent file, which would include the trackers.
In my experience, you don't receive the trackers from other peers (though maybe other bittorrent clients can?). However, if at least one peer you discover via DHT supports the peer exchange protocol, and has the trackers, your client can quickly query all the relevant peers.
> SQLite compiled into WebAssembly fetches pages of the database hosted on IPFS through HTTP range requests using sql.js-httpvfs layer, and then evaluates your query in your browser.
Please don't do that with the non-static data. This is okay only for archived project, except the moment when my PC downloads SQLite indexes from some blockchain based paifully slow storage for the first time. BTW, how about a user-governed local data cache, is there any recent quirks in browsers for that? LocalStorage is still inconsistent and unreliable, right?
FYI, this dump has 826'201 magnets (torrents) with an associated imdb ID and 2'017'490 without, including lots of porn but also random music and software.
Category breakdown (careful though, most items aren't categorized or use a category code I couldn't interpret):