
Show HN: Wget-finder - chilicuil
http://javier.io/blog/en/2015/02/27/wget-finder.html
======
heavenlyhash
There's almost nothing to say because it's just such a self-evidently good
idea.

It's high time we start seeing software understand that data identity as
totally severable from data location -- and if our existing tools can't do it,
we're gonna start seeing more and more clever hacks like this to make it
happen.

(This (or bifrost, the upstream) should probably not be using md5 in this day
and age though! sha384 and blake2 are both much, much better choices, and
immune to length extension issues. sha512 is also fine, as long as the content
length is also tracked.)

~~~
coke12
> It's high time we start seeing software understand that data identity as
> totally severable from data location -- and if our existing tools can't do
> it, we're gonna start seeing more and more clever hacks like this to make it
> happen.

I believe it was Alan Kay who said, "names don't scale". We've known about
this problem for a half century. wget should be a semi-intelligent software
agent, so that we humans can free up mental resources for the truly difficult
problems.

~~~
signa11
> I believe it was Alan Kay who said, "names don't scale".

apparently in
([http://joearms.github.io/2015/03/12/The_web_of_names.html](http://joearms.github.io/2015/03/12/The_web_of_names.html))
joe-armstrong also says something similar.

------
moe
So in essence a poor man's magnet link[1].

Interesting hack, nonetheless.

[1]
[http://en.wikipedia.org/wiki/Magnet_URI_scheme](http://en.wikipedia.org/wiki/Magnet_URI_scheme)

~~~
colechristensen
(I'm sorry I didn't see your comment and said the same thing)

------
colechristensen
It seems like this is a hacked up version in the spirit of
[http://en.wikipedia.org/wiki/Magnet_URI_scheme](http://en.wikipedia.org/wiki/Magnet_URI_scheme)

You'd never have to worry about losing links to things if all of your links
were magnet links and you hosted files with bittorrent not http.

A magnet link can be just a sha hash. You could write a browser plugin to
rewrite all sha hashes into magnet links.

The real hurdle with that is releasing a bittorrent client that separated
itself from the grey area of media piracy.

If firefox included native libraries for downloading magnet links, it would be
invisible to users.

You could also write btwget using libtorrent (or patch wget to handle magnet
links)

~~~
rakoo
It already exists, and works wonderfully:
[http://aria2.sourceforge.net/](http://aria2.sourceforge.net/)

------
bruce_one
Clever :-)

I wonder if it could look for `${currentAddress%$filename}md5sums.txt` (or
similar) as that's often where an md5sum for a file is, and then comparing
that, rather than downloading the whole file and hoping for the best?

eg.

User wants file.tar.gz:aaaaaa...

wget-finder finds
[http://downloadable.com/file/file.tar.gz](http://downloadable.com/file/file.tar.gz)

wget-finder checks for (and downloads if present)
[http://downloadable.com/file/md5sums.txt](http://downloadable.com/file/md5sums.txt)

wget-finder compares the md5sum in md5sums.txt to the aaaaaa...

If it's good, it downloads the file (and still does the final check) and if it
isn't it keeps searching, having not downloaded the file unnecessarily.

Seems like it could be neat for large files (to avoid downloading the wrong
file as often).

(Could also check for md5sum.txt or md5sums or md5.txt etc)

~~~
chilicuil
That's a good idea, I'll give it a try the next time I touch the code =)

~~~
Create
[https://en.wikipedia.org/wiki/Metalink#Example_Metalink_4.0_...](https://en.wikipedia.org/wiki/Metalink#Example_Metalink_4.0_.meta4_file)

------
voltagex_
This is really cool. I wonder if there's any place for DHT here... except with
links to the files being shared rather than the file content themselves.

------
theophrastus
Sounds a wee-bit like a worthy re-invention of the ancient (veronica indexed)
gopher protocol
[http://en.wikipedia.org/wiki/Gopher_%28protocol%29](http://en.wikipedia.org/wiki/Gopher_%28protocol%29)

------
zwischenzug
Excellent! I was going to write one of these, now I don't have to. Thank you
so much! That's one big TODO ticked off.

------
herf
Does anyone index the web by {secure hash}? What a good index that would be.

~~~
rakoo
[https://en.wikipedia.org/wiki/Distributed_hash_table](https://en.wikipedia.org/wiki/Distributed_hash_table)

------
tgpc
such a cool idea :-)

