
Ask HN: Does this 'local web' tool exist yet? - newman8r
If something like this exists, I want to use use it. If not, maybe I want to open source something. Pasted from my idea log as follows:<p>premise: a local internet that has a lot of curated resources
-If  there was an apocalyptic event, this is the database you would want<p>-it is possible today to have an entirely local version of the web that would amaze anyone from early 2000.
-premise is there is no worry about ip&#x2F;copyright - it&#x27;s a hypothetical
-Can be updated, added to, shared, can run with minimal components, can run on its own os
-can store massive amounts of data on a tape drive if needed, index it well and copy blocks to a large hard disk or solid state drive. Lots of ram (64gb+) for fast searching of active blocks<p>• Encyclopedic data
	• edu sites
	• original articles&#x2F;newspaper articles
	• backups of abstracts from as many studies as possible
	• stack overflow, other similar sites
	• text of reddit, submissions with scores above 10
	• copies of classic books and technical books&#x2F;nonfiction
	• Top 200 movies, best tv shows full series - various levels of compression as needed
	• most useful youtube videos, ted talks, best documentaries
	• Links to the most useful software, highly compressed files,etc
	• Compilers and IDEs for all common programming languages and learning resources
	• datasheets for components by number, etc.
	• Top music by year, genre, album, etc.
roms and emulators for many systems
======
leojg
These stuff exists, but it usefulness depends on the administrators.

I was really surprised when a couple of years ago visiting Tigre, a city north
of Buenos Aires, argentina:

There is a huge river delta there, the Tigre delta and the people living in
the small islands formed by the many branches of the river have a sort of
local internet with their own news sites, wiki, etc... as well as access to
the worldwide internet.

Another interesting fact is that some of this island houses have electricity
but no drinkable water, they have to carry it from the city docks

~~~
newman8r
This is fascinating. I've heard of these local internets in remote areas. I'd
love something along those lines for my own personal enjoyment and to give
access to my friends.

I might go to def con this weekend and hit up the data duplication village

------
Artlav
There used to be a program called Teleport Pro, which can download an entire
web site.

I used it rather extensively to download interesting sites for local storage,
so i got a large selection of pages, a lot of which do not exist any more.

Then i moved from dial-up to something faster around 2007, and the habit to
download the sites gradually died out. Most of them are pre-2009, and the
latest one is from 2014.

I suppose making a new/free program like that might be of use today - the old
one can't parse modern sites too well, i.e. Wikipedia.

~~~
0942v8653
Try the recursive options of wget. It should be pretty much what you are
talking about.

~~~
Artlav
Huh, i was going to say that wget won't fix the links in the downloaded files,
but turns out they thought of it too.

Nice, that means the tool does exist already.

~~~
newman8r
Check my response above, feedback appreciated. downloading entire websites via
wget isn't what I was getting at although that type of tech is certainly
something that would be integrated into the hypothetical system.

What I am thinking of would really go beyond what you could achieve with wget.
No doubt offline browsing and personal crawlers have been around for a while.
Think of a 10TB+ tivo-like device that comes pre loaded with everything from
wikipedia to musicBrainz. People would't need to figure out what to download,
it would already be there, and they could augment it with anything.

Just a thought - I am going to keep looking into it and will create a public
repo for it if it makes sense.

------
cocktailpeanuts
Something relevant [https://www.indiegogo.com/projects/lantern-one-device-
free-d...](https://www.indiegogo.com/projects/lantern-one-device-free-data-
from-space-forever)

Which sources content from [https://outernet.is/](https://outernet.is/)

~~~
newman8r
cool concept, nice find. I want something like this that I can host, no
satellites needed, I can customize, let friends connect remotely, be able to
do really cool searches using regex, have some big databases, etc.

I guess it's along the lines of scraping/crawling/local browsing, etc.
Basically creating my own internet that I can invite people to, annotate,
collaborate in, etc.

I'm working on an open communications platform at the moment and am
contemplating creating a microservice to enable something like this - but I
don't want to reinvent the wheel if someone has already done it well.

~~~
cocktailpeanuts
I think you should definitely do it unless it takes years to ship the first
version. The first in market doesn't mean the first to succeed.

~~~
newman8r
That's true. I'm not concerned about being first to market or making any
profit on this type of project, would just be open source and release under
GPL or whatever is appropriate.

I just didn't want to spend the time or effort doing it if a similar
alternative already existed. My primary motivation is essentially just wanting
the tool for myself and friends.

Not sure how long it would take but I might be able to release a useable proof
of concept after a 3-day-weekend hackathon.

------
exolymph
Wikipedia seems to fulfill a fair amount of your requirements. Add in Project
Gutenberg...

~~~
newman8r
Yeah wikipedia would be part of something like this. I already have all their
data dumps and I have a lot of other cool DBs like musicBrainz, lots of stack
exchange stuff, some "top" movies lists, lots of books, etc. I like to hoard
data and hd space is cheap.

------
simon_acca
I don't know of any such curated collection, but a good tool to build what you
are describing would be IPFS: [https://ipfs.io](https://ipfs.io)

~~~
newman8r
I had no idea this even existed. This would be extremely helpful.

