Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines?
Right now it is extremely difficult to build your own web crawler that would compete with Google. And that is not because of the technology, but because multiple sites will prevent your bot from accessing them if you're not Google or Bing - either through robots.txt, or through directly banning your IP if it's trying to crawl and it's not a confirmed google-bot.
Having a non-profit, open source, crawler that keeps an up to date index (or web cache) of the web would help competition spring up.
Isn't that one more data sets for ML other research purposes instead of a highly up to date search index (For example with news from a few minutes ago).
Roughly speaking, yep - Common Crawl provides a sizable chunk of web data (420 TiB uncompressed, over 3 billion unique URLs, as of May 2022; historic statistics here[1]), and is updated on monthly basis. Not near-real-time, true, albeit relatively fresh.
A question to ask could be: how often do users care about information from a few minutes ago, compared to information that has been available for a longer duration of time?
I mean, any time someone wants information on current or recent events is your use case right there. If you exclude news entirely, you could maybe disregard recent websites but I imagine that's statistically a pretty large portion of search.
I built something with an API that uses Selenium to image a site. It works on a large percentage of the sites I feed it.
I don't recursively call links found in the pages. I expect the user to give me the URLs to crawl and save.
In order to "find" new content, I let the user specify where they want to search for things the engine hasn't "crawled" yet. So, a search for scooters to buy might end up searching Amazon directly, then lets the user "save" the site by passing the Amazon URL for the scooter they like to the system for imaging.
I use GPT-3 or other ML models to do some of the heavy lifting for adding labels to the pages or documents the user uploads.
This ends up being a "curated" list of documents important to the individual user, not an exhaustive crawl of all things which are important to all users.
> Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines?
That's a excellent idea! In the spirit of open-data, and people can do with it what they want.
I think this is a great idea. How does this work with copyright? Search engines seem to be able to download a reproduce content from scraped pages (and wrap it in ads, and derive content from it) this is called “indexing” when they do it but scraping when everyone else does it.
E.g. on Google if you search for "how to tie a tie", a little info box may pop up with step by step instructions. This content is taken from some website, but that website gets no page hits or ad revenue. Instead, Google gets to serve ads on the search engine results page.
(I don't know if this happens for this specific example, but Google does this for some searches)
Part of why sites participate in the infobox program is that in practice you do get quite a lot of hits from it: many people click through to see the answer in context.
I think they're referring to how Google "extracts" answers from your website and shows it on the search results page. Effectively meaning that the user doesn't even need to go to your site to get the answer, because Google extracted it and gave it to them directly.
It seems to me that what they usually extract is some junk only vaguely related to the query and often cut apart and reassembled in a way that's just wrong.
I'm curious if you think a co-op would be feasible between Mwmbl and other like-minded crawlers who are interested in taking a divide-and-conquer approach to crawling the web.
Right now it is extremely difficult to build your own web crawler that would compete with Google. And that is not because of the technology, but because multiple sites will prevent your bot from accessing them if you're not Google or Bing - either through robots.txt, or through directly banning your IP if it's trying to crawl and it's not a confirmed google-bot.
Having a non-profit, open source, crawler that keeps an up to date index (or web cache) of the web would help competition spring up.