> Well, the search engines decided that robots.txt was the better approach for them. Which makes sense, since they want control over as much data as possible, that's their profit motive. The jury is still out on whether that's a long-term win-win social contract between search engine companies and the world.
Are you really arguing that the internet would be _more_ accessible if search engines had to reach out to every site they wanted to crawl?
How many companies out there complain about being scraped by Google? How many companies benefit from search-driven traffic?
Are you really arguing that the internet would be _more_ accessible if search engines had to reach out to every site they wanted to crawl?
How many companies out there complain about being scraped by Google? How many companies benefit from search-driven traffic?