> It may be possible technically. I doubt it's possible in reality: they've probably made promises to a bunch of people over the years that this is how it works. People furious that Their Content is appearing somewhere else and go straight into lawyers on the first email.
They won't be furious when they're dead.
I think the main value of the Internet Archive is not so much in the near term, but in the long term. I hope in the future they enact some policy that ignores any robots.txt for scrapes older than, say, 50 years.
They won't be furious when they're dead.
I think the main value of the Internet Archive is not so much in the near term, but in the long term. I hope in the future they enact some policy that ignores any robots.txt for scrapes older than, say, 50 years.