On: “Stop reverse engineering our code”

[hughw]

Would archive.org typically honor a robots.txt for a resource it already retrieved? I never understood the intent of a robots.txt to be retroactive.

[mikeash → hughw]

Apparently yes, it would: https://archive.org/about/exclude.php

[syncsynchalt → hughw]

My understanding is that sites like archive.org honor robots.txt retroactively not because they are required to, but to best honor the wishes of the content provider.

[X-Istence → hughw]

Yes, it simply hides the content, it is still kept in their database so if the robots.txt disappears, it pops back from their archive.

New pages won't be archived though.