I think part of the difference is that robots.txt was established early enough to become a de facto standard way to opt out of automated indexing. There is no equivalent system for print, where the convention is that everything is fully protected by copyright by default.
Another difference is that automated indexing has become inherent to web publishing, so much that people publish with the expection that Google will index their pages. Book authors, on the other hand, publish with the expectation that their works are not indexed in this fashion.
