I don't think robots.txt has any legal standing, but I do think the fact that Google is willing to respect requests not to index content from everyone except print publishers of paid content is indicative of bad faith on their part.
robots.txt actually has little to nothing to do with copyright, and more to do with "unauthorized computer access" which is protected by the CPAA.
It's a guard against accidental access or publishing of sensitive data - which isn't necessarily copyrighted but has other legal protections - as well as placing an unwanted burden on the server.
