In common-law jurisdictions (like the US), I wouldn't be so sure of that. It has a lot of precedent behind it, as a longstanding convention for indicating site-owner/rightsholder wishes. Ignoring it – deploying software designed or configured to be oblivious to it – could create legal risk.
A literal reading of copyright law and laws about 'authorized' use of computer systems would assess all bulk copying/reuse of web content without explicit advance permission as illegal. It's the force of traditional/reasonable industry practices (like robots.txt), and offsetting considerations like fair-use, that make it legally defensible.
In common-law jurisdictions (like the US), I wouldn't be so sure of that. It has a lot of precedent behind it, as a longstanding convention for indicating site-owner/rightsholder wishes. Ignoring it – deploying software designed or configured to be oblivious to it – could create legal risk.
A literal reading of copyright law and laws about 'authorized' use of computer systems would assess all bulk copying/reuse of web content without explicit advance permission as illegal. It's the force of traditional/reasonable industry practices (like robots.txt), and offsetting considerations like fair-use, that make it legally defensible.