On: Twitter just updated its robots.txt to exclude all scrapers

[guiambros]

IMO, that's exactly the reason. Before search engines could scrape the data and load the content for free. Now they'll need to reach firehose data agreements.

[Paulods → guiambros]

Take a look at the robots file without the "www" subdomain. Its likely to prevent content duplication/push for only the non "www" url to appear on search engines.

https://twitter.com/robots.txt

[guiambros → Paulods]

Uh... Excellent point. Seems the entire speculation is/was unfounded.