

Twitter.com/robots.txt # Crawl-delay: 10 -- Googlebot ignores crawl-delay ftl - guynamedloren
http://twitter.com/robots.txt

======
zzleeper
A quick question: What is "crawl-delay ftl". I googled but couldn't find much,
besides that this particular robots.txt dates from April 2007 (~4 years ago)
and the first mention of it was in 2008

~~~
arctangent
It may be an acronym for "For The Loss/Lose", which would be a corruption of
"FTW" (=="For The Win"). They're saying that it sucks that Google doesn't
respect rate-limiting with the crawl-delay parameter.

~~~
zzleeper
ty!

------
danielamitay
The most honest robots.txt file

