

Ask HN: Examples of interesting robots.txt - r4vik

I'm sitting here going through some of our competitors robots.txt files wondering if they were doing anything interesting or innovative but they all seem pretty run of the mill. HN can you share some robots.txts that do something unusual or innovative?
======
balducci
How "innovative" do you expect robots.txt to be?

~~~
r4vik
Things like rate-limiting google by serving different disallows on every
request, or tuning sitemaps for different crawlers.

