They also send a crawler to any urls that they've not "registered" before (by IP). About 30 secs after navigating to a new page their bot will crawl a visited page. robots.txt is not honoured.
Something to consider when testing, or using, "hidden" sites and pages.
Something to consider when testing, or using, "hidden" sites and pages.