But robots.txt provides minimums: don't scrape this page, don't refresh more tha...

		thatthatis on Jan 16, 2014 \| parent \| context \| favorite \| on: Show HN: Kimono – Never write a web scraper again But robots.txt provides minimums: don't scrape this page, don't refresh more than once every x, these crawlers are allowed this access, etc.