Hacker Newsnew | comments | show | ask | jobs | submit login

What are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at all"..



Right, but his point is that at least this way you won't get delisted :)

-----




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: