If a site provides a sitemapindex in the robots.txt, would you assume they're al... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

shaunpud on June 6, 2017 | parent | context | favorite | on: Ask HN: How do you feel about web scraping?

If a site provides a sitemapindex in the robots.txt, would you assume they're allowing their site to be crawled on the pages they've set out within?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact