Hacker News new | past | comments | ask | show | jobs | submit login

"Our goal here is to be able to distinguish between the good type and bad type of scraping and give webmasters full transparency. Obviously this is a hard problem. If you have any feedback on any of this we'd love to hear it."

Yes as said before plus:

- Obey robots.txt to the full extend

- Name your access, i.e. label your bot

- Don't use shady tactics such as IP rotation

- Provide web site owners the option to fully block access of your bots (yes, communicate your full IP ranges)

Again - this is from a content owner who paid for his content.




Registration is open for Startup School 2019. Classes start July 22nd.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: