Hacker News new | past | comments | ask | show | jobs | submit login

I'd like to see an actual case reference for this. I've never heard of ignoring robots.txt resulting in any kind of legal action.

It would take a pretty big leap to go from robots.txt is advisory to ignoring it constitutes a criminal action.

Internet Archive was sued unsuccessfully. As I understand it a lawsuit is still in process against Google on the topic. So I guess the precedent is weaker than I thought, but still: tread carefully.

Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact