I believe you have to look at robots.txt file. If it lets you crawl, you should ... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

rgovind on Nov 27, 2013 | parent | context | favorite | on: Introducing the Priceonomics Business Model: Data ...

I believe you have to look at robots.txt file. If it lets you crawl, you should be fine.

qq66 on Nov 27, 2013 [–]

Does the robots.txt have any legal significance? I would think that the Terms of Use can be more thorough and are more legally relevant.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact