Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
gnud
on Feb 9, 2018
|
parent
|
context
|
favorite
| on:
Facebooks crawls every page recorded by its tracki...
I assume the crawler only does HEAD/GET-requests Your fault if your webpage changes anything based on a GET.
Now, if the crawler doesn't honor robots.txt, then you can complain (loudly).
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Now, if the crawler doesn't honor robots.txt, then you can complain (loudly).