Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
nostromo
on Oct 12, 2011
|
parent
|
context
|
favorite
| on:
Googlebot now makes POST requests via AJAX
I wonder if you could add your POST URLs to robots.txt if you don't want the crawler to access them.
If other crawlers start doing this, it should probably be added to robots.txt formally.
martian
on Oct 12, 2011
[–]
I'm an engineer at Thumbtack. And yes: we have noticed that Googlebot does seem to obey robots.txt even when issuing these AJAX requests.
Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
If other crawlers start doing this, it should probably be added to robots.txt formally.