Hacker News new | past | comments | ask | show | jobs | submit login

The reason shown in the crawl error is "Robots.txt fetch".

This isn't very surprising as 'fetch as Google' returns a page unavailable for everything.

(It's my understanding that a 404 on robots.txt is OK)




In the last few days I've seen errors with our robots.txt on sites that have NEVER had a problem with it and no errors. I even looked around on Twitter thinking it was a Google Bot bug. Then it just went away. Out of curiosity, why would a 404 on a robots.txt be OK? Shouldn't you want "Fetch as Google" to return the proper HTTP/1.1 200 OK?


I don't have a robots.txt, so -for this url alone- I'm expecting a 404. What I've read in the google documents is that when a request to get the robots.txt returns a 404, it will consider it as OK to crawl.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: