Hacker Newsnew | comments | show | ask | jobs | submit login

Google indexes URLs despite measures such as robots.txt when these URLs are discovered by Google software including Chrome and their Toolbar.



Robots.txt is about fetching content, it has noting do to with indexing URLs or anything which is part of the content at non-robots.txt restricted locations.

-----




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: