Hacker News new | comments | show | ask | jobs | submit login

Googlebot also does some other crazy stuff. Like looking at url patterns and then trying out variations.. they're almost trying to sniff URLs!

For example if I have a page: www.domain.com/xyz/123

Googlebot (without any links to other pages, will actually try URLs like) www.domain.com/xyz/1234 www.domain.com/xyz/122 www.domain.com/xyz/121 and so on...

It's crazy how much 'looking around' they do these days!




I believe that one's mostly a search for duplicate content - looking for URL parameters that don't make a difference.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: