Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Ah, ok, strange that they have /sitemap.xml in addition to that then. I would assume any robot that would understand sitemap also understands that it can be specified in robots.txt


Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: