Hacker News new | past | comments | ask | show | jobs | submit login

"In this case, you should not disallow the page in robots.txt"

But don't worry, we'll ignore the information in robots.txt anyway, so maybe it's better to have both information there.

And maybe if it's relevant they'll ignore the X-Robots-Tag as well.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: