Hacker Newsnew | comments | show | ask | jobs | submit login

Their robots.txt file is being delivered as the wrong content type for me. It is being served as a "application/octet-stream" instead of a "text/html"

I have no idea what consequences this might have since I've never seen another site which does it. Some quick Googling turned up nothing.

Maybe Bing choked on the wrong content type and decided to blackball their entire domain in order to avoid indexing the unindexable?

Why would you want to serve up a robot.txt as "text/html" when in reality it is "text/plain"?


Applications are open for YC Winter 2016

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact