Hacker Newsnew | comments | show | ask | jobs | submitlogin

Not my site but this may help you: http://www.sxw.org.uk/computing/robots/check.html

------------ Results for https://safeshepherd.com/robots.txt Error at line number 1:

User-Agent: * Capitalization. Field names are case sensitive - the User-agent field should be written with that capitalization Error at line number 2:

Allow: / No User Agent. An Allow line must have a User-agent line before it. As records are delimited by newlines, there cannot be new-lines between the User-agent and Allow lines. Warning at line number 2:

Allow: / Allow is not widely supported. The Allow field was a late addition to the robots.txt standard, and is not currently widely supported by crawlers. You should consider alternative ways of constructing your robots.txt file

Error at line number 3:

Disallow: /login/auth No User Agent. A Disallow line must have a User-agent line before it. As records are delimited by newlines, there cannot be newlines between the User-agent and Disallow lines. Error at line number 4:

Disallow: /users No User Agent. A Disallow line must have a User-agent line before it. As records are delimited by newlines, there cannot be newlines between the User-agent and Disallow lines. Error at line number 5:

Disallow: /signin No User Agent. A Disallow line must have a User-agent line before it. As records are delimited by newlines, there cannot be newlines between the User-agent and Disallow lines. Error at line number 6:

Disallow: /upgrade/submit No User Agent. A Disallow line must have a User-agent line before it. As records are delimited by newlines, there cannot be newlines between the User-agent and Disallow lines.




Applications are open for YC Summer 2015

Guidelines | FAQ | Support | Lists | Bookmarklet | DMCA | Y Combinator | Apply | Contact

Search: