
RobotsDisallowed: Make Sure Your Site Doesn't Do This - danielrm26
https://github.com/danielmiessler/RobotsDisallowed
======
0x006A
The title of this post is wrong, its also not what the project is about.

There are good reasons to put urls in robots.txt. You can use it to exclude
parts from your website from search engines. This is however not a way to hide
parts of your site. So don't use it to hide parts, but for parts you don't
want indexed.

/admin/ is a good example. This is a login page to your admin backend. Telling
search engines to not list that page is fine. Assuming /admin/ would not be
accessed if you put it in robots.txt is not. Claiming because /admin/ is in
robots.txt it would be a greater security risk is just wrong.

