

Obama's New Robots.txt - r00k
http://codeulate.com/?p=24

======
tlrobinson
The vast majority of the entries in Bush's robots.txt were filtering out the
plain text versions which are linked at the bottom of the HTML versions
containing identical content. This prevents duplicates from showing up in
searches. This is likely done automatically by whatever software they use to
manage the content.

Want proof? Pick any of the entries ending in "/text", for example
"/911/911day/text", search Google with the "/text" removed like this:
"site:whitehouse.gov inurl:/911/911day" and you can still see the page in the
Google cache (at least until Google's index is updated).

If you want to view it as a metaphor, fine, but there's no evidence Bush's
administration was trying to hide anything on their website like this article
implies. If they wanted to hide it, why would they put it on there in the
first place?

------
miketheburrito
This is a great and semi-metaphorical comparison (woohoo transparency!), but
to be fair, the Obama administration hasn't done anything yet, so there isn't
even anything to hide at this point.

------
nir
Having /includes/ under document root - and trying to fix this via a
robots.txt entry (??) - wouldn't reflect well on Obama, if they actually had
any meaning :)

~~~
mdasen
The includes folder looks like it's just JavaScript and CSS (including jQuery)
so it has to be under the document root.

~~~
nir
Sorry, I'm taking that back... I was thinking of Obama's campaign site where
you could actually access stuff like /includes/footer.php (at least for a
while, seems to be fixed now)

------
gojomo
Why aren't we allowed to crawl their JS and CSS?

 _What are they trying to hide?_

~~~
riffic
I hope you're joking.

------
dejb
I'm more interested in what CMS they are using. Any ideas?

------
jamesv
/firstlady/newborn/text !?

~~~
Anon84
Humm... maybe there's a little Obama on the way?

Doubtful, but you never know.

~~~
jonursenbach
That was from the old robots.txt from the Bush admin, not Obama.

