On: Google already knows its search sucks (and is working to fix it)

[RyanMcGreal]

Google's build-out has been nothing short of astonishing. I remember when Google indexed the web once a month - and the web was much smaller then. Now my sites get crawled several times a day - and new articles on my news site are in Google News within less than an hour. The mind boggles.

[candeira → RyanMcGreal]

I just did a search for an error message in a piece of software. The one result from Google was my own pastebinned traceback from five hours earlier.

Google indexes random pastebins faster than I can forget I posted them.

This still cracks me up, but now I laugh nervously: http://www.ftrain.com/robot_exclusion_protocol.html

[diego → RyanMcGreal]

The main reason their cache was useful back then is that many pages were long gone and the links were 404s. Inktomi marketing always tried to play up our freshness compared to Google. In 1999 they had some pages in their cache that were 3-5 months old. We were pushing new indexes once a week, no page was older than a month. Seems quaint now.

[nostrademons → diego]

I find that the cache is useful now because articles get posted to social news sites, the influx of traffic brings down the site, and Google Cache is a handy mirror that usually seems to have the page already.