

Stream of crawled sites from the Internet Archive (somewhat NSFW) - Mithrandir
http://crawl432.us.archive.org/crawling/domain/liveupdates.html

======
DigitalSea
I see images for a few seconds and then the images disappear and I just see
moving black squares in place of imagery like it caches only a set amount and
doesn't fetch any more unless you reload the page again. Cool, but pretty
pointless.

------
aj
The speed at which the crawling is taking place, the tile source (the site
being crawled) changes before my internet connection can even download the
tile. So after a few seconds, all I see are black blinking tiles

------
ronyeh
Pretty cool. It's funny how many thumbnails include the parked domain girl
(with backpack).

<http://www.urlesque.com/2010/11/03/parked-domain-girl/>

------
frozenport
It only works for few seconds when I switch the browser focus (by changing my
tab), then the tiles don't show anything. Is this a feature?

~~~
nodata
Are you using Adblock?

~~~
frozenport
yes

------
ck2
Either there is a security loophole and you can force it to crawl sites or
wayback has an open thumbnail service.

[http://crawl432.us.archive.org/crawling/domain/thumb.py/news...](http://crawl432.us.archive.org/crawling/domain/thumb.py/news.ycombinator.com)

[http://crawl432.us.archive.org/crawling/domain/thumb.py/www....](http://crawl432.us.archive.org/crawling/domain/thumb.py/www.google.com)

Judging by the delay, basically changing the end url forces it to crawl and
then the thumbnail is cached and served.

They might want to fix that.

~~~
cleverjake
nah - its a lookup.

[http://crawl432.us.archive.org/crawling/domain/thumb.py/www....](http://crawl432.us.archive.org/crawling/domain/thumb.py/www.urbanoutfitters.com)

vs

www.urbanoutfitters.com

the slide that is shown is from august.

------
Nursie
Somewhat NSFW?

As at work not going to visit, but it does now strike me that archive.org
probably has probably accumulated the world's largest porn collection.

~~~
brazzy
Actually when I just looked, there was not a single obvious porn site.

------
Mithrandir
For future reference, it looks like this: <http://i.imgur.com/QHPWW.jpg>

