

Show HN : I have developed a robots.txt full text search engine. - Joyfield
http://robots.dnsdigger.com/
As a sub project of my pet project DNSDigger.com i have developed a search engine that index robots.txt (http:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Robots_exclusion_standard).
I actually don&#x27;t know what poeple could use this for but i suspect poeple that are into SEO could use it and black&#x2F;white-hat hackers(?).
What i need help with is VPS&#x2F;$$ for spidering. And as i sit on a considereble index of domainnames (though DNSDigger.com 300m hosts) i thought that
if you want a TLD indexed sooner than later you could sponsor it with either VPS or Paypal. See it as a strange &quot;kickstarter&quot;.
And if you think the &quot;design&quot; suck (which it does because there is a lot of things i suck at and design is one of them) you are free to submit <i>anything</i> better. I can put up a banner for your service a limited time.
I am adding new hosts&#x2F;TLD all the time.<p>And if you have to ask what this is you will not have any use for it even if you did ;)
======
Xeroday
Very cool project. I'm curious as to how you're getting all these robots - are
you scraping them yourself?

~~~
Joyfield
Downloading them as we speak. I have a big list of hosts/domains i have
collected through spidering for my DNSDigger.com. This is a hobby project that
has grown a bit over my head hehe. And there is no scraping needed. Robots.txt
is just simple textfiles. Download and parse, repeat a couple of million times
and build an index :)

