

Tell HN: Nuuton crawler is active.  - orangethirty

The second generation of the Nuuton crawling system (codename: Gravity) is in the wild. I will be testing a lot of different parameters. The speed of the crawl is not quite ideal, and I will be adjusting it to respect your bandwith. If you experience a sudden jolt of bandwith consumption, let me know. It is not on purpose, because this thing is a prototype. It should not happen, because I have included safety mechanism, but its not 100% perfect.
At the moment, I'm only using the test server, so the probability is not very high. Still, let me know if anything. I dont want to damage your things.<p>Also, if you want Nuuton to return to your site (you have a blog), then include the following line on your robots.txt file:<p>nuuton: return(<i>n</i>)<p>where <i>n</i> is the amount of time between crawls in the unix timestamp format (epoch).<p>I make this post to alert the community of possible issues. I will be posting on twitter/orangethirty any issues that may arise.<p>Please report any issues to orangethirty@nuuton.com<p>:)
======
tectonic
It would probably be a good idea to tell us what this all means.
<http://nuuton.com/>

------
cpeterso
What is Nuuton? Why invent a proprietary robots.txt feature instead of the
(common but non-standard) "Crawl-delay:"?

~~~
orangethirty
1\. Nuuton is a search engine.

2\. Because this feature ties in with others that I cannot talk about. Its
optional, and meant for optimizing for Nuuton.

