

Show HN: Domain Crawler - cblock811
https://github.com/zillabyte/domain_crawl

======
AznHisoka
does this obey robots.txt and have some sort of delay so it doesn't take down
a site?

~~~
cblock811
It's a starting point for a crawler, and does not have limiters on robots.txt
and sleeps yet. Users could easily add them in if want to crawl with it.

