
Proxy Web Crawler - rootVIII
https://github.com/rootVIII/proxy_web_crawler/blob/master/proxy_crawler.py
======
flatfilefan
I read the Readme. Still not sure about the problem it solves. What’s the use
case?

~~~
rootVIII
Ah... well I used to run a mildly popular website... I originally wrote it to
"bot" my site 24/7.... it can be used to see the index of your site depending
on keywords. Also you can drop payloads with it... use it for brute force
password guessing... or you can use it for SEO as well... even though it
searches with Bing, Google may know about the visits if you use Google
Analytics... many use cases may present themselves with very minor
modifications to the code... enjoy!

