
Sponge 0.22: A website crawler and links downloader command line tool - spypunk
https://github.com/spypunk/sponge
======
zzo38computer
I might suggest to add options to configure the referrer and user-agent
strings, and an option to specify whether or not to use robots.txt. Another
possible option would be to refuse to access the local network.

~~~
zzo38computer
Actually, an option to refuse to access the local network is worthless if the
program does not already have the ability to cross domains. But if such a
thing is added in future, then for security purposes the user might not want
to allow crossing the domain from the internet to the local network.

