
Show HN: Exoskeleton – a framework for polite, but tenacious crawlers - RuedigerVoigt
https://github.com/RuedigerVoigt/exoskeleton
======
RuedigerVoigt
Exoskeleton is an open-source framework for crawlers written in Python. It
utilizes MariaDB to manage the queue. It is built to download a very large
number of files and / or web pages. Meanwhile it avoids to overload servers
and spreads the tasks - if necessary - over multiple days. Furthermore, it
automatically avoids duplicates and allows you to organize file collections
with labels.

