

How to fetch URLs in parallel using Python - coderdude
http://petewarden.typepad.com/searchbrowser/2010/10/how-to-fetch-urls-in-parallel-using-python.html

======
riobard
Won't using multiple curl processes consume more resources than threads?

~~~
coderdude
Yes. However, I think how well this works depends on how many cores you have
and how evenly you spread those processes across each core.

~~~
riobard
If the goal is to fetch as many URLs as possible, the best bet seems to be
using async epoll to simultaneously fetch tens of thousands links.
Processes/threads consumes too much memory to be viable.

------
mumrah
Seems like it would be easier to use xargs or GNU Parallel.

