Python parallelism in one line - A Better Model for Day to Day Threading Tasks

iurisilvio · on Dec 30, 2013

It works for I/O bound tasks. For CPU intensive tasks, you have almost no improvement. I use it a lot for web scraping.

You can from multiprocessing.pool import ThreadPool. It is not well documented too, but at least the package makes sense.

goostavos · on Dec 30, 2013

I'm not sure I entirely agree with your CPU intensive statement.

Multiprocessing definitely brings Python speedups in the CPU bound arena. I've used it for everything from template matching, to building movies from still frames. Heck, my primary solution to speeding things up is throwing more cores at it rather than figuring out why my algorithm sucks ;)

A couple of quick benchmarks and you'll be a believer, too!

rbanffy · on Dec 30, 2013

It depends on your workload - the GIL is still there after all. If you use the multiple processes you may get away with that. The article mentions this case, but uses PIL, so, the CPU-bound part is running compiled C code with very little GIL interaction.