
Practical threaded programming with Python - gklein
http://www.ibm.com/developerworks/aix/library/au-threadingpython/
======
herge
Please note that any use of python threads (in cpython, at least) will still
use only one core at a time. So things datetime stuff, using Beautifulsoup,
etc, will not be done in parallel.

To do stuff on more than one core, look at the multiprocessing module.

~~~
rdtsc
That is true. I think date was more of an example on how to use threads. But
yeah, Beautifulsoup bit might actually run a bit slower if it is only doing
parsing. (!unless there is a C extension underneath that does release the
GIL!)

Any discussion of Python and threads is always confusing and it seems to me
there aren't many people who understand how it works (I know you do, not
talking about your comment, just in general).

In one camp you have people who like to write how Python is no good and
threads are just broken. Never use them. In the other camp are people who say
threads are fine, they work great, I never had any issues with them. A lot of
time people in this camp are just reacting to the ones in the first camp but
also without understanding the underlying mechanism.

I think the first thing that should be mentioned in any introductory article
on Python threads is that Python threads work great for IO concurrency but
they won't help with CPU concurrency. Things like downloading files, sending
data over a socket will work nicely. Thing like computing determinants won't.
You can still structure your code in a threaded fashion so can use
multiprocessing module in the future but you won't get a speed up. So threads
are not completely unusable and broken but they also have a surprising
limitation.

Overall in Python in my career I probably deal more with IO concurrency and
threads helped there quite a bit. Others will have a different experience
depending on their area of expertise.

Also it is worth mentioning that libraries like Numpy and C extensions in
general have the option of releasing the GIL if they want to they can get a
speedup. I have done this once by hand and it did help (with a hand written
extension). Didn't personally test numpy's speedup.

ADDITION:

It is also worth mentioning that even though you not get a speedup for CPU
related concurrency, you still have to deal with synchronization issues. So
you get the worse of both worlds. Just something to keep in mind.

~~~
mrbrowning
> _I think the first thing that should be mentioned in any introductory
> article on Python threads is that Python threads work great for IO
> concurrency but they won 't help with CPU concurrency. Things like
> downloading files, sending data over a socket will work nicely._

Python's threads certainly work _acceptably_ for IO-bound purposes, but given
the overhead of creating a real OS thread and the potential for GIL thrashing
when using Python 2.x on a multicore machine, I'm not sure why you wouldn't
favor a greenlet-based solution in most such cases, especially since you don't
even really have to drop the threading idiom to do so.

~~~
rdtsc
Good point. I use gevent and now switching back to eventlet. But that is a
different post perhaps.

Not only do you get more threads with greenlet you also don't have to worry
about a whole class of synchronization side-effects since a greenlet will only
switch contexts on an IO operation. (Now some might argue that is bad since
you could be calling a function and not know what happens in side or what
might happen in the future so you should lock anyway).

~~~
mrbrowning
Out of curiosity, why the switch? I went with gevent pretty early on because
it seemed like eventlet had some weird quirks, but I have to admit never
really giving it a close look.

~~~
rdtsc
Because it is supported in older Pythons on some servers I work on. Works with
PyPy. It has less dependencies (just greenlet) and easier to build.

gevent also has been thrashing around is it 1.0 beta? Switching to libev or
libevent. And eventlet picked up more steam with more test coverage.

So no one big issue just a bunch of small ones.

------
iffycan
The author mentions Twisted, and I wanted to show what it would look like to
use Twisted:
[https://news.ycombinator.com/item?id=6220124](https://news.ycombinator.com/item?id=6220124)

------
jmspring
Even old, the examples given are good for those not familiar with Python
threading. IBM Developer Works has been a surprisingly decent site for
assorted topics/examples over the years.

------
japaget
The article is from 2008, so it applies to Python 2. I don't know enough about
threading to say whether the same code will work in Python 3.

~~~
briancurtin
It will. If you just tweak some things in the examples like urllib2 imports
and print-as-a-statement usage, the threading API is the same so it's still
generally relevant.

