
Python Concurrency Decorators - orangepenguin
https://github.com/alex-sherman/deco
======
thebigspacefuck
Another page you might be interested in bookmarking:

[https://wiki.python.org/moin/PythonDecoratorLibrary](https://wiki.python.org/moin/PythonDecoratorLibrary)

A similar decorator to thread function calls for concurrency:

[https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thu...](https://wiki.python.org/moin/PythonDecoratorLibrary#Lazy_Thunkify)

~~~
elpres
Doesn't this second decorator only work with functions doing IO or releasing
the GIL in any other way (like time.sleep() that they use in the example)? If
all the function is doing is actual computation (i.e. using the CPU), no other
code can run in parallel with it because of the GIL.

~~~
thebigspacefuck
Yes, that's right, it is best used for I/O-bound applications. You could use
it for:

* running multiple Database queries

* SSH-ing into multiple devices to run a command

* loading multiple web pages

* calling multiple APIs

------
rcthompson
So, for me, replacing imap with pool.imap is the easy part. The hard part is
dealing with things like handling exceptions, catching keyboard interrupts,
and so on. Does this module do anything to address these issues?

~~~
hyperion2010
I have a hack for dealing with KeyboardInterrupt's on a ProcessPoolExecutor:
[https://github.com/tgbugs/desc/blob/master/util/process_fixe...](https://github.com/tgbugs/desc/blob/master/util/process_fixed.py).
I used this in concert with asyncio run_in_executor which helps with some of
the exception handling.

~~~
lqdc13
That's a nice hack when you have only a few elements in the iterable, but
Futures are pretty heavyweight, so this is unfortunately a lot slower than
multiprocessing.Pool when you have many elements.

The other thing is that this won't work on Windows. Efficient multiprocessing
always a pain in Python in my experience.

------
gshulegaard
A couple of things offhand:

\- It has Python 2 and 3 support

\- It's a wrapper for the Python built-in "multiprocessing" library

\- It spreads out work over all cores (so the abstraction hides the ability to
control the pool)

Seems like a great way to get your feet wet with multiprocessing in Python,
but it likely has limited use in production...although certain infrastructures
like resource limited containers might be able to accommodate it.

~~~
rcthompson
Is there no way to specify the number of cores to use?

~~~
jonesetc
Looks like there is:

[https://github.com/alex-
sherman/deco/blob/cee63391bf4c6d66ee...](https://github.com/alex-
sherman/deco/blob/cee63391bf4c6d66ee40628fb6ab61e62d3bc236/deco/conc.py#L108-L117)

~~~
gshulegaard
Interesting, so when they claim it automatically scales out to all cores what
they mean is it defaults to 3 unless overridden.

~~~
lqdc13
Should really be multiprocessing.cpu_count() / 2 or something like that.

~~~
xapata
I found out recently that some tasks do better if there are slightly more
processes than cores.

------
japaget
See paper at
[https://drive.google.com/file/d/0B_olmC0u8E3gWTBmN3pydGxHdEE...](https://drive.google.com/file/d/0B_olmC0u8E3gWTBmN3pydGxHdEE/preview)

------
Klasiaster
With Python 3.5 there is native support for concurrency by using the keywords
await and async.

For simple usage if you are familiar with Go there is this library:
[https://github.com/pothos/awaitchannel](https://github.com/pothos/awaitchannel)

~~~
zardeh
Asynch and concurrency are very different. Python 3.5 allows first class
aysnchronous calls, but concurrency is still "hard".

------
tomkinstinch
Of note is that the version of deco on PyPI[1] (0.2) is incompatible with
Python 3.

There have been few commits to fix compatibility, but it's not there yet.

1\. [https://pypi.python.org/pypi/deco](https://pypi.python.org/pypi/deco)

------
mpdehaan2
Perhaps a strange choice of the word "synchronized" when coming from Java,
this typically implies a critical section. Here it seems to initialize a
multiprocessing pool for use in the function labelled concurrent (perhaps)?

~~~
chrisseaton
I think it's like sync in Cilk, meaning that all concurrent jobs must have
finished before the part that is annotated synchronised is left. So it's not
without precedence.

------
cyberfart
"We have proposed DECO, a simplification of concurrent programming techniques
targeted at programmers with little understanding of concurrent programming."
(from paper)

gave me the chills

------
hcarvalhoalves
Related: [https://github.com/hcarvalhoalves/python-
pmap](https://github.com/hcarvalhoalves/python-pmap)

------
awinter-py
this is such a good idea! the @synchronized decorator to collect the
parallelized task at the end of a parent call is very very smart & simple.

~~~
nomel
It also has a potential for slowing down your code if used willy nilly, of
course.

Multiprocessing gets pretty useless for anything outside of independent CPU
bound tasks with little IPC and simple data types that can be stuck into
shared memory.

If you're using multiprocessing pools so often that you think you need a
decorator to clean up your code, then wow, I'd like to see what you're up to.
;)

~~~
m_mueller
> If you're using multiprocessing pools so often that you think you need a
> decorator to clean up your code, then wow, I'd like to see what you're up
> to. ;)

.. and I think once you do that in Python you should probably use NumPy.

I've been thinking that there should be a way to just program kernels in
Modern Fortran because it's the easiest to interact with NumPy data structures
(NumPy is nothing but glue code around a collection of very efficient Fortran
numeric code). f2py [1] is doing that basically, but I've never had the chance
to setup a project like this.

[1] [http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.python-
as...](http://docs.scipy.org/doc/numpy-1.10.1/user/c-info.python-as-glue.html)

------
tbarbugli
source looks surprising, for instance the decorators parse the decorated
function code and build an AST (still have to find out why)

~~~
madisonmay
My best guess is that this is to circumvent issues with naming encountered by
using decorated functions in conjunction with multiprocessing pools. Typically
you would run into serialization errors.

~~~
tbarbugli
it turns out that the decorator "rewrite" the fn code. that's at least how the
synchronized decorator does its own magic. Once I saw that in the source code
(and other few hacks) I was sure I was never going to use this lib :)

------
bmh_ca
This reminds me of AppEngine's ndb tasklets.

------
tacos
I've been looking for a way to replace functools.partial and pool.map with
something that could cause me to make bad architectural decisions. This could
be the ticket.

~~~
madisonmay
Haha, having mucked around with similar mechanisms for dealing with
concurrency via decorators I have to agree with you that this is likely to
cause hard to debug behavior (especially since direct modification of the AST
is going on behind the scenes here). That being said, it's an interesting
thought experiment and likely an excellent class project.

------
eximius
Let me sum up my feelings towards this: neat!

------
lormayna
Then I can easily parallelize existing code without touching anything?

------
max_
I love developing in Flask cause of the way decorators written.

