
Guide to Concurrency in Python with Asyncio - LiamPa
https://www.integralist.co.uk/posts/python-asyncio/
======
jonahbenton
Forgive me, but this is such a ball of mud. All of the "easy" introductions
into these primitives are basically whitepaper length, with long digressions
into the high-level vs the low-level and historical vs modern patterns. And
nothing gets into error cases or problems with cooperative scheduling or
debugging/troubleshooting or where the GIL applies. In 5 years debugging the
rats nest of concurrent python code that people are writing right now will
make clear that go really got this right. So sad that python did not.

Edit: elsewhere on HN right now:

[https://nullprogram.com/blog/2020/05/24/](https://nullprogram.com/blog/2020/05/24/)

~~~
j88439h84
Agree, asyncio is a mess; Trio gets it right.

[https://trio.readthedocs.io/](https://trio.readthedocs.io/)

It's a much simpler model that's just as powerful, and makes it easy to get
things right. It eliminates the concepts of futures, promises, and awaitables.
It has only one way to wait for a task: await it.

For a theoretical explanation of it's "structured concurrency" model see the
now-famous essay [https://vorpus.org/blog/notes-on-structured-concurrency-
or-g...](https://vorpus.org/blog/notes-on-structured-concurrency-or-go-
statement-considered-harmful/)

~~~
devxpy
Nathaniel has done a great job of drawing parallels between go and goto.

I've been stung, time and again by asyncio.create_task() swallowing
exceptions[1], and functions not telling callers about background tasks[2].

For others looking to NOT rewrite all their code using a 3rd party event loop,
here's a simple reproduction of nursery's behaviour in raw asyncio -

Hopefully, this will help lift the curse of asyncio :)

[EDIT] A library is also available -
[https://github.com/Tygs/ayo](https://github.com/Tygs/ayo) (with amazing
operator overload)

    
    
      async def main():
          async with Nursery() as ny:
              ny.start_soon(a_1(), a_2(), ..., a_n())
              ny.start_soon(b_1(), b_2(), ..., b_n())
              .
              .
              .
              ny.start_soon(x_1(), x_2(), ..., x_n())
    
    
      class Nursery:
          def __init__(self):
              self.tasks = set()
      
          def start_soon(self, *coros: typing.Coroutine):
              for coro in coros:
                  self.tasks.add(asyncio.create_task(coro))
      
          async def __aenter__(self):
              return self
      
          async def __aexit__(self, *args):
              try:
                  while self.tasks:
                      tasks = self.tasks
                      self.tasks = set()
      
                      done, pending = await asyncio.wait(
                          tasks, return_when=asyncio.FIRST_COMPLETED
                      )
      
                      try:
                          for task in done:
                              await task
                      finally:
                          self.tasks |= pending
              finally:
                  for task in self.tasks:
                      task.cancel()
    

[1] [https://stackoverflow.com/questions/60287285/starlette-
async...](https://stackoverflow.com/questions/60287285/starlette-asyncio-
create-task-doesnt-log-error-if-task-object-is-stored-in)

[2]
[https://github.com/encode/starlette/issues/947](https://github.com/encode/starlette/issues/947)

~~~
j88439h84
trio_asyncio lets you write trio code that uses asyncio libraries.

------
parhamn
This looks great. I wish python would have made the event loop generally more
transparent to the end user. Why didn't they just use a single global loop?
Sorta like javascript and golang. It would have been more pythonic too. Anyone
doing heavier context switching could've had their own loop management.

Making the event loop self-managed added a ton of clunkiness in aio apis (use-
your-own-loop, loop lifetime management, etc) and becomes mentally complex for
newcomers. Theres also the issue that these huge aio frameworks rewriting the
same TCP clients have emerged only differentiated by their loop management
patterns.

~~~
bluedays
The more time I spend with Python the more reasons I find to hate it.

However, it's still my main language. Everything I end up making just ends up
being written in Python because that's the language I know the best.

I could learn another language, and I have learned other languages, but Python
seems to be the language that mentally clicks with me the most. I think it's
because it's the first language I learned.

Suffice to say, there are so many things wrong with Python and they're all
compromises that were made to appeal to wider audience. Unfortunately, I don't
think that the "wider audience" is those who are looking for concurrency.

~~~
parhamn
We agree. I think python is great. I actually asked because python usually
makes things dead simple and straightforward.

> Unfortunately, I don't think that the "wider audience" is those who are
> looking for concurrency.

I don't buy this. Many network heavy shops use Python so concurrency is
relevant. The language is used in web, finance, cloud, data pipelining/etl,
and so many more. Dropbox, Robinhood, Spotify, Instagram, Uber, Google, etc,
are the "wider audience" and they certainly care about concurrency.

------
mcdermott
Sorry, but after using Golang with its very simple and powerful goroutines and
channels approach to concurrency, this seems like a convoluted mess. When I
need concurrency, I certainly don't think of Python as the right tool.

Python seems to have lost its way after the 2 to 3 shift and is no longer what
I'd consider "pythonic".

~~~
wegs
I think programming language design as a having a large random component. New
languages pop up every day, and it's really hard to predict which constructs
will resonate well with human brains.

Guido was competent, but not nearly as brilliant as, for example, Larry, but
Python 2 was dramatically better than Perl. I think this is mostly by chance.
Hundreds of new languages come out every year, and by whatever fluke, a few of
them really work. Python 2 was that.

In an ecosystem like that, past performance is no indication of future
performance. There are very few language designers who did well more than once
(Guy Steele is the only one I can think of who wasn't a one-hit-wonder).

So I too felt like Python 3 was more of a step backwards than forwards
linguistically. But at this point, it has features I need which Python 2
lacks, so I'm mostly working in Python 3. But it definitely feels
(unnecessarily) less clean.

~~~
metreo
Python's origin story as being a replacement to Perl is so completely out of
scope compared to what modern scripting languages are doing it's laughable.

------
css
One thing I haven't seen any blogs write about is that multiprocessing in
3.8.x uses `spawn()` and not `fork()`[0] on MacOS. Granted, most applications
are not running on OS X Server, but an update that changes a low level API
like that led to some issues where running code locally will fail when running
it remotely will work. The following code will run anywhere except on MacOS on
3.8.x, where it crashes with `_pickle.PicklingError` as it tries to resolve
the name `func` that is not in the new stack:

    
    
        import multiprocessing
        some_data = {1: "one", 2: "two"}
        func = lambda: some_data.get(2)
        process = multiprocessing.Process(target=func)
        process.start()
    

[0]: [https://bugs.python.org/issue33725](https://bugs.python.org/issue33725)

~~~
jamestimmins
I'm in the process of adding 3.8 support to Airflow. This was the first (but
not the last) obstacle in doing so.

------
birdyrooster
To those trashing Python in favor of Golang: A compiled language with no OO is
not a replacement for Python. Let's talk about the complexity of putting code
generators in all of your projects. I've seen real golang projects and the
complexity gets moved into your repo.

~~~
rumanator
> To those trashing Python in favor of Golang: A compiled language with no OO
> is not a replacement for Python.

Why?

I mean, come on. Who in their right mind would ever claim that Go is not a
good choice to develop web services?

And are we supposed to turn a blind eye to Python's problems such as sub-par
performance and the GIL?

I understand how Python fanboys will pick Python over alternatives whether
that makes technical sense or not, but asserting that any compiled language
cannot be used to deliver software that has been implemented with Python, and
that explicit support for OO is a relevant requirement, is something that
flies in the face of reason.

~~~
birdyrooster
With some very simple sharding (we are talking splitting lists and
dictionaries) you can bypass the GIL and not balloon your memory usage. To do
so you would use aiomultiprocess which inherits from asyncio classes (and
therefore interfaces) but lets you await on methods running in their own
processes.

[https://github.com/omnilib/aiomultiprocess](https://github.com/omnilib/aiomultiprocess)

------
michalc
Shameless plug for what is essentially my own much shorter intro to asyncio:
[https://charemza.name/blog/posts/python/asyncio/I-like-
pytho...](https://charemza.name/blog/posts/python/asyncio/I-like-python-
asyncio/)

------
theelous3
Or just use curio or trio and have an infinitely better time with async in
python :)

~~~
nurettin
I see no reason to use curio or anything instead of standard asyncio. Works
really well for me. Queue tasks, they finish at their leisure, create futures
and locks to control flow, all really easy and bugless code. And you get async
redis, async http, a lot of asyncio functions for free. So no, I will not
change it for some "really well architected library that takes away all your
pains" because there aren't any.

~~~
downerending
The reason is that _curio_ is well-designed and has a reasonably compact story
for how things work. It fits in my head. And the "cool new things I can do
with it" to "how hard is it to learn" ratio is pretty high.

Asyncio on the other hand, feels like being beaten with a stick. It's so
complicated that normal humans are never really going to understand it all.

On top of that, it's been morphing all through the 3.0 series, so there isn't
even one spec to learn--it's more like a set.

~~~
nurettin
What do you mean compact story? How do humans not understand asyncio? I think
I pretty much do, and I use it in production, with several interconnected
services. How has curio been morphing? How is morphing actually a good thing?
Isn't it bad that an api is unstable?

Overall I understood little to nothing from your comment.

~~~
downerending
The curio module has _not_ been morphing, which is part of what I like about
it.

The async module _has_ been changing, which like you, I dislike. The "right"
way to do _asyncio_ would have been to properly mull the design into its final
form and then introduce it into Python3. Instead, it appears to have been
dribbled in piecemeal over several releases.

Beyond that, it's just plain complex and confusing. I haven't bothered to use
it for anything so far. I'm hoping someone will show up with a machete and cut
it down into something usable. Right now, it resembles C++.

~~~
nurettin
Yeah I haven't actually used it until 3.6.9 where it turned into its current
final form. What is confusing about asyncio and how does curio solve that
problem?

------
adonese
There's this really interesting blog post Im not remembering about nodejs and
async concurrency style as compared to golangs one. I cannot remember the
author's name, but it goes into length as of comparing async functions and the
differences between normal regular func and the async ones, and calling async
func only on async scope and so on. Only I remembered I found it retweeted by
Brad Fitz. I though maybe someone could guide me to it here

~~~
bobbiechen
Possibly "What Color is Your Function"? One of my favorites on concurrency
approaches. [https://journal.stuffwithstuff.com/2015/02/01/what-color-
is-...](https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-
function/)

------
migueloller
I've found this [1] screencast on the low-level principles of async in Python
an amazing resource to understand async at a deep level.

[1]
[https://www.youtube.com/watch?v=Y4Gt3Xjd7G8&list=PLKLmIMXskJ...](https://www.youtube.com/watch?v=Y4Gt3Xjd7G8&list=PLKLmIMXskJQ9WpioXF4LPJJ6Tp9hbDXg-&index=2&t=0s)

------
citrin_ru
I don't see anything about I/O in this guide.

Can anyone recommend a guide which explains how to actually do I/O using
asyncio: how to accept connections, how to write to and read from a socket?

------
mkchoi212
Every time I see concurrency and python together, I’m immediately turned off
by it. First of all, it probably won’t be a true “concurrency” due to GIL and
by the point you are looking for “advanced” libraries that allow you to do
whatever you are trying to do, maybe it’s time to switch languages.

~~~
nurettin
Did you mean: not true parallelism?

------
dilandau
Asyncio detractors have been drowned in a stream of assurances that, really,
it's not that bad.

Obviously it is, or this wouldn't be post #800 that promises to finally make
asyncio clear to newcomers.

Twisted sucks. It was a joke. Now it's been tacitly blessed to the point you
can sprinkle the `async` keyword all over the place. Good luck. I hope you
don't forget about any spots.

I truly think part of the reason Guido left is because he couldn't defend this
shit.

~~~
nessunodoro
Not to stoke the flame, but I never used Twisted, where did it fail?

~~~
carapace
Bad docs.

Everything else about it is solid as hell.

 _Pathlib_ comes from Twisted IIRC.

