
Python Coroutines with Async and Await - doublextremevil
https://lwn.net/SubscriberLink/643786/9c0bd83dff0df3b8/
======
PopsiclePete
The async/await model gaining traction makes me really sad. I use it all the
time and it just seems like a giant hack, a band-aid to hide "scary
complexity" from the developers. A typical Microsoft approach to things, but
it should stay there.

First issue - the necessity for at least _two_ of each method: "DoSomething"
and "DoSomethingAsync". Weak. Now as a library developer, you just doubled
your work. Sure, sometimes one method could wrap the other. But it's still
weak sauce.

I much prefer the Go model - you _only_ code "sync" API's, and let the
consumer of your API wrap it in a go-routine and make it "async" if they wish.
Much simpler, better, composable.

Async/await is also gaining traction in Javascript. Really, really sad we
still try to avoid concurrent programming and how to do it _right_ vs easy.

~~~
jcranmer
I disagree. What you're arguing, in effect, is that async/await is bad because
you should handle asynchronous APIs by taking a synchronous API and running it
in a different thread, which experience has generally shown to be a recipe for
buggy code, inefficient code, or most likely both.

The underlying idea behind async/await is that synchronous methods shouldn't
exist in the first place--you'd only have the asynchronous API. Async/await
enhances that model by turning the promises' functional interface (which
requires interesting binding to handle loops/recursion) back into an
imperative control interface.

~~~
PopsiclePete
Not true from what I've seen _Every_ C# library with support for async/await
has synchronous versions of those same methods. In fact, Microsoft's own
documentation recommends appending "Async" to the method name for precisely
that reason - to distinguish an async method from a sync method.

Furthermore, "asynchronous" code doesn't need separate threads. A scheduler
and green threads is more than enough.

Have you seen Go? You should take a look at it. It makes writing concurrent
code basically trivial, using "green" threads and channels and a neat little
"select" statement.

~~~
giulianob
They only support both because most libraries still support older version of
.NET

If they were to support only .NET 4.5 or higher, then you can just call any
async method in a sync manner simply by doing httpClient.GetAsync(...).Result

async/await in C# is a beautiful thing.

------
tveita
The Python standard library comes with support for a lot of common network
protocols - TLS, HTTP, SMTP, etc.

I'm wondering how this will be handled in async. Will everything need to be
reimplemented as separate async implementations, or is there a way to have a
common implementation that supports both sync and async? Will e.g. async HTTP
be added to the standard library?

I think this was a topic of discussion when they first added asyncio, but I
never saw an official decision.

------
strommen
It's fascinating how async/await are spreading from C# to other languages like
JavaScript and Python. As far as I know, no languages had anything like it
before C# added it in ~2011.

What was the last truly-new language feature that became popular across other
languages like this?

~~~
ceronman
The idea of using coroutines for asynchronous programming is quite old
actually. ML has used it since the 80s [1]. Python itself had it way before C#
with Twisted's inlineCallbacks [2].

What C# (actually it was F#) introduced was the specific 'await' and 'async'
syntax keywords. Those are being copied in other languages, and they're just
sugar over coroutines.

[1] [http://www.cambridge.org/gb/academic/subjects/computer-
scien...](http://www.cambridge.org/gb/academic/subjects/computer-
science/distributed-networked-and-mobile-computing/concurrent-programming-ml)

[2] [http://blog.mekk.waw.pl/archives/14-Twisted-
inlineCallbacks-...](http://blog.mekk.waw.pl/archives/14-Twisted-
inlineCallbacks-and-deferredGenerator.html)

------
gamesbrainiac
Python 3.5 is turning out to be a version that people might actually be
interested in.

~~~
pekk
I don't see why this should be a tipping point; this introduces no new
capabilities, and a lot of hairy new syntax and magic methods.

~~~
nine_k
Async operations were the one biggest hurdle when writing efficient I/O-bound
programs. (Definitely, Python is not about efficient parallel CPU-bound
programming, but it's fine for many I/O-bound tasks.)

------
mhomde
Having used async & await extensively in C# they're pretty great... but I
can't help thinking we're explicitly declaring something that mostly could be
implicit if the language was designed right.

For example: if I'm calling an IO method that returns a value it should be
able to figure that out and orchestrate the code to run asynchronously and
await for the value when it's referenced. If I have several asynchronous calls
methods in a method it should be able to optimize it so they all run in
parallel as optimized as possible.

It might need some added declarative syntax on methods but I'd rather prefer
that and make the compiler smarter, instead of offloading that complexity on
the programmer per default. Sure you should be able to explicitly control
asynchronicity if needed but in most cases that doesn't seem optimal nor
necessary

Adding async/await all over your code adds to the "crud" that makes code more
verbose and harder to read and understand

~~~
tveita
The main purpose of the async/await keywords is to help the programmer reason
about what can happen when.

Having calls be implicitly parallel makes it difficult for the programmer to
predict what could be happening elsewhere in the program. E.g., this passage
from the Oz documentation[1]:

"Oz 1, supports a fine-grained notion of concurrency where each statement can
potentially be executed concurrently. This results in a fine-grained model
similar to the actor model. A good exposition of the Oz 1 programming model is
given in [Smo95]. Our experience using Oz 1 showed that this kind of model,
while theoretically appealing, makes it very hard for the programmer to
control the resources of his/her application. It is also very hard to debug
programs and the object model becomes unnecessarily awkward."

For I/O in particular it is also difficult for the machine to know what is
safe to do in parallel. For instance, two writes to the same file probably has
to happen in order. What about writes to different files? What about HTTP
requests? Most I/O has the potential to produce wrong results if rearranged in
arbitrary order, and requiring the programmer to mark them sequential becomes
a potential source of error and a cognitive burden.

In most cases it's better to be slow and correct by default, and allow the
programmer to explicitly run things in parallel in the cases it matters.

[1]
[http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node1.h...](http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node1.html#chapter.introduction)

~~~
mhomde
I agree that those can be issues... but that can be solved by annotating the
methods to those constraints. The compiler can even highlight potential
problems and ask that you annotate them.

The benefit of this approach is that the compiler should be able to see more
opportunities for optimization and also result in fewer bugs.

------
kanja
This is pretty great - One feature I really liked from go was channels and the
ability for goroutines to communicate with each other. Anyone know there a pep
open for a similar idea in the python world?

~~~
e271828
You can get channel-like behavior using async/await and a custom event loop. I
wrote a little example[1] that has a bare-bones implementation of this. I used
it to translate the prime sieve example from Go[2] almost directly to Python.
The code uses "message = await channel.receive()" to mimic Go's "message <\-
channel". Instead of using "go func()" to fire off a goroutine, I use
loop.run(func()) to add the PEP492 coroutine to my simple event loop.

It's not an efficient implementation - it was really meant as a proof of
concept that you can use async/await in your own code without any reference to
asyncio.

[1] [https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-
si...](https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-sieve-py)
[https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-
ch...](https://gist.github.com/vrajivk/c505310fb79d412afcd5#file-channel-py)

[2]
[https://golang.org/doc/play/sieve.go](https://golang.org/doc/play/sieve.go)

------
midko
relevant: [http://lwn.net/Articles/644128/](http://lwn.net/Articles/644128/)
("PEP 492 vs. PEP 3152, new round")

~~~
ptx
It's linked towards the end of the article, after it's been explained what PEP
3152 is, so reading it after the main article is less confusing. :)

------
themartorana
Coming from a Go mindset, having moved a significant portion of dev to Go from
Python, in many ways for "goroutines", it seems as if there still isn't a
parallel, unless I'm missing something.

In the simplest case in Go, it's easy for me to toss off some concurrent
processing and not wait or care about what happens in those functions. For all
intents and purposes, it feels like parallel execution (I know, I know).

I can't quite wrap my head around how generators or async/await gives me the
same functionality. What am I missing?

~~~
falcolas
Do you regularly set "GOMAXPROCS"? If not, you're just using coroutines in Go
as well, just with less control over the scheduling of the execution.

~~~
themartorana
I do. But it looks like I'm comparing apples and oranges. Goroutines and
Erlang processes feel very much like lightweight threads. Async/Await still
pause for return, but doesn't block the calling thread (it seems like). I'm
trying to do some research on the patterns (mostly C# vs Go because this PEP
is still newish) and it SEEMS like that's what's happening here.

~~~
Rohansi
Goroutines and C# async/await are quite similar in use and behavior, the
implementations are very different though. Goroutines have their own stack but
Tasks don't, they store values that cross await boundaries in a compiler-
generated class and everything else uses the worker thread's stack.

Async/await doesn't feel like lightweight threads because you need to
explicitly await tasks. You could invert the keyword usage to match Go
(implicit await, explicit go) and end up with something like goroutines.

------
dom96
This is something that a Python-like language by the name of Nim already has.
Interesting to see Python get it now too. [http://nim-
lang.org/docs/asyncdispatch.html](http://nim-lang.org/docs/asyncdispatch.html)

------
lyschoening
How would one await multiple coroutine/futures?

EDIT: await asyncio.gather(fut1, fut2)

~~~
dcarp
Why not: async asyncio.wait([fut1, fut2])

~~~
thomasjames
The async keyword is used in coroutine declarations, so it wouldn't be valid
syntax.

~~~
dcarp
My bad... of course I wanted to write

await asyncio.wait([fut1, fut2])

