
Ask HN: Are you doing async programming with Python? - hguhghuff
If yes, what are you building, and what’s your opinion of async Python?
======
nothrabannosir
Bit of an unconstructice rant, but this always bothered me: Async is the nail
in the coffin of The Zen of Python.[1]

Open a python REPL, type:

    
    
      >>> import this
    

_…_

 _There should be one—and preferably only one—obvious way to do it_

 _…_

It has not been true for years. Python 3 especially, took a sledgehammer to
that ole poem.

See [http://journal.stuffwithstuff.com/2015/02/01/what-color-
is-y...](http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-
function/) which is about this effect. I am reminded of that blog post every
single time I work with async in Python :/

[1]
[https://www.python.org/dev/peps/pep-0020/](https://www.python.org/dev/peps/pep-0020/)

~~~
jzl
The story of "import this": [https://www.wefearchange.org/2010/06/import-this-
and-zen-of-...](https://www.wefearchange.org/2010/06/import-this-and-zen-of-
python.html)

~~~
vram22
There's also:

from __future__ import braces

and

import antigravity

------
siddhant
I've been writing web apps using Tornado since 2013 and that has been an
excellent experience.

But I'm not sure if I have the same opinion of asyncio. To be honest, I don't
even know if I understand it completely. There are so many concepts and IMHO
they're not really intuitive, so it's difficult to put pieces together and
find out how those individual concepts relate to each other or work with each
other.

On the surface it looks simple, but it's really like going down a rabbit hole.
The more I try to understand it, the more I agree with Armin Ronacher's
article on asyncio [1].

[1]: [http://lucumr.pocoo.org/2016/10/30/i-dont-understand-
asyncio...](http://lucumr.pocoo.org/2016/10/30/i-dont-understand-asyncio/)

------
gwillz
Yes - and it was great fun!

There was a post on HN that was asking about real-world examples of asyncio
code that wasn't `asncio.sleep()`. I think my project is pretty good example
for that - and a strange example too because it involved a lot of hardware
interfaces.

I've been meaning to open-source the project ever since we axed it. I was the
only real programmer on the job, so I'm itching for feedback on it. So if
anyone's interested I'll post it here.

I had actually written it 3 times over. First as raw socket server, then in
twisted, then on top of asyncio.

So yes, asyncio is a bit of a mess. It's documentation is a bit odd and I have
several gripes. What helped we was realising that there was an 'async way' and
an 'callback way' to everything in asyncio. So not mixing the two together and
ignoring the callback parts really _really_ helped.

I don't know if this is true, but I recall reading somewhere that asyncio was
not designed with coroutines initially and this explains the dual-personality
of the library. I can't find any evidence that asyncio existed before 3.3, so
I can't back that up.

~~~
personalcompute
I would be interested in seeing and reviewing your asyncio work gwillz! I have
a small (750 sloc) network service implemented with asyncio and ran into some
design problems around modularly handling healthchecking, exception
handling/monitoring, and auto recovery/restart. I was unable to find much in
the way of large software written with asyncio to learn better patterns from,
so ended up with what, to me, feel like mediocre solutions. My email is in my
profile.

------
karanlyons
Yes, and it is...not great. It's generally fine as far as the language goes
(considering it's a retrofit), but the builtin asyncio library is not the way
to go in my opinion. trio is _far_ better for just aping a lot of the Erlang
way of things.

But even then it's hard to debug, or work out ideas in the REPL, testing is a
bit more complex than it should have to be, etc. There are probably better
languages to reach for (I'd love to see rust firm up a one true way of async).

~~~
adamcharnock
FWIW, I was at a talk by an asyncio core developer at EuroPython and he said
they are looking very closely at how to improve the asyncio API in upcoming
3.8/3.9 releases. In particular, looking at good ideas they can pull in from
libraries such as trio.

Maybe Python 4 will mark a maturing of asyncio?

------
petr-k
Yes, and it has been a good experience so far. Mostly we've been doing
applications involving several RabbitMQ producers/consumers, a HTTP server and
database access here and there. It relies on the fact that there already are
asyncio-enabled libraries for the particular technologies we routinely use.

Note that our use case might be different to many in this thread - in contrast
to HTTP, we write mainly event-driven message-oriented applications. It is
often practical to have e.g. multiple different consumers within one process -
without async, we'd need to resort to threads which brings a plethora of new
problems. Async Python does offer a good solution for this.

That said, there are a lot of places where async could be improved.

Language-wise it's pretty okay, but the asyncio library contains too many
concepts (some perhaps introduced historically and now deprecated) and the
docs aren't doing very good job explaining those. IO loop management could be
simplified and some of the related functions should have different, better
names. Error messages and stack traces could also be better when async is
involved. As far as testing goes, pytest-asyncio could also enjoy some love.

But overall, no major complaints. Few catches aside, it works pretty much the
same as in C# or JavaScript syntax-wise, which is what you would expect from
this approach to asynchronous programming.

------
aequitas
You might want to ask this question over at [https://github.com/home-
assistant/home-assistant](https://github.com/home-assistant/home-assistant).
They have been porting most of their codebase from threads and such to Python3
async over the past months/years. I'm sure the've hit almost all corners of
async doing that.

I did some small contributions for HA components and didn't have to get deep
into async just yet, but my general feeling is that it is less 'natural' that
async programming in a language like Go. But Go has the bigger advantage is
was born with async in mind, so it doesn't lag behind that much with 3rd party
libraries support like Python does.

------
euphetar
Honestly, no. Despite all the supposed advantages of async, I haven't found a
use for it. When it comes to web apps a nginx + uwsgi + a sync framework is an
unbeatable combination. The only times I have used async were when I made
website scrapers and telegram bots for And if I was to scrape websites now, I
would just use scrapy. My favorite telegram bot framework is synchronous too.

~~~
siddhant
There are some use cases in web programming. For example, if a request handler
(view, in Django's terminology) is in turn calling out some external
endpoints, you don't want those external requests to block other requests to
your API. For things like those, async is actually quite nice.

~~~
nsomaru
You can mix asynchronous code with Django?

~~~
dec0dedab0de
You kick off an async task in celery or use some other way to keep track of
it. Then return a key, and have your front end poll for status

------
logronoide
Yes, in [https://apility.io](https://apility.io) all the backend services, API
endpoints at the edge and parsing services take 'advantage' of async in Python
>3.

My opinion is -like somebody said- it breaks the Zen of Python. When I started
using it I had doubts about moving to a more async 'friendly' language. But I
decided to stick with my old friend Python... even when I don't recognize him
very much after some changes after version 3.

------
poooogles
Yeah I'm using it plenty for web services where I've got a concurrent app but
most data processing is done upstream.

It's pretty easy to get started with and with the new keywords simple enough
to teach. The testing story is a bit horrible due to how coroutines work
though, having to create asynchronous True and False when patching returns for
example is a pain.

------
detaro
Using it for what you might describe as API proxies (take requests, fetch a
bunch of other APIs, combine/transform their results to a different format,
deliver to client). Works for that case well, but it's admittedly basically
the perfect application for it.

------
torte
Yes.

Can not really tell what I am working on, but so far I am more or less happy
with it.

It is not as simple as advertised though ("just use async/await"). In order to
write stable, performant applications you have to understand more or less the
underlying mechanisms of asyncio (what are tasks/futures, how does their
lifecycle work). Then there are quite a few things which work not as expected
and you have to understand them as well (e.g. exception handling). If you get
into the nitty-gritty details you will have a good experience with asyncio. If
you don't you better stay away from it or you will have a hard time.

Performance wise it seems pretty good so far btw.. Did not really hit its
limit yet though.

------
pointsphere
I’m a beginner with python as my first language.

I used it to write an arbitrage trading bot. I learned python and asyncio just
for this, and it is working as intended.

Asyncio with aiohttp is used to capture price data from 4 websockets and
handle requests to exchanges via their APIs.

Python was easy to learn with “learning python the hard way”, asyncio was also
easy to learn thanks to lots of available material online, and it looked less
confusing than threading.

Again, speaking as a beginner, I quite like asyncio because to me it looks
just like synchronous code. To me it’s easier to think about and keep track of
the control flow and shared data than with threading.

Asyncio looked like the easiest and fastest way to accomplish my goal, and it
worked.

------
sethammons
We had a couple services in Twisted. Aptly named. It was a big reason I've
enjoyed Go so much.

One was an in-house DB abstraction/API that handled multiple DB hosts,
read/write query routing, connection pooling, query queueing, metrics, and
more. The other was a link forwarding service that took encoded links, decoded
and tracked them and their metadata, and forwarded the user.

------
ceidsness
Yes, we are using it to build a set of system configuration and monitoring
daemons, controlled through D-Bus.

It's okay, it can be a a little confusing at first but worth the time
investment required to understand it. The API has been improved with all of
the recent 3.x dot releases. The new run command in 3.7 should make it a lot
easier for folks to get started.

------
simon_acca
I have implemented the Raft algorithm with asyncio. Although this was an
academic exercise and I would choose something else for real world usage, I
was pleasantly surprised with the performance:
[https://github.com/simonacca/zatt](https://github.com/simonacca/zatt)

------
zawerf
A lot of machine learning datasets only include metadata with a link to the
original source (e.g., a full resolution image which might be too wasteful to
include with the dataset).

So the first step is to write a crawler and async is a really fast way to do
that because everything is network IO bounded.

------
maayank
Async programming in python? yes, as part of distributing parameter finding
for genetic algorithms and backtesting.

Does it use 'asyncio'? Nope.

------
baq
I would if there was an async sql server driver... Thread pools don't cut it.

I don't have my hopes high for this one though.

------
Dowwie
Check out the asphalt project if you'd like to give a native asyncio framework
a try.

------
some_account
I love python but if I wanted a fast async app, I would prefer to write it in
go with goroutines. I don't like the async programming style where every
function involved in the logic chain has to be async. Goroutines with channels
are just nicer.

Python is awesome for a lot of things but go really shines for async style
programs.

