

The Twisted Way - pablohoffman
http://glyph.twistedmatrix.com/2012/12/the-twisted-way.html

======
vosper
Twisted is the least accessible of the big Python libraries that I've come
across. I've come to it a couple of times when it seemed like it would suit a
problem I had, only to throw up my hands in frustration and go look for a
solution somewhere else.

Unfortunately this article did nothing to enlighten me - it has some
pretensions of being philosophical without explaining any of the
practicalities.

Does anyone use Twisted in production? Do you consider it worth the learning
curve, and would you use it again if you were implementing your system now?

~~~
influx
I use Twisted and love it. What really helped me was reading the source code
instead of trying to decipher the documentation. The code is really readable
and things started clicking once I made that investment.

~~~
vosper
If you don't mind - what did you build with it? Was Twisted the only suitable
library - what about Tornado?

I'm curious because Twisted has been around for ages and seems to be in active
development but I've never talked to anyone who actually uses it.

~~~
lvh
It is my understanding that Spotify is (or at least was, when I saw their
presentation), pretty much a Giant Twisted App with some C++ for parts that
were too slow.

Lucasfilm, justin.tv, Launchpad and TweetDeck are some of the larger names you
may have heard of. A bunch of smaller startups do too.

If you've been doing XMPP you've probably used a Twisted server (it's either
that or ejabberd, mostly) at some point.

If you want Tornado, why not use Cyclone? It's a port (to Twisted) of Tornado;
you get to write webby parts just like tornado's and you get the benefit of
all of Twisted when you need it.

FWIW, I did the Twisted Help Desk and co-hosted the Twisted tutorial two years
in a row now at EuroPython, and two years in a row that tutorial has had to
move to a bigger room. Asking around showed a large portion of people were
there because there place of work was either using or considering using
Twisted. I never cease to be surprised by the stories I hear of people using
Twisted to great success for pretty nifty applications.

------
pekk
It's a library, not the Tao. And it is a big, complex library which introduces
plenty of its own unique jargon and abstractions, which is why people find it
confusing.

The fact that Twisted advocates have to keep browbeating others to accept it
is just one indication that it's not perfect and it's not for everyone.

------
rhettg
I've used twisted for semi-major project (<https://github.com/Yelp/Tron>).

It's certainly a difficult thing to get your head around. The parts of the
project that touch twisted is always the most complex to explain to anyone
else. However, async code is always hard. It also doesn't help that the code
base shows it's age.. non-PEP8, strange naming conventions etc.

However, one thing that I've noticed with recent projects using Tornado
(<https://github.com/rhettg/Dynochemy>) is that some concepts that I thought
was overly complex in Twisted makes sense eventually. For example, handling
errors in async code is really hard. The framework twisted came up with with
Deferred, callbacks and errbacks is pretty smart. You almost have to re-invent
twisted before you can appreciate it.

I'd recommend reading the original paper for more information:
<http://www6.uniovi.es/python/pycon/papers/deferex/>

------
ishbits
I used Twisted in 2005 to develop and server and agent infrastructure..
Portions of the system are now in Java, but the distributed agents are still
in Python and out there in production..

I am currently not using Twisted for anything, but often think that things I'm
currently working on would be easier if done in Twisted.

I came to Twisted after 5 years of doing non-blocking servers in C with
libevent (or hand rolled event loops). So I didn't have to wrap my head around
don't call us, we'll call you aspect. I'm currently using Tornado for a
websocket based service, but will probably use Twisted again for my next
Python project.

------
inglesp
For anybody wanting to learn Twisted, I can only really recommend the
excellent introduction to asynchronous programming (with Twisted) here:
<http://krondo.com/?page_id=1327>

------
sneak
Twisted is great. The only real problem comes from their utter unwillingness
to accept less-than-100%-perfect ipv6 patches. It's 2012 for fuck's sake.

------
leoliu
I am more interested in a post about what's wrong with twisted. Anyone care to
comment?

~~~
w0utert
Having used Twisted for a few projects and now using Pyramid for another, I'd
say there are two big problems with Twisted that kill the experience for new
developers:

1) The documentation is terrible.

Even though there's a lot of it, and even though the API documentation is
pretty good, the 'narrative' and tutorial-like documentation is downright
horrible. It tries to cram so many things into the same contrived examples at
once, that reading it creates more confusion than giving insights. Take for
example the 'evolution of a finger client', which is one of the first things
you get directed to when you want to start 'learning twisted'. Instead of
introducing some basic concepts that affect design decisions and then working
them out in simple, individual, small examples, it starts out with a server
example so trivial that it's completely un-representative of how a typical
Twisted service would look like, and then, in a few pages, crams so many
things into it at once that you have to go back, re-read and actually try to
implement and run the example yourself to figure out what (and especially:
why) the heck things are working like they are working. In one swell swoop it
'teaches' you Twisted by throwing around component-based programming, Zope
interfaces, adapter classes, the factory pattern and how you can basically use
it everywhere even though there's no real reason to do so, how you can expose
4 different protocols talking to the same service at once, how you can add
asynchronous versions of the same services, and so on. All of it combined in a
single 'Twisted application' that reads like some kind of intentially
obfuscated crypto-program that somehow magically does all these things.

Compare to that the Pyramid documentation, which is absolutely fantastic and
got me up to speed and writing very clean and stable applications in just a
few hours. Where reading the Twisted docs the main thing going around in my
head was 'wtf do I have to do here and why', while with Pyramid it everything
immediately made sense. Note that I'm not saying this is a problem with
Twisted itself, just with the way the documentation tries to explain how to
use it.

2) Twisted doesn't advertise its specific benefits and downsides very well.

Superficially looking, many people might think Twisted is just like other web-
application frameworks like Pyramid, Flask or Django, but it isn't. You can do
most of everything using Twisted, but it's much lower-level than those. Just
like this article mentions, Twisted is more like a toolbox to create event-
based applications than it is like a web-app framework. This makes it a lot
more flexible, but also more complex. While this allows all kinds of use cases
that nobody would ever consider e.g. something like Flask or Django for (e.g.
game servers, chat servers, etc), it also means that if your application is
'yet another web application', the other options are probably going to be a
lot simpler and cleaner than using Twisted.

I think because Twisted doesn't want to choose and declare what it's main
strengths and purposes are, it's attracting many developers of applications
that are a much better fit for the more specialized web-app frameworks,
leaving them with a really bad experience, spending too much time figuring out
the framework and ending up with cryptic gibberish that doesn't really
implement any actual application logic.

Don't misinterpret this as a critique of Twisted itself, I still like it a
lot, exactly because it is so flexible and generic. It's just not the best
tool for most web applications, but if if you are implementing some kind of
event-based application that doesn't fit the philosophy of the other popular
frameworks, it's extremely powerful and well-designed. I think the Twisted
community should make this difference more explicit for new developers.

~~~
vosper
Your analysis of the problems with the tutorial is spot-on. It also really
doesn't help that it's presented as "this is a description of what's changing
on this page, and here's wall of updated code - go figure out the differences
from the previous page"

------
cmccabe
GREAT SOLUTIONS IN ENGINEERING

Problem: bicycle seats are _hard._ They hurt.

Analysis: there must be something wrong with your pants.

"Solution": dorky pants.

from <http://dilbert.com/strips/comic/1994-07-18/>

Problem: Python has poor support for threads, and concurrency in general.

Analysis: you must need more middleware.

"Solution": incomprehensible callback-based frameworks.

~~~
readme
If by incomprehensible, you mean you need to read more than 10 pages of
documentation to understand it, then sure.

~~~
pyalot2
There are two problems with callback based frameworks.

#1 In python specifically, we do not have proper closure and no anonymous
blocks, this makes operating it tedious as the definition of the logic for a
callback is always somewhere else from where the callback is setup.

#2 In any language supporting proper closures (javascript for instance, hello
node.js) excessive, nested use of callback based frameworks leads to the
phenomenon of the pyramid of death, whereas you keep nesting closures and you
have to keep them all in one place because each depends on its outer scope.

I intently dislike twisted and node.js for these very reasons. I've written my
own little framework based entirely on greenlets in python, which is
delightful and easy to use and entirely avoids both dislocality of action and
the pyramid of death.

~~~
lmm
What do you mean by greenlets?

Twisted feels to me like a grand experiment that's groping towards something
that we know is there, but not quite catching it yet. I guess I/O monads feel
like slightly further along the path - but they're still not the true
solution. What does your system look like?

~~~
lvh
Greenlets are one-shot coroutines (with no implicit scheduling). They
generally work with a C extension that slices the stack.

In IO frameworks that are greenlet based, the greenlet is resumed when it has
stuff waiting for it.

The obvious difference to the programmer is that a greenlet won't have yield
statements in them, whereas a generator would.

