
Twisted Python's lead hacker discusses Tornado - thristian
http://glyph.twistedmatrix.com/2009/09/what-i-wish-tornado-were.html
======
pvg
There seems to be a rather unrealistic assumption that people who don't like
your product (be it website, framework, sandwich) owe you well-reasoned
feedback. It's a great and useful thing to get but the fact is, most people
are interested in solving their problem and unless they are quite sure your
product is the best way to solve their problem, they'll simply move on.

Perhaps this is an artifact of how seriously the Twisted people and their
'lead hacker' take their project. But the twisted-centric view of the world
leads to petulant posts full of odd statements:

 _My main point here is that if you're about to undergo a re-write of a major
project_

They wrote Tornado, they didn't 'undergo a rewrite of Twisted'.

 _For a tiny fraction of the effort invested in Tornado, FriendFeed could have
worked with us to resolve many of the issues creating that chaos._

Interfacing with an another team to solve problems in a framework they didn't
like (even if just aesthetically) would have probably taking significant
resources for close to zero return.

There's quite a bit more in the same vein. It's wonderful that the Twisted
people care. But perhaps a little too much.

~~~
ellyagg
I'd be pretty damn upset too if some other project got a lot of press for an
announcement which included comments undermining the quality of my project.
Sorry, but if you're going to strongly imply that my project's ease of use and
performance are subpar, them's fighting words.

~~~
pvg
The comments were really in passing. 'Looked at it, wasn't for us', more or
less. Perhaps them's fightin' words. But getting huffy about them probably
helps your project even less, given they'd have otherwise gone largely
unnoticed.

~~~
rapind
And not responding to disparaging comments helps your project?

I think he makes a pretty strong case in this article.

It's a bit passive aggressive though, I.e. "hey I don't have any sore feelings
about it, but they went about it like real jerks". Just come out and say they
were real jerks without the "I don't have sore feelings" crap. We can tell you
didn't like it, and I'm completely OK with that.

I think the mindset that created Tornado is one that a lot of us share as
coders. If there's an existing framework to do something but it's messy in
your opinion, and the job you need to do is simple enough and not covered very
well by that existing framework, well then often it costs a lot less in terms
of investment to build it yourself. Then you get exactly what you need too,
and nothing more (which increases complexity). It worked out for them too!

I also think twisted could market itself better, but that takes effort that
could be spent elsewhere, and I doubt they're all that interested. We'll see.

~~~
encoderer
Exactly. Exactly. Exactly.

And not everything is a cold impersonal engineering decision. Programmers are
people (really! we are!) and people like greenfields and blank canvases and
fresh starts.

And I'd also submit that a team of talented developers almost certainly will
make a superior product if they have the ability to use the original as a
reference and resource. The saying "Pioneers get the arrows, settlers get the
land" comes to mind.

~~~
rapind
Agreed. Greenfield projects are far more enjoyable, and you tend to learn
more. Obviously we can't always do that and succeed, but they did well enough
in this case. Preferably they'd go easier on the remarks about existing open
source frameworks though.

------
crad
I like the idea of the performance and simplicity of Tornado, as I primarily
use web.py. After reading this blog entry I decided to give Twisted another
look. It took me about 2 minutes to get to the info about the web module and
the examples were not very clear. I've walked away wondering a few things:

If I use twisted, should I use web or web2? How do I map URI's to functions?

When I go to the tornado homepage, I scroll and in < 1 minute I see how I
would code for it and can be off and running. To develop web apps, I shouldn't
have to spend time figuring out an async network communications framework, I
should be worried about programming my app.

My perception from the Twisted site is that I would need to spend time
figuring out the twisted way of doing things, then figure out the web add-on
way of doing things, then worry about my app. With Tornado, I'm very close to
a drop in replacement for web.py and I can be off to the races.

~~~
dlsspy
Your perception of twisted may be as it is because you can do _SO MUCH_ with
twisted. If it's a web.py replacement you want, it doesn't add complexity.

Take a look at the diff to go from a friendfeed ioloop to a twisted reactor
loop (with my adaptor) in tornado:

<http://pastebin.com/f707e5ae9>

You replace two imports with two other imports, and three lines of startup
code with three other lines of startup code. Your app doesn't change.

~~~
nuggien
Is the code for tornado.twister available anywhere?

~~~
gsiener
Yes, Dustin forked it on github. You can read about it here:
<http://dustin.github.com/2009/09/12/tornado.html>

tornado.twister here:
[http://github.com/dustin/tornado/blob/205a13b3c0afac843abdeb...](http://github.com/dustin/tornado/blob/205a13b3c0afac843abdebb86bff9f7336414510/tornado/twister.py)

------
n8agrin
An observation: Python coders seem to believe that when someone has solved a
problem in Python and open sourced that problem every Python coder must use
their solution or else they are somehow doing an injustice, perhaps even
undermining the community!

On the contrary, 1) Friendfeed can do whatever they want. There is no part of
the Python license that says you must use Twisted or else. 2) I doubt FF's
statements really caused damage to Twisted's user base, just as we use
Cherrypy at work and didn't drop it on Thursday because of Tornado. 3) All
open source contributions are a Good Thing. If Twisted devs are really feeling
screwed FF is basically saying "here's how we did it, use it if you want".
Sounds pretty win win to me.

Instead of complaining and accusing I would suggest Twisted devs take the
publicity opportunity to show why Twisted is still great, benchmarks, what the
do better than Tornado and what they can learn from it.

~~~
cturner

        An observation: Python coders seem to believe
    

That's a poor generalisation, and I assume you've built it on the back of a
single observation. There are dozens of python template libraries that solve
problems with different levels of complexity, and there's no animosity over
it. Indeed, one of the reasons I like python is that there's a strong standard
library, and then a healthy diversity of tools so you can mix and match to get
what you want.

    
    
        I doubt FF's statements really caused damage to Twisted's user base
    

They implied that it had bottlenecks.

Your suggestion in the last paragraph might cause benefit.

General comment on difficulties learning twisted - I found it difficult to
grasp until I'd practically reimplemented the patterns by accident in the
course of writing my own client/server application. I got stuck on a concept
at one point in my non-twisted socket application, and the guys on #twisted
helped me past it. For those who are interested, the O'Reilly book is obsolete
but a better starting point than the online docs.

~~~
n8agrin
_That's a poor generalisation, and I assume you've built it on the back of a
single observation._

On the contrary, my observation is based on comments I've read in many Python
community forums, especially when applied to Python vs (your other high-level
language of choice). For example, I've read many comments to the effect of "I
like Python over (Ruby, Perl, etc) because it promotes one right way of doing
things and having one package for handling problem X." I am not stating that
this belief is a Python language requirement, nor do I believe that this
observation adheres to all Python coders so yes, it is a generalization. But
my generalization is certainly not based on a single observation. Finally, I
will gladly play the devil's advocate and agree that there is little way to
prove this generalization, short of spending several days collecting emails on
Python mailing lists, which is why it is offered as an observation only. It is
an observation of the practices of a community not something codified into a
PEP, at least as far as I know based on my limited research.

I completely agree with the rest of your comment and value your critique.
Python is a great language that provides a diverse set of tools, which is why
I also choose it to handle many tasks.

------
paul
The great thing about open source is that people are free to share ideas and
code. If the Twisted team likes parts of Tornado, they are free to replicate
those parts back into Twisted while ignoring the parts that they don't like.

Meanwhile, users are free to choose whichever approach suits them better.
Twisted is no doubt more complete in terms of protocols and such, but many
people also value simplicity. One important point that I think has been
overlooked here is that the core of Tornado is only a few hundred lines of
code. For me, the fact that I can actually read _all_ of the code in just a
few minutes is very valuable. I'm sure many others don't care though, which is
why it's great that we live in a world where there doesn't always have to be a
single "right" answer.

~~~
dlsspy
"Twisted is no doubt more complete in terms of protocols and such, but many
people also value simplicity. One important point that I think has been
overlooked here is that the core of Tornado is only a few hundred lines of
code."

I think this is a bit misleading. The twisted port involved removing almost
1,400 lines of code and didn't change the tornado API. That is, tornado is not
more simple without twisted.

I do understand the value of being able to read all of the code quickly, but
there are other turtles below these. For me, it's very valuable to know that
the part upon which I'm building is a well-tested codebase whose semantics I
understand, and for which I can write tests when things begin to violate my
understanding.

If tornado is used in more applications, it will have to grow in complexity to
meet the needs of those applications. I see tornado as having two faces. The
web portions can certainly continue to grow, but the network IO handing just
means that people will have to port over things that already exist within
twisted, and expand it in complexity (such as the kqueue work that is in
progress) to handle more low-level stuff.

At some point, you can't just easily sit down and read all of it (at least,
not any more easily than you can with twisted).

~~~
randallsquared
_That is, tornado is not more simple without twisted._

That's not exactly what I'd say. :) Tornado without twisted _is_ simpler than
tornado _with_ twisted, because you can't ignore the huge system you'd
otherwise be able to avoid.

Edit to remove mild flame.

~~~
dlsspy
I get what you're saying, but twisted does all that stuff with a great deal of
testing.

It's the same reason smart companies are compelled to contribute changes to
open source projects. That's a lot of maintenance you get for free.

That's why friendfeed didn't do their own python implementation on top of
their own posix operating system to make sure they had a firm grasp on all the
parts. A working and well-tested component is almost always better than
writing a new one from scratch that you have to deal with yourself.

~~~
paul
Your point is fair, but I think you may be dramatically overestimating the
amount of time that was spent writing and debugging Tornado. Overall, I'd
estimate that less than 1% of our time was spent on Tornado and other similar
"reinventions", and that in exchange we got simple systems that we understand
and that work the way we like, which seems like a pretty good deal to me.

~~~
dlsspy
Oh, I don't mean to be perpetuating the argument. I think you guys did awesome
stuff. Most of my twisted work doesn't even have a web interface, some of it
does, but it's fine. Some could really use some help. Tornado seems to be a
great answer there.

I suppose I'm just letting you know how you can better serve my personal needs
next time you invent something great. :)

------
acangiano
He makes excellent points, but let's compare the two scenarios for a moment.

A) Helping out Twisted, they'd have gotten:

* Kudos from the Twisted community for their contributions to the project;

* Less code required.

B) Creating Tornado, they got:

* To avoid other people's code they didn't like in the first place.

* All the credit. They are the authors of the framework, not just contributors;

* A fresh start. A project designed to their needs;

* Ability to lead the project in the direction they like;

* No stigmas attached (e.g., "Twisted is too complex");

* Major kudos from a large community of developers interested in "real-time" web services and applications;

* Tons of free publicity to the project and FriendFeed/Facebook's open source initiatives.

~~~
Confusion
_To avoid other people's code they didn't like in the first place._

'Not Invented Here' syndrome. A bad reason.

 _A fresh start. A project designed to their needs_

That's shortsighted. Maybe the short term goals had different accents, but in
the long run, Twisted and Tornado have the exact same goals.

 _Major kudos from a large community of developers interested in "real-time"
web services and applications_

For rewriting something that already exists and not doing an significantly
better job and arguably a worse job? Applying those resources to Twisted would
also have yielded this result.

 _Tons of free publicity to the project and FriendFeed/Facebook's open source
initiatives._

Also belongs under A. Large contributions to OSS would yield that same effect.

 _A fresh start. A project designed to their needs_ _Ability to lead the
project in the direction they like_

You are assuming, without any reason, that they wouldn't have had that chance
with Twisted. If that wouldn't work out, they could easily have forked
Twisted, effectively taking over the project.

~~~
encoderer
"'Not Invented Here' syndrome. A bad reason."

That seems a little strawman-ish, honestly. On the contrary, they embraced the
technology and architecture of the app.

What's wrong with looking at something like this.. an important infrastructure
piece... liking the idioms and architecture but having issues and thinking
_.... We have a benefit the Twisted guys didn't have... we have prior art... a
map of this territory_ ?

I don't think anybody here has a strong conviction that FriendFeed was
_right_. But you and others seem to have a strong convication that they were
wrong, and all I think....and I think others here too.... all I think is that
it is silly to attack them and be so certain they were wrong.

These are talented developers. Smart guys. With a little bit of business
talent, too.. All of us have been in complex project cycles. Who are you or
anybody to pick them off like they somehow caused somebody harm? Why the
intensity? What gives?

------
calambrac
Twisted has always been an "emperor has no clothes" project to me, a project
that's no doubt very useful but at the same time seems to revel in its
unnecessary complexity (just read its documentation - how many times do you
really have to tell us that something might seem overly complicated but is
worth it for its flexibility?) I'm honestly glad to see a no-bullshit
alternative that doesn't have that same sense of worshipping complexity as a
sign of quality.

~~~
enneff
That viewpoint seems quite narrow to me.

The documentation of Twisted attempts to introduce several event-driven
programming patterns. Most people have a 'wtf' moment or two when they come
across design patterns that approach things seemingly backwards.

The further I got into my Twisted app, the more I found myself reaching for
things and finding that were already there for me. It was a great experience.
You should try it!

~~~
calambrac
I've done plenty of work with Twisted. My complaint was not that it lacked
functionality, my complaint was that the culture of the project favors
'flexibility' over everything else. It's like the C++ of networking libraries.
I'm sure there are plenty of times and places where its approach is exactly
what you need, but it's just rarely a good fit for what I need.

------
lacker
The Twisted people don't seem to get that their framework is too complicated
for many uses. You can't just work on someone else's framework and make it
simpler, if they don't agree that it should be simple in the first place.

~~~
dlsspy
"You can't just work on someone else's framework and make it simpler"

Are you suggesting that python is more complex to use than x86_64 assembler?

Similarly, when I removed code from tornado to make it work with twisted, I
neither tornado more complicated to use, nor did I introduce any limitations.

The "too complicated for many uses" thing kind of bugs me, though. It's the
same argument people use to use for postgres vs. mysql. Something along the
lines of "it has features I don't need" (where need probably means
"understand" or "need today").

That's where tornado left me. What you call complication, I call features. I
use some of these features.

For example, I'd be interested in knowing how something like my tdf web server
could be written in tornado. It attaches processes to resource paths, captures
and transforms the output from these processes (in a stream) and streams
replicates of that data to whatever connections come in.

The answer, of course, involves reimplementing reactor.spawnProcess on top of
ioloop. Then you find some of the subtleties contained therein. Then you begin
to understand why it seemed complicated to begin with.

~~~
evgen
> Are you suggesting that python is more complex to use than x86_64 assembler?

No, but twisted is much more complicated than regular python. The inversion of
control and the fact that until quite recently it was almost impossible to
write code for using the framework that was easy to read/understand by
experienced python ciders who were not familiar with the idims and conventions
used by the project. Twisted generates more WTFs per minute than any other bit
of python kit. Some of this is due to a tradition of poor documentation within
the project and some due to the nature of the code. Twisted is a complicated
system, and sometimes this complexity is not worth embracing.

------
rams
At one time the Twisted guys were accused of walking away instead of trying to
improve Python's async module. History repeats.

~~~
nostrademons
I find it interesting that nearly all the comments here are on Tornado vs.
Twisted rather than the larger issue of communicating vs. coding that the
author brings up.

There've been a lot of times - most recently yesterday - when I've hacked up a
solution, mailed it off, and got back a response of "I just did this a couple
days ago. With a one-line change, my code could've covered your needs too."
And then of course I'm like, "Oh. Shit. Let's modify your code instead and
I'll throw away my changelist."

I get the feeling this happens a lot outside of my own personal/professional
sphere as well. Google's famous for its NIH syndrome. Python's got about 5
different competing web frameworks. Lots of people still write their own
machine code generators instead of using LLVM. I'm kinda curious how much
coding people are willing to do to save one e-mail.

Ironically, the folks who insist on building their own stuff are often
_right_. Google's certainly done well for itself, and it's great that we can
use Django instead of Zope now. So maybe the problem is that communication
really is so expensive that people are better off rewriting large chunks of
code to avoid it. But if that's true, we should be looking for ways to reduce
the cost of communication, rather than glorifying the folks who go to heroic
measures to avoid it.

It's curious - yesterday, _Coders at Work_ was at the top of the hotlist, and
most of the comments said, "I'd rather learn about the business & management
aspects than the technical aspects." Yet here's an interesting business &
management problem, but everyone's commenting on the technical aspects.

~~~
zaphar
The reason I think that folks who build their own stuff are right is that they
understand their problem domain better than anyone else.

The author says communication could have cut their time down considerably.
That may or may not be true. Getting into a debate with an external project
about what should or shouldn't be the correct way to fix/change the external
projects code can be a huge drain on time and mental energy. You can't fault a
group for deciding not to bother.

Not to mention that if you become the person responsible for a piece of an
external project you are suddenly not just supporting your use of that piece
you are supporting everyone else's use of that piece. That leads to more
debating/discussion about the right way to do it. If you have to find a way to
scale fast it might actually in the long run be less time to just write it
yourself for your specific needs not an entire communities needs who have
different scaling issues than you and different goals. Many times One size
does not fit all and convincing someone else of that is just too time
consuming to try.

The author did bring up an interesting point about friendfeed developers not
even trying to communicate though so I can understand what prompted him to
write the article.

------
roder
I thought that glyph's response was appropriate and on the mark. It seemed to
me that Bret had taken a few jabs at Twisted without really providing
quantitate evidence. Particularly the comments about being "demo-quality" and
"tons of bugs".

It would have been nice to see innovation happen collaboratively between
FriendFeed and the Twisted team. Everyone would have benefited more IMHO.

Twisted has certainly done a lot for event-based architecture. As Glyph points
out, the use of Deferreds have made their way into javascript frameworks like
dojo. To me, that's credible and proof of twisted's impact in the development
community.

~~~
boucher
I really think that calling Deferreds a "de-facto standard" in JavaScript is a
huge stretch.

I'm having trouble pinning down exactly what makes Deferreds different from
any asynchronous callback system, but it seems to be the chain of callbacks?

~~~
dlsspy
One of the frustrating differences I ran into is the callback mechanism in
tornado. You can't attach more than one callback, and the one kind of callback
does two different things (meaning the callback has to determine whether or
not the call was successful before it does its work).

To me, this makes things more complicated. What if I want to use the same
error handling code for all bits of code (really common for me is
`d.addErrback(log.err)` to just log it), but each successful call has to do
something different?

Twisted deferreds also allow you to _compose_ chains.

This is a somewhat trivial example, but by having the callback do both success
and error cases, I had to build an adaptor to pass to callbacks to include the
"is this a failure" information.

The easy way to do that is to create a wrapper class and construct it one way
or another based on whether a success or error occurred. I ended up just
composing it like this:

    
    
        deferred.addCallback(lambda x: Response(body=x))
        deferred.addErrback(lambda e: Response(error=e))
        deferred.addCallback(lambda r: callback(r, *args, **kwargs))
    

The first callback and errback create a Response (wrapper) object that
contains either a body or an error. The second callback receives the response
that was made by one of the above and calls the initial callback.

This is sort of unrolling complexity and sweeping it away. The _real_ problem
is that you can only have one callback in tornado (and you can't easily chain
them). That means that what actually happens when this deferred's callback
fires is that it ends up calling the original callback, which is an inner
wrapper function inside of another function that calls a curried function that
calls your original callback -- and you have to create these things by name
when you're building out your auth class.

To me, _that's_ complicated.

deferreds seem like there's more involved than you get with a simple callback
until you actually try to _use_ the simple callback.

The interesting properties of deferreds are

1) You can add multiple callbacks and errbacks and build chains as complicated
as your system needs including _composing_ common cases (such as success and
error counters).

2) If a callback (or errback) returns a deferred, you spin off another
dimension of the chain.

#2 is why the above code and about twenty lines of web.py exist for a single
use case.

Here's a concrete example (close to something I do in the real world™):

Imagine a simple twitter API. You go grab a tweet and then give it to
something else: `t.grabTweet(something).addCallback(processTweet)`

Now someone adds a requirement that between `grabTweet` and `processTweet` you
want to perform short URL expansion. In twisted, I'd just toss an
`addCallback(expandURL)` in the middle and call my awesomely well-tested URL
expander -- which makes an async http call to another service and doesn't call
the next callback in the chain until that service returns the data I need to
fill satisfy the request.

And then after I did that, I found that the URL expansion stuff I was using
was blowing up the upstream and causing me to fail to expand some, so I added
async cache support to those _inside_ the API (not as an additional toplevel
callback). For this new step, the toplevel API doesn't change, but now I can
do more async stuff that may lead to further async stuff without having to
write additional glue code.

------
moe
I agree that FF should have asked and should have used twisted under the hood.
I hope the porting efforts will be successful and eventually replace the
original Tornado impl.

I cannot help, though, but yet again in this blog post notice the unfortunate,
defensive tone that is so omnipresent in the twisted community.

The post starts out by (politely and rightfully) pointing out FriendFeed's
communication mistake and then goes on to describe how green the grass would
be in twisted land.

What I miss is a bit more critical self-reflection about the reasons for FF
skipping twisted, even without FF spelling them out in detail. It's not likely
that FF found completely new and unseen problems with twisted. I'm more
inclined to believe that they simply stumbled over the standard, long-known
twisted problems; first and foremost the lack of documentation and (probably)
the verbose coding-style that many parts of twisted demand.

I have the highest respect for the effort that went into twisted and for their
often cited code quality. But they need to realize that they can't display a
"your own fault if you're not smart enough for this"-attitude towards
documentation and at the same time mock anyone who doesn't want to put up with
that.

Like any OSS project twisted is ofcourse not _obliged_ to anything. But
neither are people like the FF guys obliged to choose twisted.

~~~
rg123
I 2nd this completely re: Twisted documentation. When Twisted's communication
via their documentation has been so lacking/off-putting/etc., it is hard to
feel sympathy for a Twisted guy complaining about the lack of communication
from people who chose a different route.

------
tdavis
One thing to remember about Twisted is that it is basically worked on by _two_
people. It is an enormous code base for those two to maintain, develop on, and
document at the same time, even if they are ridiculously smart (which they
are).

I gave kudos to Tornado, but Twisted deserves some appreciation too -- and
contributors! I tried to take twisted.web into the present (the client
anyway), but I unfortunately wasn't sufficiently TDD to get it contributed.
Perhaps next time!

~~~
codexon
I submitted a patch to Twisted to fix their SOCKS implementation.

It was "accepted" after a month, and by accepted it means put into a branch
where it will take up to a year to be incorporated into the mainline depending
on whether the few maintainers feel like taking a look at it.

------
frognibble
There are several async programming libraries for Python including Allegra (
<http://laurentszyster.be/blog/allegra/doc/> ), libevent-python (and other
libevent bindings to python), the standard asyncore module and more. Why
should Facebook use Twisted instead of one of these other libraries?

~~~
imbaczek
more importantly, why should it roll it's own given a plethora of more mature
alternatives?

~~~
frognibble
Perhaps there are many alternatives because they all have problems.

------
davidw
Nice article. As someone who has been involved with open source for a while, I
think one of the most annoying things is indeed when people don't even tell
you about their problems. I'm already doing work for you for free; I don't
have money to go do market research, so I'm counting on you to step up and say
something if there's a problem.

------
smoody
I understand the response at some level, but I also totally get the idea of
making things work the way one's mind works. It makes coding much more
enjoyable and speeds the process of app development once the core work is
done.

And it was nice of the author to do some QA work for Tornado, too. Hopefully
he'll go the extra step and submit those bugs he found as tickets so that he's
walking the walk and not just talking the talk. :-)

------
Tichy
It sounds very good, but I must admit I immediately remembered lots and lots
of comments from open source developers that amounted to "not happy with our
project (a ka 'free toilet')? well duh, hack it yourself, we owe you s __*".
Obviously not the fault of the Twisted developer, but I am not surprised if
people actually prefer to hack their own stuff.

------
johnbender
One reason I wouldn't have reached out, as a private company, is tipping my
hand to competitors. Maybe an NDA would solve that?

------
andrewvc
Well, if this benchmark from yesterday is accurate, then Twisted can't handle
as many concurrent connections as Tornado. I'd say that's a good reason not to
use Twisted.

[http://www.apparatusproject.org/blog/2009/09/twisted-web-
vs-...](http://www.apparatusproject.org/blog/2009/09/twisted-web-vs-tornado-
performance-test/)

~~~
roder
If you read the comments in the blog, it's likely just related to the
platform. I did the test on my mac, which certainly isn't the final say...
I'll redo the test today different hardware today.

