
Twisted.web vs Tornado Performance, part deux - roder
http://www.apparatusproject.org/blog/2009/09/twisted-web-vs-tornado-part-deux/
======
PhilChristensen
I question the relevance of these tests. With all the attention being given to
performance, everyone seems to be overlooking the fact that Tornado provides
_zero_ support for truly asynchronous web applications written for it.

If you have _any_ kind of long-running task to perform during a web request,
you have to roll your own way of dealing with that task.

To make it worse, the demos included with tornado that make use of a
relational database do all their queries in blocking fashion, which would
render the server completely unusable during the runtime of every single
query.

Given the fact that Twisted provides a robust API for dealing with long-
running algorithms, as well as support for a huge number of other protocols,
you're paying an awful lot for only slightly better performance.

In fact, in a real-world application, the performance benefit of Tornado is
directly impacted by how good you are at dealing with asynchronicity.

~~~
roder
I think your point is really valid, though I think the relevancy of the tests
were to test the claims made by Bret from tornado.

Glyph made a really good point, tornado comes with zero automated tests...
Ultimately after conducting these tests and listening to the discussion that
has taken place, I am personally glad that we're writing our code with
twisted.web

~~~
wehriam
What are you writing with twisted.web?

~~~
roder
Stay tuned :) !!!

~~~
wehriam
I will. Ping me if you get a moment.

------
FraaJad
Summary:

    
    
      * Mean response time -- Twisted.Web better than Tornado
    

Average performances for Concurrent request:

    
    
      * @100  -- TW:Tor::0.50s:0.70s . TW better than Tor
      * @500  -- TW:TOr::   3s:3.5s  . TW slightly better than Tor
      * @1000 -- TW:Tor:: 5s:6.5s    . TW better than Tor

~~~
moe
Executive summary:

Ignore this "benchmark", the author simply doesn't have a remote clue what he
is doing.

500ms Mean Response Time at 100 concurrent requests, are you kidding? Did you
run the "benchmark" over a 56k modem?

For the record, one would expect sub-100ms response times under that load,
even on modest hardware.

~~~
roder
wow...

The test was run on two VMs on the Rackspace cloud using the public IP
addresses.

------
smoody
I still don't understand all of the fuss. These two are close enough in
performance to where people should pick the framework that best serves their
needs, their designs, and their brains.

Big deal -- there are now two ways (at least) to accomplish the same outcome
in Python. Competition is good. I'm guessing that this will be a call to arms
for the Twisted community and we'll see development accelerate.

At the same time, it is likely that a development community will form around
Tornado. Great!

It's like people saying, when the Mac OS first launched, "Why didn't they just
use DOS --it's there and it works." And yet the Mac OS has had a positive
impact on operating systems produced by Microsoft. And Microsoft has impacted
the Mac OS. It's all good.

And why is it that this same argument doesn't occur every time a non-async
framework appears for python or other languages? What is it about the async
nature that polarizes people? Maybe it's the fact that the both also start
with the letter "T" and the name "Tornado" is a better named than "Twisted"
(which implies increased complexity even if it isn't there...of course
Tornados ultimately make a mess of things, so that's not it).

I, personally, am excited about Tornado. I think there's room for more than
one player -- especially given the 'real time' direction the web is heading.
And if this negatively impacts Twisted, then it'll be because Tornado has come
up with an easier to understand approach and they'll benefit from being new
and easy to grok -- in the same way Rails was once an easy-to-grok framework
and that's what made it easy to understand.

The biggest winner, in my opinion, is the Python community and who doesn't
want that?

------
jacquesm
I don't think these tests are much more meaningful than the first.

They still use virtual machines, which give you all kinds of invisible
overhead.

Even if that were not the case you should definitely split the machine that
does the tests off from the machine that is being tested otherwise you get
contention between the test program and the programs being tested (and java
being quite cpu intensive running both tests on the same physical box is not a
good idea).

~~~
teilo
Given how common it is for production applications to be running in the cloud,
this test is both valid and useful. Therefore these tests are much more
meaningful than the first.

~~~
jacquesm
I disagree. If you run a test on a virtual machine then you are testing a lot
of other software besides the software under test so the picture gets
distorted.

For instance, in the graphs that are in the report you can see a bunch of
drops to '0' traffic, these suggest serious problems with the software under
test. But knowing that the tests were done in a VM environment it is possible
that the cause of these drops is not in the software under test but with the
VM, or alternatively, with other software running on the host the VM is
running on.

That makes it very difficult to assign meaning to all this.

If you test software you do it in an environment that controls all the
variables as much as possible otherwise the results will not help you in
making decisions.

If people run software 'in the cloud' then that's a different story
altogether, then you test the same software side-by-side on your bare machine
vs your 'in the cloud' setup and you benchmark those.

This test is labelled 'twister vs tornado', not 'twister in the cloud' vs
'tornado in the cloud'.

~~~
teilo
If the tests were all over the map, I could agree with your original
statement. But they are not. They are consistent.

Do you honestly expect anyone to believe that when test after test is run on a
VM, and one framework consistently outperforms another, that this is due to
the VM, and not the frameworks themselves?

~~~
jacquesm
I'm more concerned with the drops to 0 and the overall throughput. The drops
to 0 look quite bad, overall throughput is low compared to what I would expect
from a setup like this.

------
buster
I guess i will never understand why they didn't choose twisted and finally
gave it the revamped twisted.web it needs.. I have yet to see a more
featurecomplete internet framework with which i can do practically everything
the internet is about, with so much flexibility and yet it is suffering from
the state of twisted.web. Aynchronous database access, http, pop3,imap,smtp,
amqp, irc, msn and about every protocol out there can be so easily integrated
in twisted, it amazes me everytime. Sad, that this framework isn't used more
widely, it could rule the web.

------
adnam
Innocently daft question coming up:

As I understand, you can run web.py apps behind lighttpd, and the web server
handles the processes. E.g. in lighttpd.cfg you can set {"max-procs" => 4} and
I would see 4 instances of my web.py script running. But how is this done for
a web-server done 100% in python? Python uses a GIL that only allows one
thread to interpret bytecode at a time, so despite the amazing performance of
Twisted and Tornado, wouldn't we see this seriously degrade when the server
had to anything more than 'hello world'? My worry is having a Twisted/Tornado
app and only using 1/4th of my availble processing power: am I wrong to do so?

~~~
joshu
That applies to threading.

Twisted and Tornado are epoll/kqueue/etc-based and do not use threads.

Here's a good article on the differences between strategies:
<http://www.kegel.com/c10k.html>

~~~
adnam
Great article, thankyou!

------
akamaka
These tests are completely useless, since most of the measurements are done at
a point where response time has already degraded unacceptably. I don't need a
web server that performs slightly better when massively overloaded.

The sweet spot to measure is between a response time of 0s to 1s. After that,
you need to be looking at getting more servers, anyways.

A better benchmark would start by swapping the axes of the graph and asking
the question: "how many users can I support while keeping response time
acceptably low?"

------
cnlwsu
I would really like to see Twisted vs Tornado Documentation. The performances
are close enough that I don't think it should be a deciding factor in which of
these to use. Albeit performace is always an important metric that should be
watched to keep things from getting worse.

~~~
buster
In fact, all those comparisons compare two totally different frameworks. One
is for the internet (which really includes far more then http) and one is for
the web (which mostsly includes http).

