
Vert.x (JVM async) vs Node.js Http benchmarks results - purplefox
http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/
======
andrewvc
Interesting results.

While it's not surprising to me that the pure java implementation was far and
ahead better since: 1.) Java's much faster than Javascript, and 2.) Netty
(which Vert.x is based on), has been doing high performance async IO for years
before node even existed in extremely high performance environments.

What is surprising however, is that javascript on the JVM w/ vert.x is faster
than V8 with node. In both cases I would assume not much JS is executed, since
in Vert.x most of the executing code is still Java, with a tiny piece of JS
scripting on top, and in the Node.js case, I was under the impression that
most of the executing code would be the C code that powers its HTTP server.

Does anyone with knowledge of the node internals know what's going on here? Is
Node's HTTP parser just slower? Is its reactor not as efficient?

~~~
tferris
Very good point.

V8 does heavy code optimization and Node.js http sever code should be well
optimized as well. And maybe the load balancer for the multicore Node test
isn't optimized enough. Thus, these results feel a little shady. Anyway,
Vert.x was the trigger that I am finally downloading the JVM (JDK) to try
Vert.x (and maybe later Clojure).

But there's still one major drawback—the non-existant ecosystem. I know the
hint to look for libs from the Java world but I need a concrete and precise
guide how to do this. Let's say I want to plugin some Java lib for image
manipulation. How? And who will guarantee that these libs will be concurrent
and/or non-blocking as well? The lib developer or Vert.x? At the moment there
is—except few out-of-the-box modules—nothing. No 3rd party libs, no module
manager a la npm and no guide or documentation how to glue Java libs to this
Vert.x thing. Nothing. Correct me if I'm wrong.

~~~
baudehlo
We know the java libraries are there for lots of things, but Node is all about
async and handling thousands of connections. It does this by forcing the
entire ecosystem to be async too (including things like database drivers).

By using a Java JDBC database driver you're completely losing any async
support. Same presumably goes for redis or Mongo drivers. You can do some of
the work with threads and pooling, but it's still not the same, and makes this
another useless micro benchmark.

~~~
pron
Node's _API_ is async. Under the hood everything is done via threadpools, same
as in Java or any other stack. Your hardware knows how to run threads; that's
all. Whether or not that's what's exposed to you as a programmer is a
different story.

You don't actually get a performance boost from Node being "async". Node's
async abilities simply give you transparent access to threads that are
otherwise unavailable with javascript, and it's the threads giving you
performance.

~~~
ithkuil
I don't think this is true. Nodejs uses epoll/kqueue/select etc to multiplex
access to multiple file descriptors from a single main thread.

The async API is actually a price to pay (spaghettification).

For example, the go language took a different approach: it created a cheap
thread-like construct which doesn't incur in the biggest overhead of classical
threading (namely pre-allocated linear stacks and context switching/sleeping
requiring a systemcall; all this aided by the compiler), and a cheap mean of
communication (channels).

Then, the whole core IO library was written using a multiplexing async model
(epoll...), which communicates with the user part of the library via
channels). The result is a blocking like API which under the hood behaves like
an async implementation.

A similar goal is also met by <http://www.neilmix.com/narrativejs/doc/> and
other javascript 'weavers' which convert "sequantial looking" code into
callbacks.

~~~
pron
Yes, but at the end of the day, underneath it all, you gotta have threads
because that's what the hardware understands. Even if you use hardware
interrupts to detect IO, you still need a plain old thread to handle it. The
only difference between various languages and runtimes is how you distribute
tasks among the threads. Some environments provide green threads that have a
partial stack, but even they are handed off to a thread pool (or a single
thread) for execution.

It's been found that if you employ only a single thread (that can run any
number of tasks) you get a performance boost over using a larger threadpool
under some conditions, but a single thread wouldn't let you scale by taking
advantage of all processors.

~~~
ithkuil
I feel that the cause of misunderstanding lies in the fact that "thread" in
this context is usually means "thread based IO", which means that when a
thread issues a IO request it remains blocked until the IO request returns,
leaving CPU time to other threads. All this regardless how many processors you
have; it works perfectly fine with single processors.

Async IO is different, it's a different patter of access to IO and as such
it's orthogonal to any threading or multiprocessing that's going on in order
to actually do stuff in response to that IO.

> It's been found that if you employ only a single thread (that can run any
> number of tasks) you get a performance boost over using a larger threadpool
> under some conditions, but a single thread wouldn't let you scale by taking
> advantage of all processors.

Indeed. Nodejs solution to this problem is to have a cluster of nodejs
processes and a dispatcher process on top. So multiprocessing is done the "old
way".

~~~
pron
In that case, Java gives you both options: blocking IO and non-blocking,
polling IO. Netty can use both, but most people use it with the non-blocking
option. Experiments have shown that sometimes one is faster and sometimes the
other.

------
jlouis
Perhaps a dumb question: Are the headers the same? With so small files, the
header layout is suddenly very important for your measurement.

I've seen my share of web servers which are very different in their
compliance.

~~~
IsaacSchlueter
It's not a dumb question at all.

------
IsaacSchlueter
My comment: [http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-
node-...](http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-
simple-http-benchmarks/#comment-14)

~~~
prospero
Netty has support for zero-copy file responses, I imagine that's at least a
large part of the performance discrepancy.

------
dap
A rigorous benchmark must seek to explain the difference in performance, not
just hand-wave saying "such-and-such is faster". Otherwise, you have no way of
validating (for yourself, let alone demonstrating to others) that it's not a
misconfiguration or a flaw in your benchmark.

------
pjmlp
Just goes to show that when the node.js fad is over, the enterprise world will
keep its high availability servers running on proven technologies.

~~~
fruchtose
Node.js is not a fad. It represents the first workable JavaScript-based server
with mass appeal. The real story is that JavaScript is here to stay. There
have been other server-side JavaScript frameworks in the past, but none of
them have taken off like Node.js. If Vert.x wins over the JavaScript crowd,
then that's great, because coders will be able to write JavaScript.

~~~
batista
> _Node.js is not a fad. It represents the first workable JavaScript-based
> server with mass appeal. The real story is that JavaScript is here to stay._

Javascript yes. Node.js, not so much.

 _> There have been other server-side JavaScript frameworks in the past, but
none of them have taken off like Node.js._

Javascript was not a fast language in the past, nor was it much used for
anything more than the most basic dynamic html stuff (rollovers etc)...

~~~
ricardobeat
> nor was it much used for anything more than the most basic dynamic html
> stuff

You seem to have missed Netscape Enterprise Server (1994), HaXe, Helma,
AppJet, Aptana Jaxer, Narwhal/Jack, EJScript, RingoJS, Flusspferd, and ASP.

~~~
batista
No, the industry seem to have missed them. None of those were any big success.

(With the exception of ASP. But in ASP, Javascript was just one of the
languages you could use, and not the most popular one).

------
ww520
This is consistent with a benchmark I did a while back comparing Netty vs.
Node.js. Not surprising since Netty powers Vert.x. Netty is pretty amazing.
500K concurrent connections and not batting an eye.

------
mbq
According to the code, it actually does a file read on every request -- this
is certainly suspicious because some implicit caching may significantly change
the results.

~~~
ww520
It includes a test with file streaming on Node.js, that bypass reading the
file and write to socket via the Javascript loop completely.

Also file IO are heavily cached by the OS. You can bet that one file is read
from memory most of the time. Disk IO is pretty much out of the equation.

------
jwingy
With regards to Vert.x, it seems like really cool stuff.

On the blog post about version 1 being released it mentions being able to mix
and match several programming languages. Does this mean you can use different
libraries written in different languages in the same Vert.x server?

~~~
purplefox
Yes indeed. You can mix and match Java, JavaScript, Ruby and Groovy in the
same app. We hope to support more languages going ahead (e.g. Java, Scala,
...)

~~~
eropple
Once you get Scala, I'll be heading your way. :-)

~~~
orefalo
You already have Scala+Async IO+Netty, it's called play framework 2.0

<http://www.playframework.org/>

~~~
eropple
I've used it. Not a fan of many (most) of their design decisions, though it's
better than the current alternatives.

------
exDM69
Yet another meaningless micro benchmark. There is no point in measuring a
hello world http request in one framework vs. another.

There should be a larger application that even remotely resembles some kind of
real world usage. Maybe some day we'll have some kind of a "standard" for a
web framework benchmark, an application that actually does something so it's
worth benchmarking.

~~~
andrewvc
Hardly meaningless. The problem is that the larger application benchmarks fall
prey to accusations of "I wouldn't write it that way!". Their results are just
as hotly disputed.

This micro-benchmark makes sense. It's benchmarking an HTTP server
essentially. Any benchmark further than this would really be benchmarking the
JVM and V8. While that would be interesting, I think in this case, this micro
benchmark is OK as long as you know its limitations.

The reality is that most people's app code and databases are going to
bottleneck long before either Vert.x or Node.js do. The main thing this
benchmark clears up is that both of them are really, really, fast, and that if
you do lots of simple to process responses you may want to go with Vert.x.

~~~
exDM69
> This micro-benchmark makes sense. It's benchmarking an HTTP server
> essentially.

Kind-of true. But to give any results that can be applied to the real world,
it would have to use a much larger payload than a hello world.

------
anuraj
Very exciting initial results. The JVM simply is the most optimized runtime
available right now. And Java is the dynamic language with best performance.
Can't fail to notice Ruby is the slowest even on JVM. If you continue on this
path and refine your APIs to be more user friendly, this would be the next big
asynchronous server out there!

------
alz
In the test, the node code is actually calling an asynchronous function
fs.readFile on every single request:

<https://gist.github.com/2650401>

Even with os caching there is still quite a bit of overhead there. Would be
interesting to see the benchmarks run on the corrected code:

<https://gist.github.com/2650401>

------
DTrejo
The streaming node.js example he wrote uses a blocking call. This is not the
node way, and would cause a definite slow-down.

If you're worried about your programs containing rogue & misbehaving code like
this, I recommend you use <https://github.com/isaacs/nosync>

~~~
nfriedly
I had the impression that "Node.js (readfile)", and possibly "Node.js" meant
the blocking call but that "Node.js (streams)" meant he was using something
like fs.createReadStream(). But you're right, I don't see that anywhere in the
posted source.

~~~
purplefox
I've tested several combinations of blocking, non blocking, readFile, streams
(pipe) and using chunked transfer encoding.

Results vary a little but all way below the Vert.x results.

See blog post for the stats.

~~~
nfriedly
Fair enough. After I wrote that I remembered that readFile wasn't blocking
anyways, but it was too late to edit at that point.

------
pron
I'm interested to know how vert.x compares to industrial-strength
"traditional" servlet containers. My guess is that vert.x would outperform
them under certain conditions, but all in all they would scale better.

I believe servlets are still the most scalable web stack out there.

~~~
simonw
I imagine you mean that servlets are the most performant web stack, not the
most scalable web stack. Scalability != Performance.

~~~
pron
Well, I'm sure it's easy to beat servlets with a load that requires only one
thread with a simple enough computation model. Servlets (and any truly
multithreaded solution) trades single-thread performance with the ability to
scale with the number of cores. That's what I meant.

So I would think that given a heavy load on a many-core machine with
interesting enough computations, nothing could beat servlets' performance.

~~~
simonw
My apologies - it's clear you were talking about scaling performance with
additional cores, which is entirely the right usage of the term.

------
binarymax
Don't serve static files with node. Use an nginx reverse proxy for your
html/js/css assets.

~~~
zafriedman
I'm going to ask a dumb question as someone who is just learning node. What's
wrong with serving assets out of public/ in an Express app? Why would someone
not want to do this?

~~~
trimbo
For the vast majority of sites that ever hit the web, yeah, it will probably
be fine. So go ahead and have fun learning node and do it however, you can
always fix it later if the site gets overloaded.

At some volume though, you want to do what the grandparent recommends -- front
your application server with nginx, apache, or something. Otherwise serving up
static assets is taking away resources that should be spent on the
application. Consider if you have a page with 5 images, 5 JS files and 5 CS
files for each page created. That means you're serving 15 static assets for
every 1 dynamic asset you'd really want to use Node for. While Node can still
be "good enough", it is not spending its time doing the stuff you really want
to use it for in the first place. A dedicated piece like nginx goes much
further.

And also use a CDN :p

------
courtneycouch
So a quick verification. The io is the difference between these two. The JVM
is doing some caching somewhere, whereas the v8 engine is not. Making a small
change to both (in order to ensure that both are using the exact same logic):

<https://gist.github.com/2652991>

and then I get the following results:

vert.x:

39890 Rate: count/sec: 3289.4736842105262 Average rate: 2958.1348708949613
42901 Rate: count/sec: 2656.924609764198 Average rate: 2936.994475653248 45952
Rate: count/sec: 3277.613897082924 Average rate: 2959.610027855153

node.js:

38439 Rate: count/sec: 4603.748766853009 Average rate: 4474.62212856734 41469
Rate: count/sec: 4620.4620462046205 Average rate: 4485.278159589091 44469
Rate: count/sec: 4666.666666666667 Average rate: 4497.515122894601

Making that change so they both store the file in memory and nodejs is 50%
faster than vert.x.

This is using an m1.small instance on EC2, and both vert.x and nodejs only
using a single core.

~~~
tipiirai
Yea. There is no way java/rhino could beat c/v8.

~~~
courtneycouch
In any case even if it was faster, it wouldn't be an order of magnitude
faster. That kind of a difference should be an indication that something is
wrong with the benchmark.

------
smagch
I wanted to know the benchmarks including luvit. Vert.x vs luvit vs Node.js
<https://github.com/luvit/luvit>

~~~
elliotlai
luvit, love it!

------
halayli
Why are they measuring requests/sec? Any server can accept connections at a
high rate but what matters is responding in a timely manner.

I doubt the requests number too. Writing a dummy socket server (evented,
threaded, ...) that just returns "HTTP/1.1 200 OK" will not get you anywhere
close to 120k requests/sec. The system call becomes the bottleneck.

~~~
purplefox
It's labelled badly, what is actually measured is req/resp per second.

I.e from request to corresponding response and how many of those it can do per
second.

If you doubt the numbers please feel free to run them yourselves, all code is
in github

~~~
pohl
Could you clarify that? Are you saying that if the response is sent within the
same second that the request came in that it contributes to the metric?

Or would a response that is sent 30 seconds after the request came in
contribute to the metric too?

~~~
ww520
It doesn't matter whether a request straddles a second or not in a throughput
measuring benchmark when you saturate the system. A client would only count a
request when its request call has returned. Runs it for N minutes, count up
how many requests have completed, then divide the total with the time and you
got req/sec.

Besides the benchmark has run for a minute. I doubt each request lasts 30
seconds.

~~~
purplefox
+1.

The system is in steady state, i.e. queues of requests/responses aren't
growing. Therefore it doesn't actually matter if you count the requests or the
responses.

------
anonhacker
Are there any easy cloud platforms or whatever that support easy deployment
for Vert.x?

~~~
purplefox
People have already got Vert.x running on OpenShift and Heroku, and
CloudFoundry support shouldn't be too much longer.

------
VeejayRampay
Testing A against B. A is found to be faster.

People in the B community come out in outrage saying that the testing is
flawed, that microbenchmarking is useless, that this and that. Rinse and
repeat.

------
firefoxman1
Quite interesting. Is the Vert.x API compatible with CommonJS and/or Node
modules? If not, I think it will suffer from the same chicken/egg problem that
WebOS had with apps.

~~~
manuscreationis
Seeing these numbers (although as the author states, the data needs to be
taken with a grain of salt since he didn't try to set up a more "proper" test
environment) really makes me want to take a serious look at vert.x, but what
you mention here is my main concern.

Node is a joy to use because of the NPM, and all the available libraries. I'd
hate to lose that.

That said, so long as it has a solid websocket api, and I can do routing in it
like I would express, I'd give this framework a shot

~~~
eropple
> Node is a joy to use because of the NPM, and all the available libraries.
> I'd hate to lose that.

Maven's not bad at all (from a consumption perspective--from a dev
perspective, it's a bit of a pain in the ass), and it looks like pretty much
any Java library that doesn't do anything _too_ insane should work just fine.
(Disclaimer: never used vert.x myself, but I did a quick scan of the code.)

~~~
manuscreationis
Cool, thanks for the tip

------
soc88
JVM beating the shit out of Node.js? I guess that's not even slightly
surprising to anyone outside the Node.js bubble.

~~~
thebluesky
Java much faster than Javascript. News at 11...

