Does anyone with knowledge of the node internals know what's going on here? Is Node's HTTP parser just slower? Is its reactor not as efficient?
But there's no question Netty is fast even on a non-benchmark. I'm using it for a long-polling server with upwards of 100K connections, and my CPU doesn't go much over 5%. Besides memory usage, the main limitation has been proper kernel configuration.
 C/libev, Clojure/Aleph, Clojure/Jetty, Erlang, Go, Java/Netty, Java/NIO, node.js, JRuby, MRI Ruby
V8 does heavy code optimization and Node.js http sever code should be well optimized as well. And maybe the load balancer for the multicore Node test isn't optimized enough. Thus, these results feel a little shady. Anyway, Vert.x was the trigger that I am finally downloading the JVM (JDK) to try Vert.x (and maybe later Clojure).
But there's still one major drawback—the non-existant ecosystem. I know the hint to look for libs from the Java world but I need a concrete and precise guide how to do this. Let's say I want to plugin some Java lib for image manipulation. How? And who will guarantee that these libs will be concurrent and/or non-blocking as well? The lib developer or Vert.x? At the moment there is—except few out-of-the-box modules—nothing. No 3rd party libs, no module manager a la npm and no guide or documentation how to glue Java libs to this Vert.x thing. Nothing. Correct me if I'm wrong.
Dude, I've been developing real-time military applications and we've moved from C/C++ to Java and got performance gains (I'm not saying that C can't beat Java doing specific computations - of course it can - but when building a large application with millions of lines of code and a large team of developers, Java would be a safer bet than C++ if speed is your concern). In other words, no other environment can beat the JVM.
And, yeah, you should try Clojure. It's a cathartic experience.
EDIT: The only doubts I have about vert.x is that it can consistently beat "old" JVM servlet containers under heavy, real-world loads. That remains to be seen.
(I'm not actually serious about the last bit).
var file = new File('/blah/blah/blah.txt');
Other languages would be equally easy as well. It's burried, but here is some relevant information from the docs on integrating 3rd party libs:
-cp <path> The path on which to search for the main and any other resources used by the verticle. This is ignored if you are running an installed module. This defaults to . (current directory). If your verticle references other scripts, classes or other resources (e.g. jar files) then make sure these are on this path. The path can contain multiple path entries separated by : (colon). Each path entry can be an absolute or relative path to a directory containing scripts, or absolute or relative filenames for jar or zip files. An example path might be -cp classes:lib/otherscripts:jars/myjar.jar:jars/otherjar.jar Always use the path to reference any resources that your verticle requires. Please, do not put them on the system classpath as this can cause isolation issues between deployed verticles.
Assumed I use this ImageIO package. Will it be nonblocking, concurrent and spread to all my cores automagically with Vert.x??
As far as concurrency, most libraries state in the docs what is threadsafe and what is not, so this is more or less "caveat emptor" and RTFM.
Vert.x has its own module subsystem which appears to package up .jar files and some glue code for the Vert.x-specific interface, but I assume you can call any included Java classes directly:
(Interestingly enough, Vert.x uses Gradle for its build/dependency manager, but I guess wanted something more Node-like for its module system)
By using a Java JDBC database driver you're completely losing any async support. Same presumably goes for redis or Mongo drivers. You can do some of the work with threads and pooling, but it's still not the same, and makes this another useless micro benchmark.
The jvm has amazing threading support, doubly so if you use it with a language like scala or clojure. You can and should handle the connections asynchronously and use a thread pool for things like db access. It works well, people have done this with the jvm for years.
The async API is actually a price to pay (spaghettification).
For example, the go language took a different approach: it created a cheap thread-like construct which doesn't incur in the biggest overhead of classical threading (namely pre-allocated linear stacks and context switching/sleeping requiring a systemcall; all this aided by the compiler), and a cheap mean of communication (channels).
Then, the whole core IO library was written using a multiplexing async model (epoll...), which communicates with the user part of the library via channels). The result is a blocking like API which under the hood behaves like an async implementation.
It's been found that if you employ only a single thread (that can run any number of tasks) you get a performance boost over using a larger threadpool under some conditions, but a single thread wouldn't let you scale by taking advantage of all processors.
Async IO is different, it's a different patter of access to IO and as such it's orthogonal to any threading or multiprocessing that's going on in order to actually do stuff in response to that IO.
> It's been found that if you employ only a single thread (that can run any number of tasks) you get a performance boost over using a larger threadpool under some conditions, but a single thread wouldn't let you scale by taking advantage of all processors.
Indeed. Nodejs solution to this problem is to have a cluster of nodejs processes and a dispatcher process on top. So multiprocessing is done the "old way".
It has both event loops and a background thread pool, so you can choose which to run your task on depending on what kind of thing it is.
E.g. it's stupid to run long running or blocking actions on an event loop.
You say that like it's a good thing. I'd much rather have the choice between async and sync.
Sure programming-wise it's nice to have sync. No argument there.
Yes, that's exactly Node's main selling proposition everybody forgets when presenting their next Node.js
Other platforms/languages have real concurrency constructs and don't suffer node's limitations.
I don't know whether making JS single-threaded was a principled decision -- if anything it was presumably the KISS principle at work. However, it was actually a ridiculously nice choice to offer a single-threaded-asynchrony model. It sometimes gets in the way rather obtusely -- Firefox can still (if very rarely) fail to introspect and then crash when some ad script on your page goes into an infinite loop! -- but on the whole, it is very nice to always know that while I'm in this function, modifying this variable, nobody else can interfere.
With that said, I also think that the lack of good concurrency planning is indeed missing, and that it will probably enter the language at a future time.
At best it's "making a virtue out of necessity".
Sounds like you're quoting from the "Node.js Is Bad Ass Rock Star Tech" satire video ( http://www.youtube.com/watch?v=bzkRVzciAZg ).
As an async environment it doesn't offer anything either new or too compelling.
That isn't strictly true.
If you focus on the traditional scripting languages as your competition: Ruby, Python, PHP, Perl: Then you start to realise that Node.js does offer a similar language structure (dynamic, no compilation, etc), with the benefits of thousands of concurrent connections (which those languages will do with certain modules) but while forcing all the libraries to also be async (which those languages DO NOT do).
At my last job I had to build an SMTP server capable of scaling to 50k concurrent connections. Building this in Perl was fine, except for any library I wanted to use - all of the libraries were synchronous. So now I wrote Haraka, which Craigslist are now using as their incoming SMTP server.
If you compare all that to Java you get slightly less performance but probably lower memory requirements. And that's OK. Different strokes for different folks.
What? What do you mean by non blocking in this context?
"You don't need to necessarily have non-blocking libraries to take advantage of the async I/O. Netty uses worker threads to prevent blocking the server event loop and I assume Vert.X does as well. You can do blocking JDBC queries, Redis, etc."
And threadsafe libs --of which are tons--, usually advertise it on "the box.
An interesting way to test what I'm suggesting would be to do the same test with unix domain sockets/named pipes as the transport instead of TCP. The JVM doesn't have a native implementation, so it'd have to call out through JNI. I'd wager even odds it ends up slower than TCP on localhost.
Better write a Mongo-backed fibonacci generator to check.
Vert.x (unlike node) does not force you to do everything on the event loop. It has a hybrid model.
For things like long running calculations (e.g. Fibonacci) or calling blocking APIs, we support running them on a background thread pool so you don't end up doing stupid things on an event loop which are not appropriate for it.
nodejs beats Vert.x for example. People shouldn't be so quick to accept these half thought out microbenchmarks.
I don't think these are speed tests.
Just number of connections that can be handled
These benchmarks are nothing but speed tests.
I've seen my share of web servers which are very different in their compliance.
You seem to have missed Netscape Enterprise Server (1994), HaXe, Helma, AppJet, Aptana Jaxer, Narwhal/Jack, EJScript, RingoJS, Flusspferd, and ASP.
Put another way: What evidence could you potentially see that would convince you it's not a fad?
I remember a time when my colleagues who were steeped in C++ had a good laugh at my expense because I was building server-side web applications with a new framework and a hot, new language. It woefully under-performed similar C++ applications in benchmark tests.
It was 1998, and the language was Java. I could write my applications much faster and in a more maintainable way than they could, but they didn't care. Their technology was proven, and Java was simply a fad.
I have a hard time believing the JVM is really 10x faster than v8 for such a simple server.
Any benchmark you care to look at should clearly shows JVM is much faster:
If a program spends 2% of its time in V8, 10% of its time in the network stack, and 80-something% of its time reading a file from disk, then how can you even consider that you can make it 10 times more effective by optimizing away the 2%? Even if the JVM was 100 times faster than V8, then you would expect to get faster by a factor of just slightly less than 2%.
On the other hand, if you take the 80% of time spent reading the file over and over again, and optimize that down to zero (again, impossible, but the asymptote that we approach as it is reduced), then you would expect every request to take 80% less time. So, your 1ms response becomes a 0.2ms response, and your 1000 qps server is now a 50000 qps server.
So, no, if you respond to 10x as many requests, it's almost certainly either a bug in the benchmark, or some apples-to-oranges comparison of the work it's doing. I called out one obvious issue like this, that the author is using a deprecated API that's known to be slow. But even still, it's not THAT slow.
You can't summon speed out of the ether. All you can do is reduce latency, and you can only reduce the latency that exists. Even if your VM is faster, that only matters if your VM is actually a considerable portion of the work being done. The JVM and V8 are both fast enough to be almost negligible.
It's more than likely that it spends close to 0% of its time in disk access since its serving the same file, which will be cached by the OS in memory.
About the deprecated API. Earlier on I updated the results so they don't use that API, and I also added results for using streams. The results are slightly better but not by very much.
I'm interested in checkout out vert.x. But, this goes to everyone,let's not let this whole affair degenerate. Right tool for the right job. This particular benchmark scenario is explicitly the wrong way to use node. I'd suspect that if you were to change the readFile into an HTTP request however, the numbers might change. I also wouldn't be butt-hurt if vert.x still came out on top. There are still a ton of things to love about node.
Let's get on with our actual work now, shall we?
As some other commenters pointed out, if you pre-load the file contents, the discrepancy goes away. Also, if you actually limit vert.x to 1 CPU, it gets erratic and loses any performance benefits.
EDIT: whoops, saw 4096 for the stream but it's 4896.
Honest question, why?
It's possible vert.x really is that much faster, but given history I'm reserving judgement. I.e. until profiles and root cause(s) become available.
Also file IO are heavily cached by the OS. You can bet that one file is read from memory most of the time. Disk IO is pretty much out of the equation.
On the blog post about version 1 being released it mentions being able to mix and match several programming languages. Does this mean you can use different libraries written in different languages in the same Vert.x server?
Might be nice to also see a FAQ about the differences between vert.x and Play 2
There should be a larger application that even remotely resembles some kind of real world usage. Maybe some day we'll have some kind of a "standard" for a web framework benchmark, an application that actually does something so it's worth benchmarking.
This micro-benchmark makes sense. It's benchmarking an HTTP server essentially. Any benchmark further than this would really be benchmarking the JVM and V8. While that would be interesting, I think in this case, this micro benchmark is OK as long as you know its limitations.
The reality is that most people's app code and databases are going to bottleneck long before either Vert.x or Node.js do. The main thing this benchmark clears up is that both of them are really, really, fast, and that if you do lots of simple to process responses you may want to go with Vert.x.
Kind-of true. But to give any results that can be applied to the real world, it would have to use a much larger payload than a hello world.
These micro-benchmarks aren't even a good kicking off point for comparisons, as the things that yield a trivial benchmark win are often the things that yield significant performance troubles at scale.
Even with os caching there is still quite a bit of overhead there. Would be interesting to see the benchmarks run on the corrected code:
If you're worried about your programs containing rogue & misbehaving code like this, I recommend you use https://github.com/isaacs/nosync
Results vary a little but all way below the Vert.x results.
See blog post for the stats.
I believe servlets are still the most scalable web stack out there.
So I would think that given a heavy load on a many-core machine with interesting enough computations, nothing could beat servlets' performance.
At some volume though, you want to do what the grandparent recommends -- front your application server with nginx, apache, or something. Otherwise serving up static assets is taking away resources that should be spent on the application. Consider if you have a page with 5 images, 5 JS files and 5 CS files for each page created. That means you're serving 15 static assets for every 1 dynamic asset you'd really want to use Node for. While Node can still be "good enough", it is not spending its time doing the stuff you really want to use it for in the first place. A dedicated piece like nginx goes much further.
And also use a CDN :p
and then I get the following results:
39890 Rate: count/sec: 3289.4736842105262 Average rate: 2958.1348708949613
42901 Rate: count/sec: 2656.924609764198 Average rate: 2936.994475653248
45952 Rate: count/sec: 3277.613897082924 Average rate: 2959.610027855153
38439 Rate: count/sec: 4603.748766853009 Average rate: 4474.62212856734
41469 Rate: count/sec: 4620.4620462046205 Average rate: 4485.278159589091
44469 Rate: count/sec: 4666.666666666667 Average rate: 4497.515122894601
Making that change so they both store the file in memory and nodejs is 50% faster than vert.x.
This is using an m1.small instance on EC2, and both vert.x and nodejs only using a single core.
And artificially crippling Vert.x to a single core does not prove anything. Anybody who cares about performance will be using more than one core.
I doubt the requests number too. Writing a dummy socket server (evented, threaded, ...) that just returns "HTTP/1.1 200 OK" will not get you anywhere close to 120k requests/sec. The system call becomes the bottleneck.
I.e from request to corresponding response and how many of those it can do per second.
If you doubt the numbers please feel free to run them yourselves, all code is in github
Or would a response that is sent 30 seconds after the request came in contribute to the metric too?
Besides the benchmark has run for a minute. I doubt each request lasts 30 seconds.
The system is in steady state, i.e. queues of requests/responses aren't growing. Therefore it doesn't actually matter if you count the requests or the responses.
People in the B community come out in outrage saying that the testing is flawed, that microbenchmarking is useless, that this and that. Rinse and repeat.
Node is a joy to use because of the NPM, and all the available libraries. I'd hate to lose that.
That said, so long as it has a solid websocket api, and I can do routing in it like I would express, I'd give this framework a shot
Maven's not bad at all (from a consumption perspective--from a dev perspective, it's a bit of a pain in the ass), and it looks like pretty much any Java library that doesn't do anything too insane should work just fine. (Disclaimer: never used vert.x myself, but I did a quick scan of the code.)
However, it's unlikely that node modules will work as is with Vert.x, since the API is different. (Unless someone writes a translation layer)