

695k TPS with Node.js and VoltDB - dscape
http://voltdb.com/company/blog/695k-tps-nodejs-and-voltdb

======
MichaelGG
I'd like to add an independent voice too. I tried VoltDB out around a year
ago. Ran it on 3 servers, with k=1 (all data replicated on 2 nodes). Servers
were Dell PowerEdge 860 - only 8GB RAM, and a Q6600 quad-core processor. They
cost maybe $500 or so. Even with a low-end buildout, I was able to hit around
110k TPS. This wasn't one of VoltDB's sample apps; it was a small proof-of-
concept for a telecom application. They don't seem to be riding the hype
train.

As far as open source: I was using the .NET client, and fixed a few issues
with it. The folks at VoltDB were really responsive about working with me and
accepting my patches. They also took time to explain the inner workings, sorta
give me a launching point to investigating the core code. Pretty good
experience for a commercial open source project.

~~~
signa11
> it was a small proof-of-concept for a telecom application.

would you describe what telecom application this was ? was it billing, ocs,
pcrf or something else ? thanks !

~~~
MichaelGG
Yes, very basic realtime billing/CDR storage. With traditional DBs, I have to
resort to batching balance: Read N messages from queue, compute overall
balance updates, then apply at once. With VoltDB, I can just fire off records
as they come in.

------
jqueryin

        TITLE DISCLAIMER: 695k using 20 fairly large machines
    

I was, at first glance, pretty impressed with the 695k TPS title claim. Upon
reading the article, it became apparent that this was based on running 8
8-core EC2 m2.4xlarge instances for node.js in addition to 12 8-core EC2
m2.4xlarge instances for VoltDB. In terms of server architecture, this is
quite a large setup.

Just for an idea of how big of a machine we're talking for each of these
instances:

    
    
        High-Memory Quadruple Extra Large Instance
        68.4 GB of memory
        26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
        1690 GB of instance storage
        64-bit platform
        I/O Performance: High
        API name: m2.4xlarge
        $1.800 an hour running Linux on demand
        $864.00 a day baseline for 20 of these
    

This isn't meant to discredit the author or work, just to make sure it's
apparent that this solution is for serious businesses only with deep pockets
and boatloads of traffic. You'd be spending _$26,784_ for the servers alone
over a 31 day period if using them on demand.

The results are definitely impressive for the right target audience.

~~~
spullara
I'm not sure if I am supposed to be impressed or not. On my macbook pro with 1
redis master and 1 redis slave I can do about 300k updates / second to the
master replicated to the slave. Comparable? Who knows?

These benchmarks don't mean much to me -- what matters is performance for your
application.

~~~
arielweisberg
I'll start by saying I dig Redis. I wish we had a storage engine that offered
some of the kinds of flexibility that Redis does. If I were making Volt from
scratch I would embed Redis or something with flexible schema and secondary
indexes and spend less time working on SQL and more time working on
distributed functionality.

>These benchmarks don't mean much to me -- what matters is performance for
your application.

Very true. Think about how you would implement an application like Voter in
Redis. Voter is an event counting app that applies business logic server side
to do what is basically fraud detection.

Let's take a look at what each of these transactions is.
[https://github.com/VoltDB/voltdb/blob/master/examples/voter/...](https://github.com/VoltDB/voltdb/blob/master/examples/voter/src/voter/procedures/Vote.java)
3 statements and an entire round trip to validate data is kosher. 1
insert/update statement that is actually updating two different summary views

You would need to implement your own locking to get the isolation necessary to
implement the business logic and there would be at least two round trips for
each vote. You would also have to write your own code to maintain the
materialized view used to print the summary. Sure you could implement the
validation logic client side, but then you lose the ability to transactionally
update said logic and the state it depends on.

With the new scripting functionality in Redis you would be able to do it in
one round trip but you would still end up writing your own code for the view
and rollback. How you will attach a debugger to debug your server side logic
is also worth considering. I mistakenly wrote that Redis doesn't cache Lua
scripts, but it actually does.

Now that you have server side logic you will also probably want to start
maintaining replicated state to make some of your transactions single shard
transactions. With Redis you are on your own for updating replicated state in
a way that stays consistent across the cluster even when restoring different
shards.

Also, no matter how fast Redis is you may reach a point where you have run out
of disk/network IO, or even CPU. Now you are looking at sharding Redis. You
are probably going to want to maintain replicated state to make more of your
transactions single shard transactions. With Redis you are on your own for
updating replicated state in a way that stays consistent across the cluster
even when restoring different shards.

Don't forget re-sharding and backups which are now your problem. You
definitely want to be able to roll back to before that last application update
that corrupted your data. Don't forget that you will need to bring the
replicas up in the correct state as well. Preferably in parallel with bringing
up the master since you are down right now, and it would be nice if it
resharded at the same time since you took this opportunity to add capacity.
Redis cluster is coming, but it isn't here today and depending on the
application you may not want to risk it not being ready when the time comes.

You should at least double the number of ops (where an op is roundtrip to the
DB) done by Volt if not more. That said, doing half as much work inside the
procedure doesn't make it twice as fast and doing twice as much doesn't make
it twice as slow. The dominant cost for such a trivial operation is not the
operation itself. Redis is much better at this because it is single threaded
and written in C. My bet is that the number of instructions and cache misses
to accept and execute a command in Redis is much smaller because the execution
engine isn't dealing with stored procedures, schema, undo logging, SQL, and
code expecting to deal with distributed transactions even if none are in play.

>I'm not sure if I am supposed to be impressed or not.

If you haven't tl;dr-ed already I'd call it good.

------
carterschonwald
So voltdb is an in memory SQL Db with some durability and replication
functionality. Is there any good documents on how it compares with some of its
competitors/ who are its competitors?

~~~
MichaelGG
I think durability should not be defined in terms of actual backing hardware.
VoltDBs transactions commit and stay committed. If you restore from a backup,
just like any other database, you'll have data loss until the backup point.
What if my hard drive crashes when it loses power? At that point I'm in the
same position as when I lose RAM.

VoltDB is often compared to other in-memory NoSQL systems, although I'm not
aware of any that actually offer cross-server transactions. VoltDB is best
when you have a high volume OLTP workload and need to really scale beyond what
traditional RDBMS can do.

Edit: Here's a benchmark using VoltDB as a key-value store:
<http://voltdb.com/company/blog/key-value-benchmarking>

~~~
maratd
> What if my hard drive crashes when it loses power? At that point I'm in the
> same position as when I lose RAM.

Oh sure, except that the probability of losing your hard drive when you lose
power is about 1 in a _very_ big number. Whereas, the probability of losing
the contents of your RAM when you lose power is 100%.

Now, MySQL (and in fact, most RDBMS outfits) provide tools for fixing corrupt
data. But you actually need corrupt data to fix. That's the entire point of a
hard drive.

I'm sure VoltDB would be perfect for a chat application or something of that
nature. It would be terrible for anything that involved the transfer of $$$
(banking, ecommerce, etc).

~~~
jhugg
VoltDB Community Edition uses synchronous intra-cluster replication, as well
as rolling on-disk snapshots for durability.

The Enterprise Edition adds a synchronous or asynchronous command log (a
logical journal) as well as asynchronous multi-cluster replication.

The VoltDB blog has a series of posts covering command logging in detail:
<http://voltdb.com/company/blog/intro-voltdb-command-logging>
<http://voltdb.com/company/blog/voltdb-command-logging-replay>
[http://voltdb.com/company/blog/voltdb-command-logging-
option...](http://voltdb.com/company/blog/voltdb-command-logging-options)

In short, VoltDB offers tunable durability from single-node purely in-memory,
all the way up to synchronous disk storage on multiple local nodes and
multiple asynchronous copies in another data center. All options are fully
transactional. All options are extremely performant.

We've worked very hard to make durability and performance not mutually
exclusive. We have internally achieved six-figure TPS numbers using
synchronous writes (requires a disk controller with a BBU, or a decent SSD).

------
purplefox
If you think node.js is fast try <http://vertx.io>.

It's like node but isn't single threaded so scales over multiple cores without
having to fork. Also it's polyglot so you can don't have to use JS if you
don't want.

I'm hoping to publish some performance results vs node.js before our 1.0.final
release in the next few weeks.

~~~
kolev
How does vert.x compare to to Apache Deft?

------
amouat
Is EC2 really the right way to do these tests? I would have thought it would
introduce unwanted variables related to network congestion and server load
caused by other EC2 users.

~~~
jhugg
You might get a better number on bare metal, and you certainly won't get a
worse one, so in a sense it's not ideal.

On the other hand, a big chunk of the web is run from AWS. Demonstrating
performance there implies performance in many places. It's also an easily
reproducible environment, which is helpful for benchmarking.

------
tferris
Sounds nice. Some questions:

\- I assume you did the tests with direct SQL statements without any ORM? How
would an ORM affect the test's performance (with an Node.JS-ORM of your
choice, e.g. sequelize)?

\- How does Node+VoltDB compare to i.e. Node+PostgreSQL or Node+MySQL

\- What does in memory SQL exactly mean? How is data made persistent?

~~~
MichaelGG
VoltDB doesn't really support ad-hoc SQL. It can execute ad-hoc by flipping a
flag, but performance is achieved by using stored procedures. You write the
sprocs in Java, and their compiler looks at the SQL statements and optimizes.

VoltDB smokes other RDBMS (Postgres, MySQL, MSSQL, etc.) for certain OLTP
workloads because it stays purely in-memory, _but removes all locking_. Normal
databases spend a huge amount of time dealing with locks, even if the memory
is in RAM. VoltDB was based on the H-Store project; you can read more about
the ideas at [1].

Making transactions execute serially[2] adds latency to an individual
transaction, but the point with VoltDB is to send tons of transactions so that
there's always work queued up. You should be able to get average latency of,
say, 10ms for the type of app mentioned here. I'm surprised it's not mentioned
in the blog post. (Maybe on EC2, the latency numbers were less-than-ideal, but
throughput was fine.)

SQL and ACID have nothing to do with the backing store. Using RAM is just as
valid as using disk. Practically, with VoltDB you're going to set the "k
factor" - number of replicas you want. This allows you to lose k servers
without suffering data loss (put batteries on the servers; that should be
pretty good). There's a snapshot backup feature, and the enterprise edition
has a disk logger to further minimize the loss window in case of catastrophic
failure.

1: <http://hstore.cs.brown.edu/publications/> 2: More or less. Servers are
sync'd via NTP, allowing one to know when it's safe to run a specific
transaction. There's also some ways Volt might know when one tx is independent
and safe to run before another.

~~~
pron
Well, it removes all locking and achieves scalability only if you partition
(shard) the data and don't have cross-partition transactions.

~~~
fredholahan
VoltDB does support cross-partition transactions. They're not as fast as
single-partition transactions, of course, because you're doing them over a
network. But distributed transactions have been a supported feature since 1.0.

~~~
pron
Right, that's what I meant. You only get scalability as long as you don't have
cross-partition transactions.

------
olalonde
> that’s beneficial to the scaling architectures of both Node.js

That's not very accurate (unless you run multiple Node.js processes). See
[http://stackoverflow.com/questions/2387724/node-js-on-
multi-...](http://stackoverflow.com/questions/2387724/node-js-on-multi-core-
machines)

------
salimmadjd
looks very interesting. If I architect a tracking system again, I'd look at
this setup.

