
Scaling nodes in a Dgraph cluster - mrjn
http://blog.dgraph.io/post/performance-throughput-latency/
======
trishume
The thing is this benchmark uses a small data set and no concurrent writes.
This means that the data set can be trivially cached in memory on every single
instance.

In this benchmark there is theoretically no need for different database nodes
to even communicate with each other after the initial data sync. It is
embarrassingly parallel.

In fact it is somewhat alarming that going from 2 to 5 nodes doesn't at least
double the performance, given that there should be absolutely zero need for
coordination between servers.

~~~
mrjn
The writes for Dgraph actually happen in O(log N) time where N = number of
subjects for a given predicate; and a background job then syncs the posting
lists to disk later. So, that performance wouldn't be any worse than what we
observe here.

However, to do writes like you would do them in production would require you
to sync a mutation log to disk, before applying it in memory and returning an
OK. This particular operation bounded by disk I/O would then become the
bottleneck, and not help with doing benchmarking for the database. Which is
why we didn't make writes part of this particular benchmark.

Dgraph does concurrent writes very well. To determine that, you can run Dgraph
batch loader. It can load up the entire 21M triples from the blog post, in
20mins on an n1-standard GCE instance. [https://github.com/dgraph-
io/dgraph#loader-performance](https://github.com/dgraph-io/dgraph#loader-
performance)

When jumping from 2 to 5 nodes, we did want to ensure that the nodes have to
communicate to run the queries, as they would in a real-world scenario; which
is why the performance wouldn't just double.

I think you're putting a lot of emphasis on the size of the data. While that
is surely important, the size itself isn't the only thing which proves
scalability. It's about the concurrency, throughput and latency -- and
determining that they do improve when you add more hardware power to the
cluster.

~~~
trishume
Thanks for the info. I guess my follow up question is why doesn't Dgraph end
up caching all the data on every node when it has the memory to do so? Is this
a planned feature that you just haven't implemented yet?

~~~
mrjn
It depends upon the memory available. We have a background goroutine which
checks how much memory we're using and evicts posting lists if we exceed a
predefined usage. If we have a lot of memory then we just won't evict the
posting lists, which means they'll be in RAM.

All this "caching" happens close to the disk level, just above RocksDB. We
don't cache any results -- to me caching is cheating -- and we want to build
something truly low latency. A user can easily add caching if they want to
decrease the load on Dgraph, but it's not something I think we should be doing
at the db level.

------
nl
It seems that a lot of people don't realize that a scalability test is not a
performance test!

They are close to orthogonal concerns. For example I'll take lower absolute
performance if I can scale out.

Performance itself can be optimized separately, and usually that optimisation
also applies to the scaled out version.

~~~
qaq
They are orthogonal to a point :) Thats why people want to see a real world
dataset and queries. If it's performance is 100 times less than an alternative
I don't care if it can scale out. Also if it can scale out to 10 nodes it
tell's me little about how it will perform with 100 nodes. If It takes 30 sec
for complex query on a production dataset I really don't care if I can run
1000 30 sec. queries in parallel if I need that query to run in 50ms on avg
and to never run more than 200ms

------
marknadal
For everyone complaining about this not being a performance benchmark, I
invite you to check out this test:
[https://github.com/amark/gun/wiki/100000-ops-sec-in-
IE6-on-2...](https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-
Atom-CPU) .

Doing about 30M ops/sec on a Macbook Air.

As for the authors of Dgraph, keep it up! Your tech looks promising and I'm
excited to see more and more Open Source graph databases in the market. Would
love to compare notes, shoot me a message.

------
mrjn
Thanks for your comments! I think people feel that the data set wasn't big
enough to prove scalability. While I disagree, I think this is a great
opportunity as well -- to hear from people. We can actually bring up the
entire 1.9B triples from Freebase and run some queries on them -- get some
more real world numbers. So, tell us what to run here:

[https://discuss.dgraph.io/t/give-us-freebase-graphql-
queries...](https://discuss.dgraph.io/t/give-us-freebase-graphql-queries-to-
run-on-dgraph/463)

~~~
lunula
You should run the whole of Freebase, or really the biggest thing you can get
your hands on. Hope you do. I'd love something that can comfortably handle
billion order graphs!

------
mikkom
> Freebase is an online collection of structured data which includes
> contributions from many sources including individual and user-generated
> contributions. Currently, it has 1.9 Billion RDF N-Triples worth 250GB of
> uncompressed data. On top of that, this dataset is over 95% accurate with a
> complex and rich real world schema. It is an ideal data set to test the
> performance of Dgraph. _We decided not to use the entire data set as it
> wasn’t necessary for our goal here_.

I don't follow. Wasn't using as much data as possible _exactly_ what scaling
is about?

~~~
mrjn
Not exactly. Given enough time and a lot of hard disk, you can put the entire
1.9B triples into any database -- that doesn't prove that it scales. Also, if
the database uses a single global mutex lock for all reads and writes, again
it's not a scalable architecture.

Scaling is about throughput and latency. This post is trying to determine
whether adding more machine power actually lets the db perform better.

~~~
LoSboccacc
> Scaling is about throughput and latency.

under reasonable constraint. throughput and latency from a cacheable dataset
is moot for big data.

------
jerven
If you are in graph benchmarking you should really have a good look at
[http://ldbcouncil.org/](http://ldbcouncil.org/). Especially the fact that
this is a read only benchmark. Plus your dataset is tiny. So all you showed is
that if there is no contention and all data fits on the local machine there is
no communication overhead and near perfect scaling. That is not a benchmark
that is hard to beat, at all.

~~~
mrjn
This is not a benchmark. But, there is network communication going on in the
multi node cluster tests.

------
qaq
OK if you load this dataset into Postgres it will be about 10-20X this speed
for the queries performed in this test. What is the point of benchmarking a
graph database on the tiny dataset and using queries that can be run easily on
RDBMS? Isn't whole point of Graph Databases that they can perform well on
queries that would not run well on RDBMS and handle datasets that RDBMS would
choke on?

~~~
lobster_johnson
Not the _entire_ point. Relational databases have schemas that are not well
suited to graph structures, and SQL is also a barrier. A better question is
whether you need to implement your own storage backend or whether something
like Postgres would suffice.

~~~
qaq
OK then what type of graph would be hard to store in RDBMS? There are
obviously queries that one would want to run on graph data that RDBMS will not
run efficiently (but the queries in this test are actually trivial for RDBMS).
In what way is SQL a barrier compared to learning query language used by some
graph database that is not applicable to anything else you do? "A better
question is whether you need to implement your own storage backend or whether
something like Postgres would suffice". You could just use a more established
Graph Database that actually performs well on complex graphs?

~~~
nl
Here's a thing that most RDBMS don't do well but graph DBs do: find me nodes
connected to at least X nodes of type T where those nodes have attribute A and
are also connected to node N. Now duplicate that filter a potentially
arbitrary number of times.

The issue here is that the number of joins explodes, and depending on your
schema you may be doing lots of self joins.

An additional complication is if your dataset is too big for a single host. In
Postures you shard, but that is manual and has significant cost.

In DGraph you lose some performance but (hopefully) if you know something
about your queries you can optimize the distribution function to minimize
cross node queries. This is a pretty hard problem to generalize, but even a
partial solution is good.

~~~
aschampion
Yours is a great example of what graph DBs should be good at, but many self-
styled graph DBs out there at the moment are not. Graph DB means to me only
two things: index-free edge traversal and scalable built-in graph operations.
While these would seem to be necessary and sufficient criteria to distinguish
a graph DB, some instead use only the criteria of the GGP and equivocate graph
DBs with schema-free DBs, which should be orthogonal axes of database
features.

~~~
nl
What is GGP?

I'm not aware of any schema free databases marketing themselves as Graph DBs.
I'm sure there are though.

There is a distinction between graph and graph processing frameworks (GraphX
etc) though, but I don't think that's what you mean.

~~~
jskywalk
OrientDB:
[http://orientdb.com/docs/last/Schema.html](http://orientdb.com/docs/last/Schema.html)

~~~
rspeer
OrientDB would indeed be an example of something that's only a "graph
database" because marketing said it should be.

Last time I asked how to import an actual graph into OrientDB, a marketing
person of theirs pointed me at a Java API for writing extensions to their
code.

