
Going Head-To-Head: Scylla vs. Amazon DynamoDB - uberdru
https://www.scylladb.com/2018/12/13/scylla-vs-amazon-dynamodb/
======
planckscnst
I haven't read the whole piece yet, but I saw some things that already stood
out as strange.

> Please keep reading to see how diligent we were in creating a fair test case

I hope this is true, but my cursory reading found several unfair spots that
seem at odds with this statement.

> 3-node cluster on single DC | RF=3

DynamoDB works across AZs and will continue to be available even when a data
center goes down. That cross-AZ operation has benefits as well as latency
costs that are not present in the ScyllaDB setup.

> We hit errors on ~50% of the YCSB threads causing them to die when using
> ≥50% of write provisioned capacity

That is surprising. It is quite common to use your full provisioned capacity
without problems. My guess is that there is something not ideal about the YCSB
DynamoDB library or its configuration. I'm not familiar with YCSB: does it
give you stack traces that indicate why the threads failed?

> Sadly for DynamoDB, each item weighted 1.1kb – YCSB default schema, thus
> each write originated in two accesses

This is what made me come write this comment. You specifically knew this was a
pessimistic case which could be easily addressed to allow DynamoDB to operate
at a lower cost. Is arbitrarily settling for the default 1.1kB item size on an
artificial benchmark fair? Good engineering teams use their tools the way that
gives them the most benefit. Calling out that you may have to work to ensure
your use case doesn't have pessimistic characteristics would clearly be fair,
but I'm not convinced that just picking arbitrary benchmark settings is.

~~~
thekozmo
Fair questions, let me (Dor) answer:

1\. Scylla works within different AZs too We are topology aware and can have
as good and even better HA than DynamoDB. Our design is based on C*

2\. We were surprised with getting small utilization too. As I wrote in the
article, I think Dynamo had a hard time reaching to 1TB that quick. The
population started fine and deeper in the the population it failed. Only a
decrease in the rate solved it.

3\. 1.1kb The is the default table setup by YCSB. It isn't optimal for Dynamo
but that's life.. think about if you store a blob of 1kb - together with the
key it will be more than 1kb. We were upfront about it and you can make your
own calculation. Scylla will still be better.

The only use case where Dynamo is better in price is when you store lots of
data but with a tiny IOPS reservation. We'll address this case over time too.
Cheers!

~~~
planckscnst
> Scylla works within different AZs too

Did your test use this feature of Scylla? As I understand the documentation,
it doesn't appear so (maybe that needs to be fixed). If that's the case, do
you think this is an example of being diligent in your fairness?

> Only a decrease in the rate solved it.

I don't think that is the only way to solve it. Something is probably wrong
here, and I would expect that diligence to fairness would dictate trying to
understand and correct that. This benchmark would be more compelling if
problems like this weren't just ignored, but actually solved. Any team that
cares about the cost of their database would do that.

> It isn't optimal for Dynamo but that's life.

Yes, life is unfair and this can happen. This test specifically claimed to be
fair, though. Again, for this benchmark to be meaningful and compelling, this
kind of thing should be addressed.

> The only use case where Dynamo is better in price is when you store lots of
> data but with a tiny IOPS reservation

This is a surprising statement. If this is the case, I highly recommend
addressing the issues so that you are using DynamoDB like a typical
engineering team would. A blanket statement like this with questionable
benchmarks as evidence is just not compelling. It would be much more helpful
to see where DynamoDB really can't stand a chance competing against ScyllaDB
even when it's fully optimiized.

Are you ignoring maintenance costs? Do nodes get added and removed without
paying an engineer? How do you keep the nodes up-to-date as far as security
updates or adding new features? How much overhead cost is engineering on-call
to respond when the cluster has issues (e.g. nearing capacity limits, nodes
failing, loss of AZ availability)?

I'm also curious if you are only considering use cases that you typically
already see on ScyllaDB and Cassandra. How about the use case where I have a
load that increases without warning by 10x or where I need regular backups or
that requires 5000 nodes?

~~~
thekozmo
About multiple AZ, I think we haven't used it in this case but it shouldn't
change a bit, most of our AWS customers have AZ-aware deployments and it works
great.

Since I see you're with AWS, you're welcome to help us debug this error 500 on
population. I guarantee we'll update the blog if we'll see it works fine

I didn't get your surprise, I actually am 100% open about digging deep and
come up with a uncommon use case that Dynamo can finally be a good
alternative.

Believe me, I can come with examples that will put Scylla in a much better
light.

We do offer a fully managed service on AWS with 4x-6x cost saving vs Dynamo
where all the backup and auto scale is done by the Scylla team.

~~~
otterley
> About multiple AZ, I think we haven't used it in this case but it shouldn't
> change a bit

Are you suggesting that the metrics would not change? I assure you that is not
the case: inter-AZ latency is 1-2 orders of magnitude higher than intra-AZ
latency.

You'll need to re-run your benchmarks in a multi-AZ cluster in order to
provide an apples-to-apples comparison.

~~~
thekozmo
I don't think this is the case - check this link:
[https://knowledgebase.progress.com/articles/Article/expected...](https://knowledgebase.progress.com/articles/Article/expected-
latency-between-aws-availability-zones)

The intra AZ latency according to AWS is sub millisecond! It's inline with
what our customers see.

Since all of the numbers in the blog are above a millisecond, the result does
not change.

~~~
otterley
Intra-AZ latency is indeed sub-millisecond. Inter-AZ latency is in the
milliseconds range. (The intra prefix means within, inter means between.) And
those latencies accumulate quickly when benchmarking.

Why don't you try it yourself and see the difference? Relying on some third
party website is no substitute for testing.

~~~
thekozmo
I went and run a ping between 2 VMs, one on us-east1c and the other on a
different AZ - us-east-1b

As you can see, the ping latency is 0.5ms to 1ms on rare occasions. Of course
that ping has a cost even within a single AZ so the above numbers reflect much
more than the AZ round trip time.

You're welcome to add 1ms to our latency numbers, it's 3x-4x better in the
99th case so there's enough room...

64 bytes from 35.173.249.24: icmp_seq=14 ttl=63 time=0.584 ms 64 bytes from
35.173.249.24: icmp_seq=15 ttl=63 time=0.462 ms 64 bytes from 35.173.249.24:
icmp_seq=16 ttl=63 time=0.848 ms 64 bytes from 35.173.249.24: icmp_seq=17
ttl=63 time=0.465 ms 64 bytes from 35.173.249.24: icmp_seq=18 ttl=63
time=0.451 ms 64 bytes from 35.173.249.24: icmp_seq=19 ttl=63 time=1.00 ms 64
bytes from 35.173.249.24: icmp_seq=20 ttl=63 time=0.854 ms 64 bytes from
35.173.249.24: icmp_seq=21 ttl=63 time=0.446 ms

~~~
erik_seaberg
Inter-AZ latency is going to matter more for conditional writes. I'm
disappointed to find LWT so far out on Scylla's roadmap since preventing
conflicts is a showstopper for correctness and we found it to be Cassandra's
weak point in scaling.

~~~
jjirsa
fwiw, work being done on Cassandra to speed this (cas/lwt) up SIGNIFICANTLY

~~~
ddorian43
Link ?

~~~
jjirsa
Started here:
[https://lists.apache.org/thread.html/15d1b4b5bb8a24e41d51db5...](https://lists.apache.org/thread.html/15d1b4b5bb8a24e41d51db58538b56d0bab40c17fc9c64e898f2ad3b@%3Cdev.cassandra.apache.org%3E)

Continues here:
[https://issues.apache.org/jira/browse/CASSANDRA-14448](https://issues.apache.org/jira/browse/CASSANDRA-14448)

------
jugg1es
This all looks great but one of the primary advantages of DynamoDB is that you
don't have to spend resources on maintaining or troubleshooting ec2 instances.
When you have a large deployment, not having to worry about HA on your data
store is hard to beat.

~~~
sheeshkebab
You still have to spend a lot of labor/resources maintaining and managing
dynamodb. Not managing ec2s but, at scale, it’s not exactly a fire and forget
type thing and requires a lot of SRE and devops attention.

Also, it sounds like Scilla has a managed solution too, that costs a fraction
of dynamodb, at scale.

Dynamodb to me is the new generation IMS database from mainframe cobol times -
fully proprietary, locked down, runs only on proprietary hardware, serviced by
one vendor. It’s only a matter of time before companies using it will be stuck
with multimillion $$ yearly bills, if they aren’t already.

~~~
Dunedan
> You still have to spend a lot of labor/resources maintaining and managing
> dynamodb.

Do you? At least with DynamoDB On-Demand there shouldn't be much maintenance
left to do.

~~~
ddorian43
You will or you'll pay (a lot) for it.

------
reilly3000
A theme of the comments so far has been around the fact that the benchmark is
created in a manner that feels biased.

QUICK STRAW POLL: What is the right way for that information to be gathered
and presented?

a. There is no bias. Companies can and should present research that may favor
them, as long as they cite good sources.

b. An industry analyst, paid to research multiple companies and options and
present that information to their respective (most likely paying) customers.

c. A journalist, presenting research in public, funded by advertising for
unrelated interests.

d. Reporting by peers/actual users of the system on how the products compare,
and possibly about how it aligns with their technical and business goals.

e. Trust no one. Conduct your own research, and publish it if you see fit to
so do, or keep it to yourself.

I see potential "disinformation vulnerability" with each approach. How ought
we get the best info on how to align our organizations with technologies?

~~~
thekozmo
The benchmark was done as fair as possible but I'm the vendor, yes, don't take
my word (despite good background in OSS and earlier achievements). Listen to
our users:

\- Here's a webinar by one of our customers explaining how they migrated from
Mongo+Hive, received better consistency and simplicity while saving 5X:
[https://www.youtube.com/watch?v=1hXKd_rNyuE](https://www.youtube.com/watch?v=1hXKd_rNyuE)

\- See how kiwi.com got a CRAZY gain with Scylla vs Cassandra:
[https://youtu.be/Bqh09LG_QDE?t=833](https://youtu.be/Bqh09LG_QDE?t=833)

\- See Grab (South east Asia Uber) about Dynamo vs Scylladb:
[https://www.scylladb.com/users/case-study-grab-hails-
scylla-...](https://www.scylladb.com/users/case-study-grab-hails-scylla-for-
performance-and-ease-of-use/)

Go ahead and give it a try and see for yourself

------
aynsof
Am I the only one who's a little disappointed they didn't call it CharybDB?

~~~
biggestdummy
FYI. [https://www.scylladb.com/2016/02/16/fault-injection-
filesyst...](https://www.scylladb.com/2016/02/16/fault-injection-filesystem-
software-testing/)

------
thekozmo
We'd love to hear your feedback on it. Dor (article writer)

~~~
gtaylor
With the caveat that I'd love to see Scylla succeed, I'm a bit bummed to see
the conditions around this bake-off so slanted in Scylla's favor from the
start.

Seeing a multi-AZ configuration compared to a single-AZ config is an immediate
disqualifier, without getting into some of the other comments on item size
disparity and YCSB quirks.

This was a really poor response from Scylla on these fronts:
[https://news.ycombinator.com/item?id=18677511](https://news.ycombinator.com/item?id=18677511)

~~~
thekozmo
I went and run a ping between 2 VMs, one on us-east1c and the other on a
different AZ - us-east-1b

As you can see, the ping latency is 0.5ms to 1ms on rare occasions. Of course
that ping has a cost even within a single AZ so the above numbers reflect much
more than the AZ round trip time.

You're welcome to add 1ms to our latency numbers, it's 3x-4x better in the
99th case so there's enough room...

64 bytes from 35.173.249.24: icmp_seq=13 ttl=63 time=0.792 ms 64 bytes from
35.173.249.24: icmp_seq=14 ttl=63 time=0.584 ms 64 bytes from 35.173.249.24:
icmp_seq=15 ttl=63 time=0.462 ms 64 bytes from 35.173.249.24: icmp_seq=16
ttl=63 time=0.848 ms 64 bytes from 35.173.249.24: icmp_seq=17 ttl=63
time=0.465 ms 64 bytes from 35.173.249.24: icmp_seq=18 ttl=63 time=0.451 ms 64
bytes from 35.173.249.24: icmp_seq=19 ttl=63 time=1.00 ms 64 bytes from
35.173.249.24: icmp_seq=20 ttl=63 time=0.854 ms 64 bytes from 35.173.249.24:
icmp_seq=21 ttl=63 time=0.446 ms 64 bytes from 35.173.249.24: icmp_seq=22
ttl=63 time=0.460 ms 64 bytes from 35.173.249.24: icmp_seq=23 ttl=63
time=0.509 ms

~~~
awinder
There’s still 2 unresolved problems (at least) here:

1\. running a ping test is a fine start but wouldn’t a more realistic test be
to rerun the benchmark? 2\. in a multi-az setup you’re going to have to pay
for inter-az transfer costs between ec2 nodes. data transfer between ec2 and
dynamodb is free.

Also on the pricing front reserved capacity really drops your costs with
dynamodb if you know you’re going to be with it for some term of time. Not
that that needed to be included in the benchmark but... aws pricing is
complicated, to say the least.

~~~
thekozmo
True. In our brand new service offering we took it all into account.
Networking is costly but doesn't change the game that much. Scylla is 4x-6x
cheaper, depending whether it's on-demand (vs Dynamo on-demand) or reserved
for a year (vs dynamo reservation)

[https://www.scylladb.com/product/scylla-
cloud/#pricing](https://www.scylladb.com/product/scylla-cloud/#pricing)

------
CyanLite2
TLDR: When not accounting for DevOps and SRE costs, managing your own VMs are
a fraction of the cost of PaaS services.

