It seems that several projects claim that they have built the "world's fastest key/value store" or sometimes the phrase used is even more outrageous. The following projects are in question:
- Redis: https://github.com/redis/redis
- KeyDB: https://github.com/snapchat/keydb
- Dragonfly: https://github.com/dragonflydb/dragonfly
- Skytable: https://github.com/skytable/skytable
And, these are my benchmark results (30 threads, 1 client per thread with GET/SET. See below for setup):
1. Redis: 112,100 / 99,892
2. KeyDB: 288,931 / 282,997
3. Dragonfly: 408,322 / 392,446
4. Skytable: 619,992 / 676,091
# Notes
1. Redis:
I'll start with Redis which I'd like to call the "original" key/value store (after memcached) because it is the oldest and most widely used of all. Being a long-time follower of Redis, I do know it's single-threaded (and uses io-threads since 6.0) and hence it achieves lesser throughput than the other stores listed above which are multi-threaded, at least to some extent. The best parts about Redis: it's the most feature complete of all the systems here, and is the oldest. (not saying old necessarily means better).
2. KeyDB:
The second is KeyDB. IIRC, I saw it in a blog post which said that it is a "multithreaded fork of Redis that is 5X faster"[1]. I really liked the idea because I was previously running several instances of Redis on the same node and proxying them like a "single-node cluster." Why? To increase CPU utilization. A single KeyDB instance could replace the unwanted proxying funkiness, so I ditched Redis for KeyDB. Has been a fine experience so far, except for some occasional crashes.
3. Dragonfly:
Just found it on HN and seems to be relatively new. Dragonfly claims that it is 25X faster (that I couldn't reproduce) than Redis[2] and has the slogan "Probably, the fastest in-memory store in the universe!". Doesn't support all the Redis commands yet, but I find it interesting mainly because of performance. Also, it's good to know why it is faster because it clearly outlines the underlying architecture[2]. The other three stores don't say much about it. Will be following the project.
4. Skytable:
Found it while looking for projects written in Rust. Claims to be "insanely fast." Skytable's "experimental benchmarks" claim that it is something around 10X faster than Redis and some 2X-3X faster than KeyDB[3]. I hadn't heard of Skytable and it doesn't seem to be as widely used (unless I'm missing something?). I find it interesting because of the planned features[4] and performance. Only Skytable natively runs on Windows out of the four. Will be following the project.
5. My thoughts:
Redis needs no introduction and is arguably super stable for use on production systems (using widely in our systems). KeyDB seems to be "stable enough" and it seems to be good for prod since Snapchat uses it already[5] (and so do we!). I found no Dragonfly v/s Skytable benchmarks.
The best part about Redis, KeyDB and Skytable is that they don't make any "crazy assumptions" about the system they run on. What do I mean?
Dragonfly expects you to have the latest hardware[6] and the latest kernel[7] on all your servers. I find this outrageous because not all servers run 5.10 and a majority of them are still using long-running branches on 4.x. Also, I don't expect them to have the latest processors, either. I'd argue if the other three stores started assuming the latest features, they'd be far faster than what they are today. Finally, both Dragonfly and Skytable are still early in their development so it may not be fair to compare their features against Redis and KeyDB who have been around for far longer. Also, all projects other than Skytable have companies behind them (unless I'm missing something).
Edit #1: I have run the benchmarks myself for each store. In the benchmark with Redis, KeyDB and Skytable (redis-benchmark, memtier and sky-bench): Redis and KeyDB benchmarks seem to be very consistent, Skytable is a little inconsistent at times. However, in the benchmark with Redis, KeyDB and Dragonfly (with memtier), I was NOT able to reproduce the 25X speed that Dragonfly claims. I ran all tests on two m5.4xlarge servers (one with the k/v store and one with the benchmark tool).
Edit #2: Added benchmark results
What are your thoughts? Have you tried benchmarking any of them locally or in the cloud?
References:
[1]: https://docs.keydb.dev/blog/2019/10/07/blog-post
[2]: https://github.com/dragonflydb/dragonfly
[3]: https://github.com/ohsayan/sky-benches
[4]: https://github.com/skytable/skytable/issues/203
[5]: https://docs.keydb.dev/news/2022/05/12/keydb-joins-snap
[6]: https://github.com/dragonflydb/dragonfly/issues/124
[7]: https://github.com/dragonflydb/dragonfly/issues/96
The Dragonfly benchmark which showed 25x faster was run on c6gn.16xlarge which has 64 vCPU. The m5.4xlarge you used "only" has 16 vCPU. Since Dragonfly is achieving its performance by using multiple threads it's not weird that you weren't able to replicate its performance gain.
In my opinion, when it comes to these types of multi-threaded benchmarks it's much better to separate the "baseline, one-process performance" from "how it scales with number of processes". E.g. if you first pin Dragonfly to only run on a single core you can find the baseline performance compared to Redis, and then you can run different benchmarks of Dragonfly with increasing number of cores to see how it scales. This should give you much better understanding of how the system behaves. (The release of Dragonfly actually prompted me to write a blog post on this: https://t.holmium.no/dia/apples-vs-oranges/)