Benchmarks done on only one metric are often misleading (and in commercial circumstances usually intentionally!). Would love to see visually what trade offs Dragonfly is making to achieve the numbers from the chart at the top of the README. If excellent technical work means there really is no trade off, that would also be a great reason to chart it visually.
Also as a Redis replacement, it's not clear what durability is offered, and for most Redis use cases this is close to the first question
How you would suggest to demonstrate visually that there are no trade-offs?
The tradeoff the way I see it - one needs to implement 200 Redis commands from scratch. Besides, I think DF has a marginally higher 50th percentile latency. Say, if Redis has 0.3ms for 50th percentile, DF can have 0.4ms because it uses message passing for inter-thread communication. 99th percentiles are better in DF for the same throughput because DF uses more cpu power which reduces variance under load.
Re-durability - what durability is offerred by Redis? AOF ? We will provide similar durability guarantees with better performance than AOF. We already provide snapshotting that can be 30-50 faster than of Redis.
There is usually a tradeoff between latency and throughput, although I'm not so sure this would be true for your innovation, since you've eliminated a whole chunk of fixed overhead from the system (syscall batching). However, batching in general often implies added latency
If I recall ScyllaDB has some excellent examples of demonstrating this particular tradeoff visually. A simple option would be a scatter plot where X = latency, Y = load or similar, with points coloured according to the system under test. Probably there is a better option, but this would likely be enough to sell me at least
Well in that case, I hope to have an good answer for you. Right below the benchmark graph I deliberately put a table that shows 99th percentile lantencies at *peak* throughput as reported by memtier_benchmark.
99th latency percentile of Dragonfly at its peak throughput. I put it here as well
op r6g c6gn c7g
set 0.8ms 1ms 1ms
get 0.9ms 0.9ms 0.8ms
setex 0.9ms 1.1ms 1.3ms
Probably not many in production, but the "recent kernel" is going to be the more important part.
I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.
Note: I never use kernel modules due to the issues rhel/debian and I have had with such things in the distant past.
> I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.
I'm not sure that concern is justified. It seems io_uring was pushed as part of the 5.1 linux kernel release, and Ubuntu 20.04 LTS seems to have been shipped with 5.4.
The kernel has its own LTS releases, and running Linux in the cloud is a well tested use case. Any library not available in your distro of choice can be built from source and tracked manually, which usually works well enough.
I'll be interested to see where the community lands on this years from now. It's frustrating to see so many databases relicense under licenses that are almost OSS but not quite. Yet it's totally understandable - it's hard to pay database engineers when you give your product away and then AWS neuters your hosting and support business.
I just wish more developers could follow Grafana's path and switch to the AGPL when that becomes a problem, instead of switching to fauxpen source licenses.
> I think its ok that we have innovation at legal front as well. And innovation requires exploration, choosing road not taken...
Chosing to release software under a non-FLOSS, proprietary license is hardly innovative, and I'm afraid trying to frame it that way sounds like you want to have your FLOSS cake and eat it too.
Why? FOSS is, at least initially, a charitable project. You work on a database for months then give it in its entirety away for free to all your potential customers and even your competitors with better established sales teams (e.g. Amazon).
They are explicitly not a charity. Isn’t it okay to be a for profit company that sells software with a closed copyrighted source?
Red Hat makes a Linux distribution. It was not their choice whether to be charitable or not, they are the recipients of the initial charity of the Linux founders and are bound by contract to continuing making their code open as a price for building on open code.
If Red Hat could change the license, would they? Maybe. (Not now but you can’t say the leadership won’t make that decision during financial crisis).
But there's more to a Linux distribution than just the kernel, and a lot of the other pieces aren't copyleft, so Red Hat can change the license, but they choose to keep them open source anyway.
No, it's a collaborative project initially. You work on a tool for months with other developers that each have their own employer, but all benefit from the sum of the work.
That there's freeloaders that benefit from the work without having to expend labour is a side-effect, not the primary motivation.
This looks quite cool from a technical perspective, but the unusual license definitely gives me pause. Largely because I'm not familiar with BSL. Others say it's increasingly popular, but with a little googling the acronym is still somewhat ambiguous - opensource.org lists BSL as the "Boost Software License" which looks more like BSD. This kind of confusion doesn't support the idea that this is a solid trustworthy OSS license.
Still, I really appreciate that you didn't choose a copy-left license.
On the license front, what is the "change license" clause listed? It says something about changing in 5 years. Does this mean it will become Apache licensed in 2027? Why would you put that in there?
It's exactly that. It gives us a little chance to fight against the Giants.
In 5 years the initial version becomes Apache 2.0 then the next version and so on and so forth. CockroachDB uses similar license. MariaDB uses that, Redpanda Data and others. You are right that acronym is confusing - it's not Boost license, it's Business License. Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.
I certainly don't think you should pick a permissive license, and find yourself under competition from companies running your own software.
However, I'm sad that instead of going with an Open Source license that protects against that, you're using a proprietary license. That alone is a nonstarter for many users, not because they want to compete with you but because they want to protect themselves and make sure they have a firm foundation to build on.
Much of the software you're citing as examples moved from Open Source to proprietary, harming their users in the process, and causing many users to seek alternatives.
Do you think AGPL offers such protection? I think even AGPL is fine for Amazon et al, they are happy to dump the code somewhere while selling a managed service with enough wrapper layers. But AGPL is toxic also for self-hosters who aren’t trying to build a managed hosting service, since it applies equally against all parties.
> AGPL is toxic also for self-hosters who aren’t trying to build a managed hosting service
Absolutely not. First of all, you can always use any software under AGPL as released without any limitation.
Secondly, if you were to make changes and release them you can still run it any way you want.
Thirdly, if you are making changes for internal use, and for some reasons your really want to keep them secret, you can STILL run the database to your heart's contents as long as you don't provide it as a service to the public.
AGPL is much more permissive than people think. It's just providing a degree of protection to developers and and users from patent trolls and other uncooperative entities.
Exactly - the grandparent is asking “can a true copy left / Free Software license protect a young startup from Amazon reselling their software”, and my assertion is no, it cant.
MongoDB is the most famous example to transition from AGPL to a closed license. MongoDB was AGPL until October 16, 2018. Since October 16, 2018, it uses Server Side Public License (SSPL), which they authored.
From their FAQ:
> The market is quickly moving to consume most software as a service. This is a time of incredible opportunity for open source projects, with the potential to foster a new wave of great open source server side software. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture all the value but contribute nothing back to the community.
The most recent notable example of open-source software going closed because of Amazon specifically is Elasticsearch, which switched from Apache to SSPL. The announcement blog post was titled "Amazon: NOT OK - why we had to change Elastic licensing".
MongoDB is definitely an example. Did switching licenses actually "fix" their "problem" though? And ElasticSearch doesn't count since it was a pushover license before and not the AGPL.
Redis is doing just fine, PostgreSQL is doing just fine. They are both real open source.
The performance improvements are not worth the legal/compliance overhead of adding non-FOSS to my stack. Much less so if the performance improvements are due just to some optimization choice in the underlying system. In the next 5 years, it will be easier to have Redis adding the io_ring optimizations than for this new project to become uniquely better.
You misunderstood. What I am saying is that is precisely because I am working on projects of different sizes and potentially with different customers, I do not want to have the burden of having to check if the projects have specific compliance requirements.
I have a SaaS [0], my own (AGPL, by the way) open source project [1] and also work have the occasional contract job. It's much easier to say "just use Redis because it is FOSS and it gets the job done" across the board then to try to special-case the tools based on licensing requirements.
And? AGPL is not a non-commercial license. If Amazon wanted to provide such a service, they would certainly be permitted to. As long as they provided any changes they make to the the AGPL source code. The AGPL would not "infect" their hosting infrastructure.
We do not choose AGPL so we would not harm our users. Lets not confuse the means and the goal here. AGPL is copyleft and restricting. Fair users refuse using it.
OSI does not help technological companies to find a fair solution and force us to go outside of "OSS".
> We do not choose AGPL so we would not harm our users.
This is a first. If anything, it seems clear that a FLOSS license such as AGPL achieves exactly the opposite: is highly protective of the users' best interests, both in the short term and long term.
Could you elaborate on why do you think that users are harmed by standard, run-of-the-mill FLOSS licenses?
Depends on who you consider 'the users'. If you are a 'user' in the sense of an individual working on their own stuff, then yes, AGPL protects you a lot.
However, if you are a 'user' in the sense of a company wanting to make software to sell as a service, it doesn't protect you very much, and can make you vulnerable.
> However, if you are a 'user' in the sense of a company wanting to make software to sell as a service (...)
That does not meet the definition of "user", does it?
In addition, if that was the motivation behind that absurd claim, why not be honest and just say "we don't release our software under a FLOSS license because what we actually want is to keep it proprietary"?
It means ‘user’ in the sense of the person who is choosing what software to include in their work stack, and most software of this type is going to be used in a professional setting. If you use a license that most legal departments won’t let their developers use, your software won’t gain much adoption… which is especially important if you are planning on making money from your software. People aren’t going to be paying for support for their personal projects.
Most legal departments don't allow for copyleft licenses. This is because most legal departments don't actually make their own legal determinations but instead rely on industry norms. And google set the stage here ages ago with their no-GPL (and especially, no-AGPL) policy.
However this is rather uniquely due to Google's monorepo infrastructure. Google quite literally has all its products in the same source code repository, being built and statically linked together to create a single system binary (or so I am told--I've never worked there). In this case AGPL would virally infect the rest of your code and you could find yourself in trouble. Even if they don't, they might get hit with a discovery request for the source code of some project to prove compliance, and in the process have to provide their entire monorepo and all its business secrets to the court. Etc. Etc.
But these are concerns which stem directly from Google's monorepo architecture. It's not generally true of the whole industry. And yet Google's fear of copyleft--rational though it may be in their particular instance--has been copied throughout the entire industry. 99% of the companies out there have nothing to fear from the AGPL. But hey, if you're some random in-house lawyer at company XYZ, who are you to question Google's legal precedent?
Using a non-OSI approved license hurts your users considerably more than AGPL (and this is coming from someone who generally doesn't like the AGPL). Legal teams are far less likely to approve licenses like the one you've chosen, than AGPL.
I could convince a legal team to allow an AGPL service, depending on how we were using it. It would be difficult to get them to allow us to use this license, and I definitely couldn't convince them to allow us to contribute to software using this license.
Effectively, you've ensured you won't have a community, because in reality the project is closed source, and dead if your company dies, which means it's not an option for me to consider.
This is not representative of my experience. AGPL is unconditionally banned at most companies where I know the answer to this question. They don't care if the license is OSI-approved, they care if it aligns with their business requirements and objectives. Most companies are not ideological about open source.
BSL is at least making an attempt to address the legitimate concerns of businesses -- both sides -- in a balanced and pragmatic way. I have no dog in this fight, being neither a BSL licensor nor licensee (nor AGPL for that matter), but having been party to the legal conversations at large companies I understand why BSL is generally considered to be more acceptable than AGPL. BSL may or may not be the right license for this software, I have no opinion, but businesses don't care about arguments from ideological purity. If BSL satisfies their requirements better than AGPL, and anecdotally all the evidence seems to support this notion, then they will go with BSL software.
To put it another way, if OSI is serious about being in the conversation about the future of software licensing, they must address the reasons so many companies refuse to adopt AGPL software.
I wonder how involved you've been in these kinds of discussions. I've run open source programs at companies, and worked together with the legal teams in terms of what was and was not allowed.
Transferring copyright to another company, without a contract with that company, for the most part is something a legal team isn't going to agree to. BSL only addresses one side's concerns, and it's not in your favor.
AGPL, when used for internal services that customers don't interact with, is generally not that difficult to get approved. How many companies are using mongodb, for instance?
I do not have any practical, first hand experience with either license. Re -agpl, i know for a fact that it is banned in corporations. We choose BSL to avoid this. Say if a team in amazon would want to use DF internally they could. I know for a fact that AGPL is a big no there. If we are mistaken and BSL is also banned then we will need to see what to do.
I hope people will understand the reason behind bsl. I hope people will see apache there and will help us to grow. Based on HN comments some already do.
As someone building software for a living I fully agree with your decision to not go full OSS. It’s always a hefty discussion but I feel most of the people only exist on the receiving end of software and do not have to worry how a license impacts real life distribution. Big corp (I’m from Germany) DOES hesitate to go with any form of non-Apache-2 licensed OSS software. In Germany often times OSS is even discouraged if there’s no commercial support for it.
I understand the emotions people have regarding licenses that do allow them to work with software as they please - but I also think it’s useless to argue around licensing of a software instead of focusing on the good it can do. Something being proprietary has NO impact on your choice to buy it when outside of the software world. Why does it here?
The problem you are pointing out is not the open source license, it's the lack of support. If they offered AGPL with a dual-license version and made a strong message about commercial support, they could please both crowds without compromises.
> I do not have any practical, first hand experience with either license.
I do have practical first hand experience with both. I can get a legal team to approve the use of AGPL, but I cannot get them to approve BSL, unless we've signed a contract with the company.
Having an understanding of the license, and some first hand experience seems like it would be a pretty important thing, when choosing it, especially for a business.
I don't want to enter in the merit of your company's choice, it's definitely up to you what license to use, but I'm confused about that statement around AGPL. How would AGPL harm users?
More likely they'll ignore the s/w. Huge amount of talent is working in such companies. Using AGPL will practically exclude them from interacting with the s/w.
Also, smaller companies like to play safe. If FAANG with their million lawyers aren't touching it, why should I take the risk.
Because unlike FAANG you don’t have millions of engineers to create the software from scratch. Companies pay for databases (Oracle, Gemstone, MsSQL, anything at Amazon). Almost all the risks (price changes, maintenance costs, etc.) can be managed adequately via a long term enterprise contract.
We may do it in the future. We will see if the community will accept Dragonfly with BSL. MariaDb were pretty famous and i do not remember any loud objections to their license.
I was quite excited to see Dragonfly. However now that the license has been pointed out, I won't be touching it with a 10 foot pole. I wish you the best of luck, but I'll never use it.
If it was AGPL I'd have been a happy camper though.
There were a lot of loud objections to their license, and remember that MariaDB the database isn't licensed that way (it's GPL), just the much less used MaxScale and CMAPI.
> Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.
No, there are plenty that still use permissive licenses.
Finally Redis has a BSD licensed core, a source available license for additional modules, and a closed source license for enterprise. https://redis.com/legal/licenses/
Redis has a BSD license because it was written by antirez who did not intend to make it a commercial enterprise. Today its too late to change and wont help Redis Inc to protect their ip. We looked at elastic, mongodb, cocroach labs, mariadb, redpanda, atlassian, grafana labs , scylladb. The last two are agpl but the motivation is the same: to protect themselves.
Bravo for picking the BSL and writing your license the way you have. It shows maturity in that you and your company know what they are doing and intend to be there for the long haul.
They probably don't want to end up being replaced by an AWS/GCP/Azure service. In my opinion, the BSL is a fair license model, especially if it is limited in duration (let's say 2-3 years BSL then automatically changing to Apache/BSD).
A duration limit in the license, after which it becomes a permissive license, seems the critical point.
Accomplishes the goal of preventing a cloud provider from stealing customers, but also ensures customers don't get caught in an "always tomorrow" trap when the deadline comes and the company realizes it only hurts them to fully share it.
Seems to align all interests pretty nicely.
(I'm as big of an OSS supporter as anyone, but we can't pretend we still live in a time where Google / Amazon / modern-Microsoft don't exist)
Exactly. Finding the right balance is the right thing to do. I am not angry at cloud providers , but i am angry at OSI and their blind idealism that leaves us out there alone. I think that not solving the problem by them, hurt first of all the oss community.
> I think that not solving the problem by them, hurt first of all the oss community.
F/OSS means a very specific thing. If one can't possibly build a rocketship business, that isn't F/OSS fault. Of course, you've got an alt-movement in response to OSI and FSF's rigid adherence to its principles, speared on by tech companies who (think they) got burnt by other tech companies.
There's a lot of room between {completely F/OSS, that a cloud provider can implement and bankrupt the company supporting project development} and {completely evil closed source company like Oracle}.
In the end, we all want good software, with the maximum amount of permissions and source, free. It's just a question of tweaking the support model to get there.
IMHO, the "Don't call yourself OSS if you're not 100% OSI OSS" is counter-productive. But I understand why they do it, and I understand the history and abuses that caused them to do it. I'd just say if we no-true-Scotsman our approach in a way that precludes profitable, sustainable OSS companies... we're going to have less quality OSS. :(
But my entire stack is already in AWS. One service provider deciding they don’t want AWS making a service out of it just means I have to make the effort of self-hosting it, not that I’ll suddenly end up using their service (which is outside my VPC).
can you clarify why you feel this way, plenty of companies use confluente Kafka or mongodb with VPC peering? Would AWS private link make you feel better about network security?
Apologies if this was already asked -- I saw you guys already showed benchmarks in AWS itself, but would it make sense to have some benchmarks outside of AWS (or run benchmarks in a VM with Redis/DF/Keydb installed within)?
I ask this because I'm unsure if AWS Redis has any modifications on top of the Redis software itself, which would affect the speed, or even make it a bit slower. For example I know MS Azure's version of Redis restricted certain commands, and from a quick search AWS does something similar: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...
We compared DF with Redis OSS and KeyDB OSS, we have not used any of managed services for the comparison. AWS was used as a platform for using EC2 and cloud environment.
Several years ago there was memcachedb, which could flush stuff to disk. While this operation was expensive, it was also useful, because you could restart instances without being overwhelmed by missing keys (data).
For the latter: your application quickly grinds to a halt if you need to build your cache from ground up after some kind of crash. This is a deal-breaker for many.
Indeed :)
The moment i learned about io_uring I got hooked up immediately.
People like Jens, Pavel are the type of innovators that make sure Linux stays on top of its game for the next generations.
From my experience, Redis performance seems to be all over the place depending on the circumstances. In what cases does this solution perform well and where does it fail? Most caches I worked with really love to have everything close by at low latency and works best when the consumers have about as much memory as the cache in the first place.
I honestly think it's impossible to reuse Redis OSS to make it multi-threaded besides what KeyDB did. Btw, I did reuse some of the code - but it's around single value data-structures like zset, hset etc.
With multi-threading you need to think about all things holistically. How you handle backpressure, how you do snapshotting. How you implement multi-key operations or blocking transactions. So you need special algorithms to provide atomicity, you need fibers/coroutines to be able to block your calling context yet unblock the cpu for other tasks etc. All this was designed bottom up from scratch. Seastar could work theoretically but I am not a fan of coding style with futures and continuations - they are pretty confusing, especially in C++.
My choice was using fibers - which provide more natural way of writing code.
I have not designed the distributed long thingy. Will do it in the next 2 months.
This looks like awesome work! I appreciate you operationalizing of some of the best things to come from computer science in the past decade or so.
Out of curiosity, are you discovering any new bottlenecks to performance outside of the software, given Dragonfly is able to process far more qps than most systems? I imagine the network and disk I/O could become stressed, but also I wonder if it breaks any assumptions of cross-core performance, hypervisors, etc. I know that cloud offerings typically mean that you can attach ginormous disk IOPS and NICs, but surely there are limits.
I was mostly running on AWS. In terms of hardware, for small-packets loadtests, most systems are constrained on throughput, i.e. number of packets per second. Some instances saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond in throughput. The best systems network-wise are c6gn family types. They are also better than instances that other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor.
In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem:
With share nothing architecture, I use a lot of thread-local counters that I aggregate only when stats are pulled.
As a general note, I expect that Dragonfly will stay very performant with the tailwinds from recent hardware advancements. For example, c7g (Graviton 3) is much better than c6g and DF shows it.
I'm curious: since you're targeting Graviton, and AFAICT Graviton 2 and up support ARM LSE, have you tried directly using LSE for metrics? ARM LSE offers STADD to add to a 64-bit memory location without reading the contents or ordering the access. (I think LDADD with XZR as the destination is identical, but I could be missing some subtlety. I'm not an ARM expert.) You might be able to get good performance by using a global counter and just STADDing to it. x86 will perform terribly if you use XADD because it has no corresponding optimized forms.
Wow, its beyond my knowledge:)
I use my software skills to write a performant software but rarely i go to the assembly level. And i have zero knowledge about specifics if each command on every architecture. The rare exception is a code in dashtable. the authors of the paper designed it to allow vectorizing of the find() operation.
Interesting. Have you considered/tried to use fdio/user space networking? In my experience, it greatly improves throughout (simple ip forwarding can be more than 10mpps per cortex a53 on some platforms). Fdio also has a so that you can preload in order to use its ip stack in your app (instead of Linux's).
Have you thought about also offering dragonflydb as a library, like sqlite? This would avoid context switching when running everything on the same machine.
The license is different. Redis core is BSD 3 clause, which is a permissive open source license. Dragonflydb uses the Business Source License, with some additional use grants. Depending on your situation, that could be an impediment.
I've been working on DF for the last 6 months and I implemented ~130 commands. It's still missing about 40% of the commands to reach parity with v5/v6.
I will continue working on DF. Primary/Secondary replication is my next milestone.
Quite a few :) I wish i was 20 again - to have time to learn the language and to implement df in rust :) i am an old with old habits. Btw, I spent a short time coding in rust last year and i understand why the fuzz.
curious about the consistency guarantees. redis is single threaded which seems old fashioned but guarantees that all transactions are serial, so there is no chance of conflict between reads, no issue with race conditions , a complicated problem to solve once concurrency is introduced . if the redis protocol is used how is this problem addressed?
Yes this problem is complex and i guess this is why it have not been solved before.
We do provide atomicity guarantees for all operations like Redis! We use an algorithm from a 2014 paper - see our readme, we provide the link to the paper.
I do not know about aerospike, have never run their software but Scylla are the champions in what they are doing. Having said that, Scylla provide full durability guarantees, that means they need to touch a disk upon each write or update. Disk I/O is relativle expensive in the cloud and requires either large instances or special storage optimized family types like i3 etc.
Long story short, I do think Dragonfly is the fastest in-memory database in terms of throughput and latency today. We will see if we manage to stay this way when we extend our capabilities with SSD tiering.
they need at least perform fdatasync before replying to a client, imho. so I do not think it's every x seconds. I am not deeply familiar with scylla so maybe I am mistaken.
I think you're wrong. It does that every x seconds. The disk writing is async and it's meant that you protect data-loss by keeping replicas in different availability zones.
interesting. i still need to learn a lot about how other systems provide durability guarantees. Based on what you are saying, Scylla guarantees H/A instead of full durability. I was thinking about WAL (AOF) for dragonfly and I kinda thought to start with "fdatasync once in every X seconds", but dragonfly is not a database. I would expect stronger guarantees from databases. Maybe it's inpractical to expect this in cloud environments...
I remember that I read about Anna. Very interesting paper.
From what I remember that require that the operations will be conflict free.
I think came to conclusion that it could be interesting as an independent (novel) store but not something that can implement Redis with its complicated multi-model API, transactions and blocking commands. I do not remember all the details though...
I've wrote in in the background section of Readme.
Basically, I worked in a cloud company in a team that provided a managed service for Redis and Memcached. I witnessed lots of problems that our customers experienced due to scale problems of Redis. I knew that these problems are solveable but only if the whole system would be redesigned from scratch.
At some point I decided to challenge the status quo, so I left the company and..and here we are.
Sorry, missed your second question.
Redis is using fork-based save for snapshotting and full replica sync. This means that memory pages are duplicated on write. That consumes more memory than DF implementation that implements algorithmic point-in-time snapshotting. In addition DF was designed to use less overhead memory in its keys table. As for the why, you can read about it here: https://github.com/dragonflydb/dragonfly#background
I find the Redis benchmark suspect. Did you disable write to disk and live snapshotting? In production Redis shouldn’t be configured with write to disk.
Not really but I read pelikan posts by twitter team.
One thing in common - we both thought that cache-based heuristics can be largely improved compared to memcached/redis implementations. We did it differently though. I think our cache design has academic novelty - I will write a separate post about it.
hey, great work. I couldn't find the specifics of the benchmark. Is there, by any chance you compare 1 instance of single threaded redis running on 64vcore to a multithreaded key-value store?
Can we see such disparity in benchmark even if we run Ncore instances of redis in parallel?
Thats what i did. I did not try to cheat. Running N redises is a completely different product. You will have n listening ports and your loadtest program will need to connect to each one of them. The power of dragonfly is that it hides all this complexity. You are saying that comparing single threaded redis to df is unfair, and i say - comparing n redises to df is unfair because you compare different products.
FoundationDB should have been included in their perf comparison. It’s ACID compliant and a distributed Value/Key store.
For SSD based storage, it’s getting 50k reads/sec PER core and scales linearly with # of cores you have in your cluster. (They achieved 8MM reads/sec with 384 cores)
FoundationDB does have impressive performance, but implementing compound operations like INCR, APPEND, etc. would require at least an additional network round trip between the client and the server.
For example, INCR would require one read followed by one write of the new value, and of course this will result in very inefficient mutation range conflicts (which must be retried for another couple of round trips) if you have frequent updates of the same keys in multiple concurrent transactions.
I had completely forgotten about those built-in ones. Right, so some operations can be made fully atomic server-side, but there are still a bunch of ops (append, pubsub, set commands such as SADD, etc.) that would need round trips.
SADD wouldn't need a round trip since it is idempotent. You would just do a blind set which doesn't even require a transaction in FDB. Likewise, most of the other ones can be reframed / designed to not require round trips. I would still not use it as the backend unless I already had FDB in my solution and this was just an additional database type that I needed without ultimate performance as the constraint.
SADD needs to return a value indicating whether the member was added or not, so I don't think you could avoid a round trip, since FDB doesn't have a "set if not exists".
FDB scales really well with reads, but it will bottleneck on writes a lot faster than you think. For write heavy workloads, Redis and Dragonfly would both leave FDB in the dust (and I say this as a big FDB user/enthusiast).
No there is much more to it than that. FoundationDB has a completely different architecture that is optimized for completely different things than Redis is. It’s not just about writing to disk vs. not.
Redis is basically a very performant, single-threaded (mostly) single-node in-memory datastructures system with an efficient and readable server protocol strapped to it.
FoundationDB is a completely different beast that has like 6+ distinct roles, and is optimized almost exclusively for interactive serializable transactions, range reads, and correctness.
They’re just completely different things, I recommend reading the FoundationDB paper to get a sense for its architecture. The amount of “steps” involved in processing an FDB write is much higher than in Redis.
I think the Background section[0] is pretty helpful. One paper they cite is from 2014, another from 2020. The use of io_uring as well is also somewhat novel.
Redis and memcache are in memory key value storage. Writing to disk is not a primary function of those systems, it’s only used for taking snapshots or backup of the data. io_uring isn’t used as the core functionality and thus that alone wouldn’t make DF “modern”.
Yes, we use io_uring for networking and for disk. io_uring provides a unified interface on linux to poll for all I/O events. Re disk - we use it for storing snapshots. We will use it for writing WALs.
And we have more plans for using io_uring in DF in the future.
I had the same thought, but then I realized IBM DB2 was 16 years old when Redis was released, close to Redis' age now. There is a whole generation of programmers that may consider MongoDB a "legacy database".
I like the redis protocol compatibility and the HTTP compatibility, but from the initial skim through I guess you are using abseil-cpp and the home-grown helio (https://github.com/romange/helio) library.
Could you get me a one liner on the helio library is it used as a fiber wrapper around the io_uring facility in the kernel? Can it be used as a standalone library for implementing fibers in application code?
Also it seems that spinlock has become a defacto standard in the DB world today, thanks for not falling into the trap (because 90% of the users of any DB do not need spinlocks).
Another curious question would be - why not implement with seastar (since you're not speaking to disk often enough)?
Yes, helio is the library that allows you to build c++ backends easily similar to Seastar. Unlike Seastar that is designed as futures and continuations library, helio uses fibers which I think simpler to use and reason about. I've wrote a few blog posts a while ago about fibers and Seastar: https://www.romange.com/2018/07/12/seastar-asynchronous-c-fr... one of them. You will see there a typical Seastar flow with continuations. I just do not like this style and I think C++ is not a good fit for it. Having said that, I do think Seastar is 5-star framework and the team behind it are all superstars. I learned about shared-nothing architecture from Seastar.
Re helio: You will find examples folder inside the projects with sample backends: echo_server and pingpong_server. Both are similar but the latter speaks RESP. I also implemented a toy midi-redis project https://github.com/romange/midi-redis which is also based on helio.
In fact dragonfly evolved from it.
Another interesting moment about Seastarr - I decided to adopt io_uring as my only polling API and Seastar did not use io_uring at that time.
Thanks for taking the time to reply - yes in fact seastar does not use io_uring but it's rust equivalent glommio does use it (IIRC it is based on io_uring). Any reasons for using c++ instead of Rust (are u more familiar with it? or just the learning curve hinders the time to market? or is it the Rc/Arc fatigue with rust async? I guess Rust should be a fairly easy language to pick up for good c++ programmers like you)
If I would choose another language it would be Rust. Why I did not choose Rust?
1. I speak fluently C++ and learning Rust would take me years.
2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve tha mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible.
I'm no longer involved in Redis, but I would love if people receive a clear information: from what the author of Dragonflydb is saying here, that is, memcached and Draonflydb have similar performance, I imagine that the numbers provided in the comparison with Redis are obtain with Redis on a single core, and Draonflydb running on all the cores of the machine. Now, it is true that Redis uses a core per instance, but it is true that this is comparing apple-to-motorbikes. Multi-key operations are possible even with multiple instances (via key tags), so the author should compare N Redis instances (one per core), and report the numbers. Then they should say: "but our advantage is that you can run a single instance with this and that good thing". Moreover I believe it would be fair to memcached to clearly state they have the same performance.
EDIT: another quick note: copy-on-write implementations on the user space, algorithmically, are cool in certain situations, but it must be checked what happens in the worst case. Because the good thing of kernel copy-on-write is that, it is what it is, but is easy to predict. Imagine an instance composed of just very large sorted sets: snapshotting starts, but there are a lot of writes, and all the sorted sets end being duplicated in the process. When instead the sorted sets are able to remember their version because the data structure itself is versioned, you get two things: 1. more memory usage, 2. a lot more complexity in the implementation. I don't know what dragonflydb is using as algorithmic copy-on-write, but I would make sure to understand what the failure modes are with those algorithm, because it's a bit a matter of physics: if you want to capture a snapshot at a given Time T0 of a database, somehow changes must be accumulated. Either at page level or at some other level.
EDIT 2: fun fact, I didn't comment something about Redis for two years!
Thanks for providing your feedback. As Redis Manifesto states - our goal is to fight against complexity. antirez - you are our inspiration and I seriously take your manifesto close to heart.
Please allow the possibility that Redis can be improved and should be improved.
Otherwise other systems will eventually take its market apart.
I appreciate your comments very much. I've wrote about you in my blog. I am an engineer and I disagree with some of the design decisions that were made in Redis and I decided to do something about it :)
to your points:
1. DF provides full compatibility with single node Redis while running on all cores, compared to Redis cluster that can not provide multi-key operations across slots.
2. Much stronger point - we provide much simpler system since you do not need to manage k processes, you do not need to *provision* k capacities that managed independently within each process and you do not need to monitor those processes, load/save k snapshots etc. Our snapshotting is point in time on all cores.
3. Due to pooling of resources DF is more cost efficient. It's more versatile. We have a design partner that could reduce its costs by factor of 3 just because he could use x2gd machine with extra high memory configuration.
Regarding your note about memcached - while we provide similar performance like memcached our product proposition is anything unlike memcached and it's more similar to Redis. Having said that - I will add comparison to memcached. I do believe that memcached as performant as DF because essentially it's just an epoll loop over multiple threads.
Re you comment about snapshotting. We also push the data into serialization sink upon write, hence we do not need to aggregate changes until the snapshot completes.
The complex part is to ensure that no key is written twice and that we ensure with versioning. I do agree that there can be extreme cases where we need to duplicate memory usage for some entries but it's only for the entries at flight - those that are being processed for serialization.
Update: re versioning and memory efficiency. We use DashTable that is more memory efficient that Redis-Dict. In addition, DashTable has a concept of bucket that is comprised of multiple slots (14 in our implementation). We maintain a single 64bit version per bucket and we serialize all the entries in the bucket at once. Naturally, it reduces the overhead of keeping versions. Overall, for small value workloads we are 30-40% more efficient in memory than Redis.
The complexity here can be seen in two ways: complexity of deploying more Redis instances, or complexity of the single instance. It's a trade off. But I think that Redis may go fully threaded soon or later, and perhaps your project may accelerate the process (I'm no longer involved, just speculating).
1. Your point about Cluster, I addressed it many times: the point is, soon or later even with multi-threading you are going to shard among N machines. So I believe that to have this problem ASAP is better and more "linear".
2. Already addressed in "1" and my premise.
3. Yep there are advantages in certain use cases related to cloud costs and so forth, that's why maybe Redis will end fully threaded as well.
About memory efficiency, what I meant is that to have versioned data structures, that is an approach to do user-space copy on write even in the case of multiple changes to large single keys (big sorted set example), you need more memory likely, to augment the data structure. Otherwise the trick is to copy the whole value, that has other issues. It's a tradeoff.
> soon or later even with multi-threading you are going to shard among N machines
In a world where cloud providers offer instances with terabytes of memory and 128 vCPUS (e.g. aws x2iedn.32xlarge family maxes out at 4TB, gcp m2 family maxes out at 12TB) is that really inevitable? Applications serving 10s of millions of users likely won't come anywhere close to that limitation.
It's interesting comment, phamilton. I agree with antirez and I agree with you.
full discloser - I am ex-googler. I believe that "horizontal scale" movement staryted from Google. You could see it in their GFS and Mapreduce papers from early 2000s. And by that time they were completely right, of course. Geniuses of Jeff Dean and Sanjay Gwattemat put Google year ahead compared to other state of the art for more than a decade. There was not even one system developed in Google that is not horizontally scalable (i am omitting acquisitions here). We used to joke in 2009 that the most expensive server we have is our perforce server. Nowdays it's internally developed source control system that is backed by Bigtable.
So of course, antirez is right, of course! If you need infinite scale - you must go horizontally.
But the reality is that most companies and most use-cases do not need terrabytes of data. I would say that today the comfort zone for Dragonfly is upto 512GB per instance (1). So dragonfly solves the issue for... I would say 99% percent of the use-cases. Only the last percentile would need horizontal scale, and probably their business is already big enough, so that they can affort a high-quality eng team to work with horizontal clusters.
(1) We need to improve some things (mainly around serialization format of rdb) to reach another magnitude of 4TB. Nobody wants to wait for days to load a 4TB snapshot.
512GB is probably good enough for 99%. 4TB is probably good enough for 99.9%.
Context for all this: I lead tech at Remind, where we support 30M MAU with around 1 engineer per million MAU. Years ago we focused heavily on horizontally scaling, including our datastores. We stored a few hundred GB in clustered redis, dozens of TB in dynamoDB, a half dozen postgres clusters, etc.
The past year we've reversed course and doubled down on AWS Aurora. Every time we've moved data into Aurora our stability has improved, our devs can move faster and our costs go down.
We've got an order of magnitude headroom in Aurora and frankly our code is far from well optimized. There's so much to be gained from simplicity.
As one of the folks that currently works on Redis, I want to highlight the "Redis can be improved and should be improved". There is a lot of really good ideas put forth that are likely worth consideration in the Redis project as well. There has been a lot of conversations about renewing multi-threading, especially to address the point of simplifying management and better resource utilization.
Glad to see you guys made a lot of progress, although a little disappointing you chose to go down the path of building yet another source available DB and not contributing to open source.
I think that is a pretty baseless claim, even more so given that the project had a complete leadership change only 2 years ago. There is a lot of interest in revamping the internals of Redis while still trying to maintain the original tenets that established Redis. There isn't much justification for saying it could only be changed from the outside when, AFAIK, you didn't try to engage with the system at all.
Believe me, I kinda tried to engage with the system. Maybe a bit different system ;)
I based my clain on my personal experience and it's not related to Redis specifically. People are people. They become defensive when something new tries to replace old views. Usually it does not work when you have no leverage and challenge the status quo within the system that has no good reason to change i.e loose money and margins due to increased efficiency of the underlying infrastructure. It's a classic innovator's dillema. Imho...
Sorry to hear that. But let me quote antirez response from this topic:
"...But I think that Redis may go fully threaded soon or later, and perhaps your project may accelerate the process". Lets at least agree that if Dragonfly will push redis to evolve, it's a good thing.
Yes, the core folks in Redis now have outlined a plan for moving towards multi-threaded awhile ago. It's honestly not made a lot of progress because raw performance matters a lot less than is argued here. As was succinctly mentioned by antirez already, Redis scales comfortably to 100s of millions of QPS with cluster mode. So, it's really building a lot of custom functionality to support better vertical scaling. Which is useful, especially when the vertical scaling keeps you on a single process.
The conversation happened here, https://github.com/redis/redis/issues/8340, and it's not like the most pressing issue for the project. It's also not as complex as what was implemented for dragonfly, which basically has native support from the ground up for concurrent programming during command execution. It would be hard to do in C as well.
> raw performance matters a lot less than is argued here
It matters a lot for where i work, we believe multithreading is holding redis peformance back.
> Redis scales comfortably to 100s of millions of QPS with cluster mode.
is there somewhere i can read more about that? curious about the server needs to do that. i worth key/value clusters a little larger then that. if possible it would be cool to use redis for it.
> It matters a lot for where i work, we believe multithreading is holding redis peformance back.
Do you run stand alone Redis or Redis cluster? Multithreading is a strategy, but it isn't the only strategy to improving raw performance.
> is there somewhere i can read more about that? curious about the server needs to do that. i worth key/value clusters a little larger then that. if possible it would be cool to use redis for it.
Redis scales to about ~1000 nodes with each node supporting about 200k qps, math gets you to around 200m qps as the practical upper bound for Redis these days. Multithreading would potentially help us push clusters to the billion RPS boundary.
I am not going to get into what's better and what's not, especially because I haven't released the v0.1 yet and therefore it's not usable, but I am working on cachegrand which is aims to be (also) a redis compatible platform.
I have done A LOT of research and development before picking up the current architecture (you can see it from the amounts of commits) and I am trying to test as much as possible (Almost 1000 unit tests so far, but there is still plenty to do).
if you look at the repository please bare in mind that:
- there is no v0.1, the code available in the repo only supports the basic GET, SET and DELETE (apart from a few additional commands like HELLO, QUIT, PING)
- the code in main currently supports only storing the data on the disk, which is also why the tests are failing, I am doing some general refactoring and need to bring back the in-memory storage (issue n. 88)
- there are some general performance metrics available on the repo
- don't enable verbose logging, it's currently synchronous :)
- cachegrand is able to fully run in single thread mode so I can actually compare it to redis (well when it will make sense)
- only linux, requires a kernel 5.8 at least (e.g. it's provided by ubuntu 20.04.2 lts, but I didn't really care too much as it will take quite a bit more before I get the first stable version and by that point the kernel requirement will not be an issue anymore)
What I can say is that the project really focus ONLY on performances, therefore is not as memory saavy as redis or similar platforms, and it actually aims more to compete with Redis Enterprise long term than just Redis, on the other end it implements a number of things from the ground to boost massively the performances:
- cachegrand architecture follows almost the share nothing principle with the only exception of the hashtable because it has been built around that need
- I implemented from ground an hashtable capable to deliver lock-free and wait-free GET operations and which uses localized spinlocks for the SET and GET operations, basically the contention is spread across the hashtable instead of being bound to X queues
- the hashtable also support SIMD operations (AVX, AVX2 and AVX512F), it's heavily optmized to reduce the memory accesses is able to embed short strings in the bucket to further reduce memory accesses
- cachegrand will support both memory and ad-hoc backend for the storage that is going to be basically a time-series database (cachegrand is not bound to redis functionalities, the redis command set is just a way to expose these for now)
- I implemented from scratch a fiber library able to do a context switch in just 7ns
- the network and storage backend are modular, currently it really only support io_uring but the goal is to also add XDP+the FreeBSD network stack support for the network (e.g. similar to what has been done with F-Stack and DPDK) and then io_uring with the NVME passthrough for the storage (not sure if I will also add support for SPDK)
- most of the code is built aiming to be zero-copy (there are a few places where it happens right now as I need to fix a couple of things)
Just to underline it, currently it's not possible to play with it, until I merge the branch I am working on, because performances would be terrible (only on-disk storage and currently without caching), the tests are broken for the same reason.
It seems to me you didn't address the main point from parent: did you benchmark your multithreaded implementation vs a single core Redis? Nevermind the amazing advantages that having to spawn 1 process vs N brings, the question is how does your software compare when Redis is used as inteded.
I benchmarked DF vs single core Redis. If there is a constructive suggestion for a different benchmark that compares similar product propositions I will happily oblige and do that. i.e. what do you mean by using Redis as intended?
1. I think this is how Redis the company designed their enterprise solution. You can find architecture documentation on their site.
2. Based on my knowledge it should be more or less equivalent. The reason they put it on the same machine (i am guessing here) is because shards on the same node are behind their Redis proxy, that kinda hides the complexity of connecting to each node separately. it's like a gateway to that machine and to all its redis processes.
I actually think Redis really needs to trend towards a forkless approach for replication. Although copy-on-write provides a predictable form of memory usage, it does cause rather high memory utilization and requires a good bit of over provisioning.
There was an optimization built for Redis 7 where we actually start return copied memory pages back to the kernel, https://github.com/redis/redis/pull/8974, I wonder if the testing provided on Dragonfly includes this optimization.
Interesting project. Very similar to KeyDB [1] which also developed a multi-threaded scale-up approach to Redis. It's since been acquired by Snapchat. There's also Aerospike [2] which has developed a lot around low-latency performance.
True. Keydb tackled the same problems as us. But we chose differrent paths. We decided to go for a complete redesign, feeling that there is a critical mass of innovation out there that can be applied for inmemory store. KeyDb wanted to stay close with the source and be able to integrate quickly with recent developments in Redis. Both paths have their own pros and cons.
I see from the blog posts that you looked at KeyDB and Scylla/Seastar for background. I agree with both approaches - fewer but bigger instances and shared-nothing thread-per-core architecture - and it was a major reason for switching to ScyllaDB in my previous startup.
Will definitely follow this to see how it develops. Good luck.
A lot of projects say "faster" without giving some hint of the things they did to achieve this. "A novel fork-less snapshotting algorithm", "each thread would manage its own slice of dictionary data", and "core hashtable structure" are all important information that other projects often leave out.
I’ve seen the VLL paper before and I’ve wondered how well it would work in practice (and for what use cases). Does anyone know how they handle blocked transactions across threads? Is the locking done per-thread? If so, how do you detect/resolve deadlocks?
It also be good to see a benchmark comparing single-thread performance between DragonflyDB and Redis. How much of the performance increase is due to being able of using all threads? And how does it handle contention? In Redis it’s easy to reason about because everything is done sequentially. How does DragonflyDB handle cases where (1) 95% of the traffic is GET/SET a single key or (2) 90% of the traffic involves all shards (multi-key transaction)?
It's really good questions. I invite you to try it yourself using memtier_benchmark :)
if you pass `--key-maximum=0` you will get a single key contention when doing the loadtest. Spoiler alert - it's still much faster than Redis.
Yes, I can confirm that Memcached can reach similar performance as DF. However, one of the goals of DF was to combine the performance of Memcached with versatility of Redis. I implemented an engine that provides atomicity guarantees for all its operations plus transparent snapshotting under write-heavy traffic and all this without reducing the performance compared to memcached.
Having said that, DF also has a novel caching algorithm that should provide better hit rate with less memory consumption.
You are correct - GET/SET do not require any locking as long as they do not contend and they do not in those benchmarks. You are right that for MSET/MGET you will see lower numbers. But still it will be much higher than with REDIS.
This is our initial release and we just did not have resources to showcase everything under different scenarios. Having said that, if you open an issue with a suggestion of a benchmark that you would like to see I will try to run soon...
The best tool out there is probably jepsen.io
but it requires domain knowledge to properly operate it. The guy behind it is a beast!
Anyway, I wrote lots of unit tests to cover those atomicity issues. You can also write a custom python/nodejs/golang/... scripts that simultenusly write and read from the same multiple keys in such way that some invariant is preserved. For example,
"mset x {i}, y {i}" for random `i` and in parallel do "mget x y" and to check that the response returns same values. You can also test this for other families using transactions like "MULTI; lpush x ${foo}; lpush y ${foo}; EXEC" .. and then similarly test that x and y have exactly the same lists.
Ah thanks I had heard of and then forgotten about jepsen. I had considered what you described but I thought there might be a more targeted approach where you can specify your invariants (like "x can only be in one of two states" or "pushing alway increases the length of a list and popping always reduces it if it's non-empty") and get the tool to try to break them.
I wonder if there could be a tool that instruments in the same way that afl does to try to detect races or inconsistent states.
To me the focus on speed is a wash now. They’re all fast. I’d like to hear about easy cross-region replication and failover as well as effortless snapshot and restoring of backups.
Aside nit-pick: I think is dangerous call anything "db" if is not safely stored with Acid.
People not read docs neither know the consequences of words like "eventual" or "in memory" and star using this kind of software as primary data stores, instead of caches/ephemeral ones...
All of A, C, and I only make sense defined relative to a particular transaction vocabulary. Redis is perfectly ACID, as long as your transactions are those supported by Redis's commands.
Conversely, plenty of DBs with programmable transactions (e.g. SQL) are considered work-a-day "ACID" enough, despite some massive gaps in their transactional model (no DDL in transactions, no nested transactions, atomic only when below a certain size, etc.)
> If you run a database without reading the documentation then you're the dangerous part
I think that covers almost all the whole dev population, for what I see in relation with RDBMS. Lucky us most RDBMs shield the mistakes in their usage, a lot.
That is why I see is "dangerous" to call ephemeral/eventual stores as "db". Marketing/positioning have impacts...
All databases are ephemeral if the person running it don't read the docs. Your comment is hence fully redundant, as opposed to the default single-node install of any DBMS.
This is really cool. Love a section on how things are designed with links to papers, always makes me feel way better about a project - especially one that has benchmarks.
Agreed. Really great piece of technical writing. Probably written by one of the rare developers who write better than I do. I want to clone him/her/they and work with them.
Will be very interesting to see how it plays out. I'm yet to see the database technology completely licensed as BSL to be successful in Open Source strategy.
Elastic made change after it was very popular, MariaDB is same story, and even more so only uses BSL for "Enterprise" components which have very little community adoption.
I think for Open Source Project just starting up concern of "Clouds will steal my lunch" is just stupid. If you're worth for clouds to Adopt you're in 0.1% of all Open Source Projects and already "winning" You can WHEN revisit your license, think how to get to that point, rather than create adoption barriers early on
This looks like a cool project. Is there any support (or plan to support) I/O through kernel bypass technologies like RDMA? For example, the client reads the objects using 1-sided reads from the server given it knows which address the object lives in. This could be really benefitial for reducing latency and CPU load.
I do not know much about RDMA. Our goal is to provide a memory store that is fully compatible with Redis/Memcached protocols so that all the existing frameworks could work as before. I am not sure how RDMA fits this goal.
On the picture Redis tops at 200k/seconds on an instance with 64 cores (r6g), Dragonfly 1400k/seconds, Redis is single threaded DF is not but it only got 7.7x faster how come?
If you run let say 32 instances of Redis ( not using HT ) with CPU pining will be much faster than DF assuming the data is sharding/clustered.
The reason for this is the networking limitations of each instance type.
DF consistently reaches the networking limits for each instance type in the cloud.
On r6g it's 1.4M qps and then it's saturated on interrupts due to ping pong nature of the protocol. This is why pipelining mode can reach several times higher throughput - your messages are big. c6gn is network-enhanced instance with 32 network queues! it's the most capable instance in AWS network-wise. This is why DF can reach there > 3.8M qps.
Each CPU core doesn't have it's own independent channel to memory, there are usually 2-3 channels to DDR memory shared by all cores via multiple intermediate caches (usually shared hierarchically).
This looks really good! I gave it a thorough look and is definitely something I’d consider using. Maybe as canary in one of our systems.
Youth of product makes it bit scary to use fully in mission critical systems - given how many problems with Redis started to show up under proper load. But definitely on my watch list.
> Probably, the fastest in-memory store in the universe!
Redis is fast enough. Read/write speed isn't usually the bottleneck, it's limiting your data set to RAM. I've long ago switched to a disk-backed Redis clone (called SSDB) that solved all my scaling problems.
Haha, I thought the same thing until it wasn't! It turns out there are a lot of humans in the world and if you are unfortunate enough to get a large portion of them to start utilizing the software you write you'll find some bottlenecks in almost every system you thought was fast enough.
You can make incredibly fast pseudo dynamic webpages by combining webdis and NGINX SSI. http://nginx.org/en/docs/http/ngx_http_ssi_module.html The way we did this, when the user logged in, we generated their customized content which was the same on all pages -- think of it, this is very common, say, their avatar, the links to their profile etc -- and put it in redis. Now the user gets a customized page without any costly SQL queries, without a scripting language running, it's just NGINX processing SSI include virtuals -- in parallel, no less.
Have Redis and Memcached aged so much we need a modern replacement? Or is this a webdevy 'modern' which just means the first commit is newer than redis' first commit?
memcached was born in 2003.
Redis was born in 2008. Both have strength and weaknesses. DF tries to unite them into a single backend and keep the strengths of each one of them.
> Have Redis and Memcached aged so much we need a modern replacement? Or is this a webdevy 'modern' which just means the first commit is newer than redis' first commit?
The docs make some of the differences clear. Worth reading the GitHub repo readme.
And no, it's not just because of io_uring it is faster. It's also because it's multi-threaded, has absolutely different hashtable design, uses a different memory allocator and many other reasons (i.e design decisions we took on the way).
we use io_uring for everything: network and disk. Each thread maintains its own polling loop that dispatches completions for I/O events, schedules fibers etc. Everything is done via io_uring API. All socket writes are done via ring buffer etc. If you run strace on DF you won't see almolst any system calls besides io_uring_enter
Can’t wait to see this turn productional and feature-rich! I loved Redis Streams and the sheer power of it as a generic database.
Redis is being undervalued for financial applications such as high performance bookkeeping that it doesn’t even have a fsync operation. Hope someone will pick it up.
If it doesn't support macOS or Windows, how do you suggest developers on those platforms use it? An install guide for those platforms (even if it's mostly Docker commands) would be extremely useful.
KeyDB does network I/O and parsing on separate threads, but still has one big lock around the data structures themselves. DragonflyDB does full processing of the transaction on separate threads.
Unfortunately unlikely. Shared nothing architecture works with messaging. So threads send each other messages like "give me the contents of that key" or "update this key". Operations like SET/GET will require a single message hop. Operations like RENAME require 2 hops. Transactions and Lua scripts will require number of hops as number of operations in the transactions. When it's in the same process - the latency penalty in negligible. But between different machines - you will feel it. But who knows, AWS already have very cool UDP-like protocol with very low latency... if this will become the standard inside cloud environments maybe we can create a truly distributed memory store that spans multiple machines transparently.
Yes, VLL algorithm kinda solves the throughput problem. You will still have some issues with fault-taulerance - i.e. what you do if a machine does not answer you.
For intra-process framework it's not an issue (as long as we do not have deadlocks).
It's not a simple sharing. It's actually share-nothing architecture and doing multi-key operations atomically is pretty complicated thing. I used paper from 2014 to solve this problem
In Redis cluster the client needs to be connected to all shards and manage those connections. Multi-key operation on different slots are not supported etc... Maintaining a cluster is not a fun responsibility. DF saves you in most cases from the need to grow horizontly which should be much simpler to maintain and work with.
it is architecturally different. DF has the orchestrator underneath. The general. The guy that handles all the incoming requests and makes sure they execute atomically. Redis cluster is an orchestra without the orchestrator. Everyone to himself. It's share-nothing that is exposed outside. To actually hide the complexity in a *transparent way* is a difficult problem. Researches write articles on how to solve this problem efficiently.
Benchmarks are curiously all based on Graviton2 or Graviton3 instances, it would be interesting to add some Intel/AMD instances to truly compare with Redis...
Controversial opinion: we don't need more databases. We need better application design.
Why do you need a redis/memcache? Because you want to look up a shit-ton of random data quickly.
Why does it have to be random? Do you really need to look up any and all data? Is there not another more standard (and not dependent on a single db cluster) data storage and retrieval method you could use?
If you have a bunch of nodes with high memory just to store and retrieve data, and you have a bunch of applications with a tiny amount of memory.... Why not just split the difference? Deploy your apps to nodes with high amounts of memory, add parallel processing so they scale efficiently, store the data in memory closer to the applications, process in queues to prevent swamping the system and more reliable scaling. Or use an SSD array and skip storing it in memory, let the kernel VM take care of it.
If you're trying to "share" this memory between a bunch of different applications, consider if a microservice architecture would be better, or a traditional RDBMS with more efficient database design. (And fwiw, organic networks (as in biological) do not share one big pot of global state, they keep state local and pass messages through a distributed network)
Weird hill to die on. There’s like a million valid reasons for using an in-memory data store. Off the top of my head: chat apps, real-time video games, caching, (some) data pipelines, etc.
Could share some more on the data pipelines you mention? What kind of data store do you use for this purpose? We are reading a lot of the same data from disk to do transformation (joins) with new data, I’ve been looking for something that might speed this process up but would like to keep the setup small and simple.
I've used Apache Arrow before[1]; in-memory columnar storage. We did some AI/ML stuff with data gathered from social network APIs, but you can probably do a ton of things.
As I understand it, high-performance real-time video games do tend to use the architecture that throwaway proposed, i.e. state is in the game server itself rather than a separate caching layer.
People who like to make decisions and then studiously avoid ever looking at the consequences of their actions don't seem to notice that once caching is introduced, the data model in the system either stops improving or begins to unravel.
It becomes very difficult to benchmark the system looking for design issues because the real cost of a cold cache is very difficult to measure. And the usual argument that we can turn off the cache for performance analysis doesn't hold water for more than a few months.
Pulling the same data twice, three times, ten times in a given transaction are all about the same cost, and it gets me out of trying to figure out how to plumb new data through the sequence diagrams. I can just snatch values out of global state wherever, whenever, and as often as I need them. Once people give up and start doing that, you can't turn off the caches anymore. When you started it cut the cost of the lookup in half, or a third. Now it's four, or five, or sixteen, and that call is now over-represented in the analysis, hiding other, more tractable problems.
You've settled for your broken design, but also cut the legs out from under any of the tools that can dig you out of the hole you've dug.
Caches are just global state under a different name, but slathered in RPC issues.
Some valid business use-cases allow eventual consistency. Think of social apps, for example. FB and TW have excellent engineering teams and they built huge systems that rely on caching layers. Similarly, with gaming companies.
Once you reach a certain scale, caching becomes basically a necessity. I mean, heck, even DNS is cached. The idea that caching is a failure of design is just wrong.
Or you need the best of both worlds. A tool that can store data, serve data, convert code to data, interpret data, display data. Some kind of tool that has at its core a kind of phpmyadmin but that can be distributed on an app store, but for end-users. Something that will handle authentication, authorization, audit, provide confidentiality, scalability, remove the need for third party tools for end users and developers, just one big ecosystem that let's you host entire enterprise software , extend it, customize it for your end users and your power user. Just one big ass tool that will replace all other information management services.
Laracasts are the best tutorials I've ever seen, regardless of language, outside of how php.net/<keyword> search and commenting was structured 20 years ago. They would be the best if they got outside funding to provide all lessons for free.
Anyway, one HTTP request can easily generate 100+ SQL queries under an ORM. Which sounds bad, but is trivially fixable globally without side effects via global "select" query listeners and memoization. I've applied it and seen 7 second responses drop to 100 ms or less simply by associating query strings and responses with Redis via the Cache::remember() function.
I also feel that there's a deeper problem in how web development began as hacking but devolved into application-heavy bike shedding. We have a generation of programmers taught to apply decorators by hand repeatedly, rather than take an aspect-oriented (glorified monkey patching) approach that fixes recurring problems at the framework or language level. I feel that this is mainly due to the loss of macros without runtimes providing alternative ways of overriding methods or even doing basic reflection.
So code today often isn't future-proof (requires indefinite custodianship of conventions) and has side-effect-inducing or conceptually-incorrect changes made in the name of secondary concerns like performance. The antidote to this is to never save state in classes, but instead pipe data through side-effect-free class methods, then profile the code and apply momoization to the 20-% of methods that cost 80+% of execution time.
Also microservices are definitely NOT the way to go for rapid application development or determinism. Or I should say, it's unwise to adopt microservices until there are industry-standard approaches for reverting back to monoliths.
Yes! We just need to get our users to visit the website in order of their user_id, at their assigned time, and then we wouldn't have these random access patterns. /s
I assume the "modern" means usinng io_uring, which means requiring Linux 5.1+. If you stick with lts kernel version, means the latest versionn is Linux 5.4 20th LTS released at 2019-09-24, ubuntu 20.04, debian 11.
If I would choose another language it would be Rust. Why I did not choose Rust?
1. I speak fluently C++ and learning Rust would take me years.
2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto the standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve the mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible. (and yeah - there is zero chance I could write something like helio in Rust)...
What about it makes it a "modern" replacement rather than just a replacement? Is there something about Redis and Memcached that is "outdated" in the (relatively) short time span they've existed (compared to something like C)?
I'd say using io-uring is an example of modern since it has only been available since 5.10 of the kernel. They also sit on some more recent research on data structures that perform well in multithreaded systems.
It is "modern" in the sense that the design and architecture is idiomatic for high-performance software on recent computing hardware. The main advantage of modern (in this sense) implementations is that they use the hardware much more efficiently and completely for current workloads, enabling much higher throughput on the same hardware.
Yes, exactly. When people talk about about "modern" in the context of software performance, it usually denotes software architecture choices. If someone did a forklift upgrade to the software architecture of Redis etc, then they would be modern too.
In practice, software architecture is forever. It is nearly impossible to change the fundamental architecture of software people use because applications become adapted to the specific behavioral quirks of the architecture i.e. Hyrum's Law, which would not be preserved if the architecture was re-designed wholesale.
Mainline memcached is very well tuned, so it's sort of the odd one out here.
But yes, Redis is very much designed against the grain of modern hardware. It also is a very minimalist design, that works well within its limits, but falls down hard when you push those limits, particularly with snapshotting and replication.
Not really, it just implies that the competition is not modern, without qualification. I think asking for qualification in this case is fair if we are to conclude Redis and Memcache have aged to the point of needing a replacement.
The word is often tagged on anything new somebody tries to sell. Better to be specific. The problem is that most “modern” things are very old things sold as new ideas. Cause biz. So, nothing specific against this proj.
