Coincidentally, as someone who worked on this blog, I was surprised (and pleased!) to see that we are not the only ones who felt the need to build a Zookeeper alternative.
It's not the written in Java part, it's the running in the JVM that's the issue. Memory hungry is what I think of when I think Java apps. I'd much rather have a low memory Go or Rust service.
but if you use ZK for what it was designed for (basic cluster role/naming cordination, distributing configs to cluster nodes and similar) then this kind doesn't matter
I mean in a typical use case of it you would
- run it on long running nodes (e.g. not lambda or spot instances)
- run more or less exactly 3 nodes up to quite a cluster size, I guess some use-cases which involve a lot of serverless might need more
- configs tend to not change "that" much nor are they "that" big
what this means is
- java needing time to run-hot (JIT optimize) is not an issue for it
- GC isn't an issue
and if you look at how much memory (RAM) typical minimal nodes in the cloud have it in context of typical config sizes and cluster sizes is also not an issue
through I guess depending what you want to do there could be issues if you use it
- for squeezing through analytics or similar
- setups with very very very large constantly changing clusters, e.g. in some serverless context with ton of ad-hoc spawned up instances, maybe using WASM and certain snapshot tricks allowing insane fast startup time
- you want to bundle it directly into other applications, running on the same hardware and that applications need more memory
but all of it are cases it wasn't designed for so I wouldn't call it an "alternative" but a ZooKeeper like service for different use-cases, I guess
Long-running server processes are not only "not an issue" for the JVM, they're the main use case in which the JVM is superior to AOT compilation! Same is true for C# and the .NET CLR, by the way.
If you're running a Lambda function in which startup time is extremely important, or an embedded application where size and resources are paramount, or even just a short-lived process where you don't care either way... then AOT makes a lot of sense.
But for long-running server processes, just-in-time compilation almost always results is better performance than AOT compilation that cannot optimize at runtime based on what's actually happening.
HN should be full of people who know better, but these discussions feel like piping information into /dev/null. Web devs, students and hobbyists, and other low-information voters just have it in their heads that AOT is always a superior model, and JIT always an inferior fallback, and there's nothing you can say to break through that. There aren't enough people from the business server side world who spend enough time in online discussion forums to correct the narrative.
I agree that a good JIT and the JVM is very powerful, but it's not a silver bullet that magically works for all systems. For example, JIT-compiled code can become uncompiled and recompiled all the time in the same process, resulting in weird performance characteristics. You can definitely fine tune and code in ways that prevent this, and for many apps it won't matter, but when it does matter it's a pain in the butt.
I think there's an important distinction to be made:
Theoretically speaking, given that JIT compilers generally care much more about compilation speed than AOT ones, it is not unreasonable to assume that AOT compilers ought to produce more optimized code.
With that said, this is not the case for both JVM and .NET. Stepping away from ought to is, both have JIT compilers which produce better optimized code than their AOT counterparts, due to a variety of reasons including R&D effort done for JIT throughout their history and JIT allowing to dynamically profile and recompile the code according to its execution characteristics (.NET's Tier 1 PGO Optimized and HotSpot JVM's C2).
Theoretically speaking, JITs have access to strictly more information than AOT, so they ought to be better once you amortize out the timing issues. A really good profiling JIT will do a fast-compile pass the first time through, and then progressively re-compile with knowledge gained from profiling during runtime.
“Has more information therefor better optimized” is bordering on a lie, is the problem.
Theres truth to a point I guess, but it’s also true that lots of “more information” is useless, and sometimes harmful to optimization, and additionally that it’s really diminishing returns.
Truth is as long as you’re not over relying on RAII, JVM will virtually never outperform C++.
That's why there is profile guided optimizations for C/C++.
Which instruments you C/C++ code to collect that needed additional information (on the cost of performance).
Then when recompiling you can feed that collected information into your system.
The problem with profile guided optimization is that it's way more annoying to deploy as on every update you have to deploy it twice once slower then naive and then once faster. And because the slower part might very well be to slow you might want to only deploy it to some nodes of a load balancer and deploy naive compiled versions to the other.
It also means you have a similar slower => faster startup time, excepts it's of your program as whole instead of for each restart of a node.
And while optimized Java likely will never outperform optimized C++ that is Java specific furthermore if we speak about common non manual optimized code which isn't implementing some tight math/CS algorithms (i.e. very common daily code in many companies) then the difference really isn't that big, small enough to make choosing java over C++ for server stuff a "in general" the right choice. (If you are not a company which only gets the "best" programmers like google).
I mean just to put it into context there are companies which had success with stuff like running high speed trading code on the JVM and it was a success. So if you can do that probably Java doesn't have a major performance problem.
> not over relying on RAII
you mean like throwing out all the major improvements of C++ which majorly reduced the probability of non highly expert programmers introducing bugs which could be turned into RCEs?
Suggesting that you shouldn’t be allocating memory and then immediately tossing it doesn’t only suggest that you should practice RCE prone manual strategies.
RAII is dogshit because it encourages and hides the fact of what’s really happening, leading to crazy performance gotchas. Just knowing the gotchas around it is often enough for any half way competent programmer to devise better code.
If the "more information" is useless, it can be ignored. JIT can't possibly be a worse strategy, because a JIT always has the option of simply doing an AOT compile at program start.
That doesn't mean that in practice the JVM is faster than well-written C++; the Java language semantics almost prevent it from doing so in the general case. But in principle, if all else were equal, it should be able to be.
I mean java native compilation is a thing. I imagine you'd get the majority of the lift via that route instead of reinventing the wheel, but it's possible there are caveats to that route for these impls. Even java native/golang have GC heaps, but in general ZK is really low GC churn so I don't see that being the impetus to rewrite.
My guess if anything is that people always complain about ZK being annoying as a second piece of infra to distribute for simple setups, so my guess this is just a prelude to them embedding their keeper into the DB deployment itself which is the same general strategy Kafka is taking (a verrrry loong time) to rollout.
It's been a few years since I've checked in with distributed lock services. Why would someone adopt ZooKeeper after etcd gained maturity? I recall seeing benchmarks more than 5 years ago where a naive proxy like zetcd[0] out-performs ZooKeeper itself in many ways and offers more consistent latencies. etcd has gotten lots of battle-testing being Kubernetes' datastore, but I can also see how that has shaped its design in a way that might not fit other projects.
I think there are plenty of other projects (e.g. FoundationDB, Kafka) that also replaced their usage of ZooKeeper as their systems matured. I guess I'm confused why anyone has been picking up new installations of ZooKeeper.
There is no specific reason to start with ZooKeeper, nor with ClickHouse Keeper, if you want to use another distributed consensus system.
But: every such system is slightly different in the data model and the set of available primitives.
It's very hard to build a distributed system correctly, even relying on ZooKeeper/Etcd/FoundationDB. For example, when I hear "distributed lock," I know that there is 90% chance there is a bug (distributed lock can be safely used if every transaction made under a lock also atomically tests that the lock still holds).
So, if there is an existing system heavily relying on one distributed consensus implementation, it's very hard to switch to another. The main value of ClickHouse Keeper is its compatibility with ZooKeeper - it uses the same data model and wire protocol.
As one of the contributors, I'm always happy to see interest and people using it.
Keeper is a really interesting challenge and we're really open to any kind of feedback and thoughts.
If you tried it out and have some feedback for it, I encourage you to create an issue (https://github.com/ClickHouse/ClickHouse), ask on our Slack, ping me directly on Slack... (just don't call me on my phone)
And don't forget that it's completely open-source like ClickHouse so contributors are more than welcome.
I usually scoff at "written in.." part of such announcements, because it is a sign that the author is focused on the input ("I wrote this in X") not the output (value the user gets)
In this case though, the blog outlines specific reasons why this had to be in C++ (interoperability w. their C++ codebase) as well as benefits that are separate from the language.
It's a huge turn-off for me as well, because I interpret it as the main value it's supposed to deliver (which for me is 0 for the most part). Not talking about this specific project, just generally.
Built in s3 storage immediately sold me. I’ve used something called Exhibitor to manage ZK clusters in the past but it’s totally dead. Working with ZK is probably one of my least favorite things to do.
I used exhibitor in the past too. They were specially useful during the time when zookeeper cluster needed to expand/shrink or move host nodes. Zookeeper dynamic configuration solved that problem, which seems to be also supported by clickhouse keeper, Pretty impressive! Would definitely give it a try.
In particular, it is necessary to enable the `keeper_server.enable_reconfiguration` flag. It is pretty exhaustive coverage but if there is an important use case missing, let us know!
Thanks for this excellent article! Enjoyed it from start to finish. This gave me a good memory of the work we've done at docker embedding our own replicated and consistent metadata storage using etcd's raft library.
Looking at the initial pull request, is it correct that ClickHouse Keeper is based on Ebay's NuRaft library? Or did the Clickhouse team fork and modified this library to accommodate for ClickHouse usage and performance needs?
Yes, you are right ClickHouse Keeper is based on NuRaft. We did a lot of modifications for this library, both for correctness and performance. Almost all of them (need to check) are contributed back to upstream ebay/NuRaft library.
1. can this be used without clickhouse as just a zookeeper replacement?
2. am i correct in that its using s3 as disk? so can it be run as stateless pods in k8s?
3. if it uses s3, how are latency and costs of PUTs affected? does every write result in a PUT call to s3?
1. Yes, it can be used with other applications as a ZooKeeper replacement, unless some unusual ZooKeper features are used (there is no Kerberos integration in Keeper, and it does not support the TTL of persistent nodes) or the application tests for a specific ZooKeeper version.
2. It could be configure to store - snapshots; - RAFT logs other than the latest log; in S3. It cannot use a stateless Kubernetes pod - the latest log has to be located on the filesystem.
Although I see you can make a multi-region setup with multiple independent Kubernetes clusters and store logs in tmpfs (which is not 100% wrong from a theoretical standpoint), it is too risky to be practical.
3. Only the snapshots and the previous logs could be on S3, so the PUT requests are done only on log rotation.
2. ok. so can i rebuild a cluster with just state in s3? eg: i create a cluster with local disks and s3 backing. entire cluster gets deleted. if i recreate cluster and point to same s3 bucket, will it restore its state?
It depends on how the entire cluster gets deleted.
If one out of three nodes disappears, but two out of three nodes are shut down properly and written the latest snapshot to S3, it will restore correctly.
If two out of three nodes disappeared, but one out of three nodes is shut down properly and written the latest snapshot to S3, and you restore from its snapshot - it is equivalent to split-brain, and you could lose some of the transactions, that were acknowledged on the other two nodes.
If all three nodes suddenly disappear, and you restore from some previous snapshot on S3, you will lose the transactions acknowledged after the time of this snapshot - this is equivalent to restoring from a backup.
TLDR - Keeper writes the latest log on the filesystem. It does not continuously write data to S3 (it could be tempting, but if we do, it will give the latency around 100..500 ms, even in the same region, which is comparable to the latency between the most distant AWS regions), and it still requires a quorum, and the support of S3 gives no magic.
The primary motivation for such feature was to reduce the space needed on SSD/EBS disk.
Sometime back, I tried using clickhouse-keeper as zookeeper alternative with few other systems like kafka, mesos, solr, Wrote some notes here: https://pradeepchhetri.xyz/clickhousekeeper/
1. Absolutely. clickhouse-keeper is distributed as a standalone static binary or .deb package or .rpm package. You can use it without clickhouse as ZooKeeper replacement.
2. It's not recommended to use slow storage devices for logs in any coordination system (zookeeper, clickhouse-keeper, etcd and so on). Good setup will be small fast SSD/EBS disk for fresh logs and old logs + snapshots offloaded to S3. In such setup the amount of PUT requests will be tiny and latency will be as good as possible.
All ZooKeeper libraries are compatible with clickhouse-keeper. The most popular and mature is https://kazoo.readthedocs.io/en/latest/. We use it in our integration tests framework (with clickhouse-keeper) a lot.
Man i would love to work at one of these companies that let their engineers go off and implement alternatives to widely used and tested open source projects like Zookeeper merely for speculative performance gains and “… C++”
edit: I'm not even saying this facetiously, it would be freaking awesome.
I've been using Clickhouse Keeper in production environment since it's first release and have been really happy. It makes setting up distributed tables in Clickhouse (replicas/shards) quite easy. The only issues I've seen is when inserting at a super high throughput (>1000/sec writes) which is actually more of a n issue with Clickhouse Merge Table tree settings rather than Keeper itself.
That’s a strong endorsement. I wonder if there’s been any effect where it’s strongly tailored to the API surface area utilized by ClickHouse and there’s any gaps elsewhere
At least two comments spring to mind: this is at least _blogged_ as a drop-in ZK replacement, which for sure is not true of Corrosion, and ClickHouse has Jepsen tests for their distributed KV store, which I don't see any reference to such a thing for Corrosion
Maybe neither of those two things matter for one's use case, but it's similar to someone rolling up on this blog post and saying "but what about etcd" -- they're just different, with wholly different operational and consumer concerns
"ClickHouse, Inc. is a Delaware company with headquarters in the San Francisco Bay Area. We have no operations in Russia, no Russian investors, and no Russian members of our Board of Directors."
Also scary how disciplined US society is, with everyone informally-required to actively cheer the proxy war (while no company "stands with Yemen" for example).
Is it just me or does it look like an "alternative for use-cases ZooKeeper was not intended for"?
E.g. if we quote ZooKeeper:
> ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
and ClickHouse
> ClickHouse is the fastest and most resource-efficient open-source database for real-time applications and analytics.
Like this are completely different use cases with just a small overlap.
Are you perhaps confusing ClickHouse, with ClickHouse Keeper (one of ClickHouse's components?) Sounds to me like ClickHouse is the database, and ClickHouse Keeper is the ZK drop-in replacement. A bit like HBase being a database, and ZooKeeper being a service it is heavily dependent on.
but the question in general about weather ClickHouse Keeper and ZooKeeper are designed for completely different use-cases which happen to be similar and can work with the same interface still stands
and by now knowing more then my original comment I would answer the question with yes
But yes, ZooKeeper is pretty amazing. We are building on the backs of giants.
I'd also argue the RAFT v. ZAB is an important production scale conversation. But, as the blog says, Zookeper is a better option when you require scalability with a read-heavy workload.
You can even migrate your zookeeper to ClickHouse keeper. It requires small downtime, but you will have all your zookeeper data inside and your clients will just work when your keeper will be back
Written in C++ is not a positive in my book. New things created this decade in unsafe languages (where safe options would have worked fine) should be frowned up and criticized as bad engineering.
Software engineering is first and foremost about making good engineering choices. We are proving time and again as an industry that humans CANNOT write mistake-free code.
And when making those engineering choices there are different tradeoffs and constraints to be considered.
The language to use is one of them.
So when a Rust (wild guess) fanboy comes without any background context and makes comments like yours is very telling.
You are correct that bug free software does not exist. But choose a “memory safe” language does not prevent that. A seasoned C++ developer knows how to use memory sanitizers and other tools to guarantee the correctness of its code compared to an average Rust developer that just trust the compiler which, guess what, also may have bugs.
Clickhouse is a Yandex project and at Yandex, they historically use C++ for almost everything, I guess it's part of their culture (probably the founders were C++ programmers?) Their web services such as Yandex Taxi's backend (Uber's equivalent) are also written in C++ which is unusual for webdev nowadays.
It's been a struggle getting Clickhouse accepted here in the U.S., despite its technical prowess, even prior to the war in Ukraine.
I know, I read the blog:
> ClickHouse, Inc. is a Delaware company with headquarters in the San Francisco Bay Area. We have no operations in Russia, no Russian investors, and no Russian members of our Board of Directors. We do, however, have an incredibly talented team of Russian software engineers located in Amsterdam, and we could not be more proud to call them colleagues.
The FUD is really hard to overcome. This is coming from someone who advocated for Clickhouse, sent some PRs, and did a minor code audit.
1. Snapshots and logs take much less amount of space on disk due to better compression.
2. No limit on the default packet and node data size (it is 1 MB in ZooKeeper)
3. No zxid overflow issue (it forces restart for every 2 bn transactions in ZooKeeper)
4. Faster recovery after network partitions due to the use of a different distributed consensus protocol.
5. It uses less amount of memory for the same volume of data.
6. It is easier to setup, as it does not require specifying the JVM heap size or a custom gc implementation.
7. A larger coverage by Jepsen tests. (This could be hard to believe, but true - ZooKeeper is tested by Jepsen, but Keeper takes the existing tests and adds more).
8. The possibility to store snapshots and previous logs on S3.
C++ isn't a key detail, just a consequence of the fact that the main ClickHouse code base is written in C++.
If you need a distributed consensus system but not necessarily compatible with ZooKeeper, there are plenty of options: Etcd, Consul, FoundationDB...
2. is configurable (with specific caveats)
3. I believe this was solved a very long time ago? Don't epochs rollover automatically now?
6. is this remotely relevant? You still want limits in cloud deploys so not sure how this is remotely a consideration given it takes 2 minutes when you first set it up to use best practices settings.
Disclaimer: I worked on this blog with the team at ClickHouse.
I like your suggestions!
Some of the benefits we summarized in this summary page https://clickhouse.com/clickhouse/keeper include ease of setup and operation, no overflow issues, better compression, faster recovery, (dramatically) less memory used, etc..
There was actually a reason why C++ was important for us at ClickHouse, and it's because C++ is our main code base and managing a Java project as part of it was not natural, but you right - for standalone use of this alternative, that doesn't matter.
Given that their entire code base is written in C++, and switching to a different language would be a significant retooling for the team, I think it's reasonable to assume that it did not. Language choice is rarely made on the grounds of specific features, and is often made on the grounds of ergonomics and team knowledge.
A more revealing question is, "How are you dealing with memory safety in this implementation?" There are ways to improve memory safety in C++ through tooling and idiomatic style. Are these things being used?
CBMC is subtle and will require some code changes to use effectively.
The real key for using it, in my opinion, is to isolate individual classes and functions. Avoid instrumenting code with recursion and loops, and focus on defining and verifying function contracts, class invariants, and resource / memory lifetimes.
It will require a significant amount of work to mock up standard library and third party library APIs, but the real beauty of CBMC is that once you define the interface contracts for these APIs and libraries, you can verify every use of them.
I used CBMC previously to verify proper usage rules with C / JNI integration. JNI can be one complicated beast, and CBMC handily managed rule checks for its use.
I'm an extremely careful developer who unit tests everything and strives for 99% coverage. CBMC was still able to detect a memory overwrite flaw in a networking library I wrote that was based on undefined behavior due to integer promotion and offset math. This passed the various sanitizers and unit tests I had in place, but CBMC was able to reduce it to an actual crash condition that was potentially exploitable.
I don't think I can over-emphasize the usefulness of this tool.
Since the article states that they're using Raft and not ZAB for the consensus algorithm and leader election, it must be less prone to bugs when it comes to electing a leader. Since Raft is easier to reason about and the leader election process is more straightforward (Raft minimizes the chance that any two nodes will be candidates at the same time and thus avoids starting multiple concurrent elections).
Also, we apply many different faults in our Jepsen tests which are run 3 times a day and we never had a problem with leader election. I know this doesn't confirm that there is no bug in it but it's pretty reassuring I would say.
If even Google, Mozilla, Apple and Microsoft can't get C++ right in their apps with all the fuzing tools they invented I don't have much faith other companies are going to fare better with memory safety and C++.
Java vs C++ is very important implementation detail, especially for 'the benefits to the user'. Java is commercial platform which requires a fee to Oracle if used in enterprise, while C++ complied binary does not.
That's a _very_ incorrect statement. You can use any OpenJDK (which is GPLv2 with classpath exception) distribution you want to run Apache Zookeeper without having to have any agreement with Oracle or pay any fee. The Oracle JDK is just Oracle's commercial version of their OpenJDK distribution with Oracle support.
You can use the OpenJDK distro shipped in your Linux distro (RedHat, Debian, etc.), you can use Microsoft's OpenJDK distro[1], you can use the Eclipse OpenJDK distro, you can use Amazon's OpenJDK distro [3] and there are a whole bunch more.
Looks like folks at StreamNative did as well, with their Oxia project: https://github.com/streamnative/oxia. They were just talking about this yesterday at Confluent Current ("Introducing Oxia: A Scalable Zookeeper Alternative" was the title of their talk). https://streamnative.io/blog/introducing-oxia-scalable-metad...
Seems to be a trend :)