Since the article states that they're using Raft and not ZAB for the consensus algorithm and leader election, it must be less prone to bugs when it comes to electing a leader. Since Raft is easier to reason about and the leader election process is more straightforward (Raft minimizes the chance that any two nodes will be candidates at the same time and thus avoids starting multiple concurrent elections).
Thanks for this excellent article! Enjoyed it from start to finish. This gave me a good memory of the work we've done at docker embedding our own replicated and consistent metadata storage using etcd's raft library.
Looking at the initial pull request, is it correct that ClickHouse Keeper is based on Ebay's NuRaft library? Or did the Clickhouse team fork and modified this library to accommodate for ClickHouse usage and performance needs?
Yes, you are right ClickHouse Keeper is based on NuRaft. We did a lot of modifications for this library, both for correctness and performance. Almost all of them (need to check) are contributed back to upstream ebay/NuRaft library.
I wrote the initial implementation of the raft subsystem and it was definitely not a copy/paste. We started from scratch (using etcd's core raft) with the transport layer being grpc. My initial experiment could be found in this repository [1]. I then took the code from my initial experiment and included this into Swarmkit [2]. From there we went through many iterations on the initial code base and improved the UI with Docker swarm `init`/`join`/`leave` to make the experience of managing the cluster "friendly".
We spent quite some time evaluating different raft and paxos implementations (mostly Consul and etcd raft libraries), and found out etcd to be the most stable and flexible for our use case. It was very easy for example to swap the transport layer to use grpc. The fact that etcd implementation is represented as a simple state machine makes it also much easier to reason about under complex scenarios for debugging purposes, instead of digging into multiple layers of abstractions.
In retrospect, this came with quite a learning curve. We've had to deal with issues caused by our own misunderstandings on how to use the library properly. At the same time the fact that the developers favored stability as opposed to user friendliness was exactly what we found attractive using etcd's raft. Additionally, CoreOS developers were super friendly and helpful to help us fix these issues. We've reported and fixed some bugs as well. Kudos to them for all the help they provided at the time.
What I remember is, during DockerCon in June 2016, I went into the code to see how it worked, and I found a top-level file setting up data structures and handlers that seemed to be 90% the same as the equivalent file in etcd. And the underlying implementation is reused via vendoring.
Maybe this rings a bell with you and you can tell me what I saw, because I can't find it now.
Maybe I dreamed the whole thing.
I did, and still do, think integrating etcd into Swarm Mode was a masterstroke; we had spent the previous two years working to avoid "first you must install etcd" in a different way that nobody got. Afterwards we created kubeadm to ape the 'init' and 'join' functionality.
This is without considering students whose parents have low revenue, thus getting access to financial help from the government. This was my case. Technically, I was paid to go to University, with 3600€ per year for being one of the top students.
I have been using Ghost for more than four years now (I tried using Wordpress before but I was left frustrated). Although I know there are probably attractive alternatives (Hugo/Netlify), I don't feel the urge to try something else. I like the Ghost editor and it provides with a nice and focused experience of writing [1].
I use a very simple, minimalist (albeit, slightly modified) theme called Oscar [2].
In my French University, group or individual assignments had a very low coefficient compared to written exams (where it was almost impossible to cheat as this was basically like white board coding and students would be spaced by 2/3 seats between them).
It was in the order of 20%(group/coding)/80%(written exam). This had the effect of eliminating 50% of all students on the first semester for the first year of CompSci (DistSys) Master's Degree. So even if a group cheated on group/coding assignments, they most likely wouldn't go through the written exam. We called it: "The Purge".
From then, very few instances of cheating (topics were changed every year). One group got caught cheating on another one during the second year (iOS objectiveC class) and they got a failing grade.
Almost every professor would use automated code checkers on group and individual assignments. And if you get caught, you have to go through thorough questioning by the professor, eventually getting a failing grade and having to go through the second session of written exams (which was most of the time harder than the first session).
What was unavoidable though were group projects with only one person working on the project and the others doing nothing or just spending time preparing for written exams. This happened to me quite a few times. I didn't mind as I was learning a lot, but it got me exhausted for written exams where I was getting good but not outstanding grades.
Just talking about glassdoor, and their rankings are to take with a big grain of salt, especially for early stage startups and medium sized companies.
I worked for a company that had some issues with management and saw its glassdoor ranking plummet after several low scores were given by past employees with mention of harassment. They didn't claim the profile on glassdoor at the time. Unsurprisingly, after a few really negative reviews, they most likely encouraged current employees to post positive reviews to counter-balance for the purpose of claiming the profile. It was obvious because positive reviews were for most of them on the same date or very close to each other and came in bulk while reviews were spaced by weeks or months before (most of them rather negative).
The gist of it is that while negative reviews may not reflect the current state and culture of a company (disgruntled employees happen), positive reviews could also be misleading. And I'm not even accounting for people that have an interest for putting up positive reviews regardless of the working environment because they have big stakes into the company.
Best thing to do is to chat with several employees in different departments and ask for their honest opinion and share their experience. Generally people are open about the pros and cons of their own working environment if you talk to them behind closed doors.
This applies to the entire industry, we are often inflating our products with terms that do not describe the reality. Technical accuracy is important because it can drive a purchase decision when comparing features with a competitor. Companies and individuals invest time and money on these software/services, misleading them with inaccuracies can harm them directly or their business.
At least it's an honest statement from SpiderOak, it's better to fix a misuse of a term and admit an error than throwing a misleading term describing a product used by thousands of people and then delete it as if nothing happened.
When I see this post, I cannot help but think of how Docker described "Swarm mode" orchestration features during DockerCon 2016 using the terms "self-healing" and "self-organizing". Obviously, "Swarm mode" was neither "self-healing" nor "self-organizing" and a possibility is that they had no idea what those terms meant, but it looked good on paper and from a marketing point of view. While they have fixed it in the documentation after pointing this out internally, these terms have leaked in many blog posts and are still in plenty of talks recording on Youtube. It became hopeless to stop the spread of misinformation.
Despite this change, a lot of SpiderOak customers are still going to use the term Zero-Knowledge to describe the software to their friend/co-workers or business partners. The term will stick to them for awhile.
For having worked with Jessie while she was still working at Docker (I no longer work there either), I can confirm that. She is genuinely radiant and her excitement for technology in general is contagious.
She was always around to help or hack something cool. I remember the night of the release of Docker 1.9, we stayed late at the office to merge the last PRs and run test suites before releasing the binaries. Jess did not have to be there but she stayed to help Tibor (another amazing human being and prolific contributor) and myself 'til 23pm on the release process. You know you are surrounded with talented engineers when you can count on them regardless of the difficulties encountered. That night, her presence lifted a big burden off my shoulders.
I'm not sure I would have had the same excitement for Docker before joining if she was not working there. To me, she had a huge impact on the craze towards containers.
Amazing work on this website! It encourages exploration and navigating the folders to see all the content.
reply