
Red Hat contributes etcd to the Cloud Native Computing Foundation - pplonski86
https://www.redhat.com/en/blog/red-hat-contributes-etcd-cornerstone-kubernetes-cloud-native-computing-foundation
======
Rapzid
> it demonstrated its quality early on

I find that slightly revisionist though I suppose it depends on your
definition of "early on".

To my recollection etcd had a very rough patch early on until they overhauled
their raft sub-system. Hashicorp caught some flak at the time for giving the
raft implementation a hard pass and writing their own. Go ecosystem
fragmentation was particularly bad and a very hot topic at the time. I believe
that etcd's new found stability and subsequent track record after the new
implementation vindicated Hashicorp somewhat.

~~~
ideal0227
Let me share a little bit more story here.

The original go-raft is one of the first raft implementations. At that time,
the raft paper was not even officially published. Many other attempts around
that time were not very successful, including go-raft.

Making a production ready consensus algorithm is not easy:
[https://www.cs.utexas.edu/users/lorenzo/corsi/cs380d/papers/...](https://www.cs.utexas.edu/users/lorenzo/corsi/cs380d/papers/paper2-1.pdf).
Things like pipelining, batching, flow control, asynchronous snapshot were not
extensively explored in the context of raft. And not much effort has been put
into testing due to the immaturity of the applications of raft at the time.

We realized the problem a few months after etcd alpha was initially released
and became popular. However, I went back to CMU to continue my master degree
for 1yr, which slowed down the progress.

After I came back from school, together with Blake, Yicheng from CoreOS and
later on Ben from Cockroach Labs, we built a solid raft impl as our first
priority. Once we put etcd/raft inside etcd2, the stability of etcd greatly
improved. That is about 1.5 yr after the initial release. Now etcd/raft powers
many production level distributed systems: tikv, cockroachdb, dgraph and many
others.

Over the last couples of years, the focus of etcd/raft is always stability and
nothing else (although people are blaming us for usability :P).

------
philips
etcd has succeeded as a piece of distributed systems infrastructure beyond our
wildest expectation. When Alex Polvi, Xiang Li, and I started the project as a
README in the summer of 2013 we identified that their still was no consensus
database that was developer friendly, easily secured, production ready, and
based on a well understood consensus algorithm. And largely we got lucky with
good market timing, the invention of the Raft algorithm, and the explosion of
good tooling around the Go language. This lead to early success as etcd was
used in locksmith, skydns, and vulcan load balancer.

As the years went on we got lucky again that the Kubernetes project chose etcd
as its primary key-value database. This helped to establish the project as a
must use piece of infrastructure software. Which went on to influence the
technology selection of storage, database, networking, and many other
projects. Just check out all of the stickers of projects relying on etcd that
I could find at KubeCon here in Seattle.
[https://twitter.com/BrandonPhilips/status/107370136987218739...](https://twitter.com/BrandonPhilips/status/1073701369872187393)

For a sense of all of the projects that use checkout this list we maintain in
the project: [https://github.com/etcd-
io/etcd/blob/master/Documentation/in...](https://github.com/etcd-
io/etcd/blob/master/Documentation/integrations.md)

Some notable projects include: Kubernetes, Rook, CoreDNS, Uber M3, Trillian,
Vitess, TiDB, and many many others.

Moving into the CNCF will help to bring a few things to the project:

\- Funding and resources to complete regular third-party security audits and
correctness audits

\- On-call rotation and team for the discovery.etcd.io system

\- Assistance in maintaining a documentation website

\- Resources to fund face to face meetup groups and maintainer meetings

As a closing remark I want to thank the over 450 contributors and the entire
maintainer team for bringing the project to this point. We are solving an
important distributed systems problem with a focused piece of technology.

In fact, in Seattle this week we all got together as a maintainer team for the
first time ever:
[https://twitter.com/sp_zala/status/1073239003330015233](https://twitter.com/sp_zala/status/1073239003330015233)

If you want to learn more about the history of the project checkout this other
post blog post: [https://coreos.com/blog/history-
etcd](https://coreos.com/blog/history-etcd)

~~~
bboreham
Docker Swarm Mode also embeds etcd.

(The embedding mechanism is copy-paste, which I find both ingenious and a bit
distasteful. Maybe I’m just sore I didn’t think of it first)

~~~
abronan
I wrote the initial implementation of the raft subsystem and it was definitely
not a copy/paste. We started from scratch (using etcd's core raft) with the
transport layer being grpc. My initial experiment could be found in this
repository [1]. I then took the code from my initial experiment and included
this into Swarmkit [2]. From there we went through many iterations on the
initial code base and improved the UI with Docker swarm `init`/`join`/`leave`
to make the experience of managing the cluster "friendly".

We spent quite some time evaluating different raft and paxos implementations
(mostly Consul and etcd raft libraries), and found out etcd to be the most
stable and flexible for our use case. It was very easy for example to swap the
transport layer to use grpc. The fact that etcd implementation is represented
as a simple state machine makes it also much easier to reason about under
complex scenarios for debugging purposes, instead of digging into multiple
layers of abstractions.

In retrospect, this came with quite a learning curve. We've had to deal with
issues caused by our own misunderstandings on how to use the library properly.
At the same time the fact that the developers favored stability as opposed to
user friendliness was exactly what we found attractive using etcd's raft.
Additionally, CoreOS developers were super friendly and helpful to help us fix
these issues. We've reported and fixed some bugs as well. Kudos to them for
all the help they provided at the time.

[1] [https://github.com/abronan/proton](https://github.com/abronan/proton) [2]
[https://github.com/docker/swarmkit/commit/89de50f2092dfd2170...](https://github.com/docker/swarmkit/commit/89de50f2092dfd21705f780923cae51bf4e1bcb9)

~~~
bboreham
I apologise for my misunderstanding.

What I remember is, during DockerCon in June 2016, I went into the code to see
how it worked, and I found a top-level file setting up data structures and
handlers that seemed to be 90% the same as the equivalent file in etcd. And
the underlying implementation is reused via vendoring.

Maybe this rings a bell with you and you can tell me what I saw, because I
can't find it now.

Maybe I dreamed the whole thing.

I did, and still do, think integrating etcd into Swarm Mode was a
masterstroke; we had spent the previous two years working to avoid "first you
must install etcd" in a different way that nobody got. Afterwards we created
kubeadm to ape the 'init' and 'join' functionality.

------
polskibus
How does etcd compare to zookeeper? What made etcd the choice for kubernetes
and cncf?

~~~
philips
A few reasons we didn't use ZK at the time (some of these are out of date).
Zookeeper had:

\- No TLS security story

\- An abandoned RPC/serialization system that was hard to use in other
languages

\- A consensus algorithm that differed from systems described in literature

\- A large RAM footprint

Awhile ago some etcd engineers made an experiment in fact to try and run ZK
client protocol on etcd with a proxy:

[https://github.com/etcd-io/zetcd](https://github.com/etcd-io/zetcd)
[https://coreos.com/blog/introducing-
zetcd](https://coreos.com/blog/introducing-zetcd)

Today, etcd performs much better than ZK and I believe it is much more widely
deployed with a wider set of engaged users.

[https://coreos.com/blog/performance-of-
etcd.html](https://coreos.com/blog/performance-of-etcd.html)

~~~
smarterclayton
Pre 3.5.0 zookeeper reconfiguration of a running cluster was also much harder
- that was a significant discussion point on Kubernetes when we had the etcd
vs (anything) discussions early after open sourcing.

I still think etcd total ordering over history also made reasoning about
changes in the system while we were writing the first versions of the
controllers and caches and list-watch loops. ZK had partial order, and I was
leery of that at the time.

------
barbecue_sauce
I was wondering when this was going to happen. It's such a critical piece of
Kubernetes.

------
spullara
This is exactly the kind of thing that should be a layer on top of
FoundationDB. Too many state stores all doing largely the same thing.

~~~
pram
FoundationDB has only been open source for 8 months?? etcd is like 5 years
old.

~~~
reilly3000
FoundationDB started in 2009 and released their 1.0 in 2013. Source:
[https://en.wikipedia.org/wiki/FoundationDB](https://en.wikipedia.org/wiki/FoundationDB)

~~~
unfunco
From the same source:

> On April 19, 2018, Apple open sourced the software, releasing it under the
> Apache 2.0 license.

------
RcouF1uZ4gsC
I wonder if this the core of Redhat knowing that the clock is ticking, and
make sure that critical software that they worked on is available in an open
fashion. One only has to look at Sun and MySQL to see what can happen to a
once vibrant open source offering after acquisition.

~~~
moosingin3space
Red Hat does not own anything that's valuable aside from their developers, who
chose to work at Red Hat due to their pro-FOSS positioning. If IBM chose to
start shutting projects like Fedora down or move in the direction of closed-
source, these developers would have no desire to remain, and would leave,
making that $40B acquisition worthless.

~~~
rbetts
Red Hat has extensive customer relationships, customer databases, contracts
for future revenue, partnerships, operational processes, . . . product IP is
just a slice of the pie.

------
ross2990
Glad to see that Red Hat is still committed to the Open Source movement after
so many naysayers predicted that all contributions going forward would be
stymied due to the announcement of the acquisition.

~~~
jakevn
Not saying it will happen (just that it usually does), but I almost never see
acquired companies immediately turn into the parent. Usually the acquired
maintains its course until attrition and cross-pollination replace its
original culture with that of the parent. The acquired company eventually
exists only as a collection of intellectual property and history. Could take
many years depending on how tightly IBM squeezes. That and the acquisition
isn't even finalized yet.

~~~
wlesieutre
I wonder if "We should get this taken care of before IBM takes over" was part
of the thought process here.

------
fulafel
How is etcd security these days? Authentication still off by default?

