
Etcd 2.2 - sajal83
https://coreos.com/blog/etcd-2.2/
======
eis
Does anyone at this point have a good comparison between etcd and Consul?

The v3 API of ectd seems more powerful because it supports multi-key
transactions but isn't there yet.

Other than that I always had a slight feeling that Consul might be a bit more
stable/reliable but I have no hard facts.

~~~
philips
On the stability/reliability of etcd the team is very confident in how it
behaves. Here is how we build that confidence:

\- We have a functional testing cluster that is constantly simulating various
types of failures. You can find some information on that here:
[https://coreos.com/blog/new-functional-testing-in-
etcd/](https://coreos.com/blog/new-functional-testing-in-etcd/)

\- The team is works to ensure the raft library that we use (and that we now
share with cockroachdb) has unit tests that handle all of the edge cases
described in the raft paper. See this paper as an example:
[https://github.com/coreos/etcd/blob/master/raft/raft_paper_t...](https://github.com/coreos/etcd/blob/master/raft/raft_paper_test.go#L49)

\- We have been running etcd in production for 1.5 years for a bootstrapping
service called discovery.etcd.io and have focused on how to handle higher and
higher read/write loads there. For example see the improvements we made in
etcd 2.1 with async snapshots:
[https://twitter.com/BrandonPhilips/status/639843455141740544](https://twitter.com/BrandonPhilips/status/639843455141740544)

\- We have also focused on clear guides on how to operate etcd. All of this
testing and performance work is of no use if people can't operate the system
correctly:
[https://coreos.com/etcd/docs/latest/](https://coreos.com/etcd/docs/latest/)

Of course if you have particular issues that you have encountered we would
love to help:
[https://github.com/coreos/etcd#contact](https://github.com/coreos/etcd#contact)

~~~
SEJeff
Fantastic reply, thanks for this Brandon. I know that Kyle Kingsbury aka
"aphyr, destroyer of distributed systems" broke etcd and at the time (before
you wrote a new raft library), consul faired quite a bit better than consul.

Are you using his jepesen testing tool as part of your testing for failure
scenarios? If not, why?

~~~
philips
IIRC, etcd and consul failed in the same way in aphyr's post. Essentially
neither had quorum read.

We don't have jepesen setup as a testing tool because it is really hard to
make it a reliable false positive free system. Plus, the languages it is
written in make it hard for us to hack on it.

Instead what we have done is worked hard to build a functional testing suite
to find non-algorithm issues (weird exhaustion issues, behavior under real
disks, etc). And then have deep, fast, deterministic tests of the core raft
algorithms.

~~~
SEJeff
Oh the stuff you've (you as in CoreOS) done with raft golang libraries (to
test and fix it) is nothing short of incredible. The future with k8s with
CoreOS as the base looks really great. Please do keep it up :)

------
JulianMorrison
I really like the idea of multi-key transactions because it will allow
installation of configurations as a whole, rather than a series of piecemeal
changes, or some sort of "write a new one piecemeal and then update the
current version" manual hack.

------
dsies
Congratulations on the new release! I actively use etcd in multiple projects,
in production and am overall very happy with it. However, I've ran into some
issues with the go-etcd package, namely with watches and bad behavior when a
node goes away (things like a channel being spammed with updates even though
no updates are taking place).

Is this something that the new go-etcd package addresses?

~~~
unihorn
We publish
[https://github.com/coreos/etcd/tree/master/client](https://github.com/coreos/etcd/tree/master/client)
as our new client, and it is cleaner and better than go-etcd. You could try
this out.

Back to the problem, I didn't notice this behavior in go-etcd before. We are
more than welcome to solve it if it is a problem in new client.

------
schmichael
So excited for the v3 API. The v2 HTTP API with its traditional filesystem
like structure makes getting started easy, but quickly becomes limiting.
Recursive operations on directories allows for some multi-key transaction like
behavior, but in an extremely limited sense.

Having native multi-key transactions in v3 will make a lot of use cases
easier.

------
andrewflnr
Can someone explain exactly what problem etcd solves and specifically how I
would use it? It seems interesting, but I can't figure out how "distributed
key value store" translates directly into "service discovery", whatever that
is. What exactly are the keys and values that make this work?

~~~
divideby0
So one of the challenges in a containerized environment is that services start
up on random ports. If I'm running a Postgres container with the default
Docker networking mode, for example, the internal port of 5432 may be bound to
the host port of 12345. This allows me to spin up multiple instances of
Postgres on the same machine for greater service density.

However, in a distributed environment, services can spin up on different
machines. The instance of Postgres my application needs could be on
server1:12345 or server2:23456. But in a distributed system, you need a cross-
cutting service that's available to all servers so that if my app is running
on server1, it can find the right Postgres instance running on server2.

I'm not an expert on etcd, but my understanding is that the most common use
case is to run etcd on each host machine. When services start up, their
supervisor registers the service's hostname, port, etc with etcd's key-value
store. This registration is then propagated to other etcd nodes in a
consistent manner, using a consensus algorithm called Raft:

[http://thesecretlivesofdata.com/raft/](http://thesecretlivesofdata.com/raft/)

Consensus actually turns out to be one of the harder problems in a distributed
system design. If I have a network partition that prevents etcd instances from
seeing each other, you don't want one instance reporting incorrect or stale
data. Otherwise, my application could be writing to the wrong service, causing
data loss.

Etcd does consensus extremely well, and in a way that scales to support
hundreds of nodes. It's one of the two distributed systems I'm aware of that
have (mostly) passed Jepsen testing:

[https://aphyr.com/posts/316-call-me-maybe-etcd-and-
consul](https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul)

There are also alternatives like Consul and Zookeeper, but in the case of
Zookeeper you have to do a lot of the heavy lifting yourself to support
service discovery. There are also some well-documented caveats:

[https://tech.knewton.com/blog/2014/12/eureka-shouldnt-use-
zo...](https://tech.knewton.com/blog/2014/12/eureka-shouldnt-use-zookeeper-
service-discovery/)

Consul also has a pretty fair writeup (IMHO) on the tradeoffs of each solution
on their website:

[https://consul.io/intro/vs/zookeeper.html](https://consul.io/intro/vs/zookeeper.html)

