
Infrastructure discovery with etcd - iwalz
http://ducky.cloud/blog/infrastructure_discovery_with_etcd
======
bogomipz
The OP states under "Why not Consul"

"Consul agent takes part in Raft, so doing infrastructure discovery in your
whole environment means every system takes part in Raft"

Consul agent runs in two modes - master and client. From the Consul Docs:

"Client - A client is an agent that forwards all RPCs to a server. The client
is relatively stateless. The only background activity a client performs is
taking part in the LAN gossip pool. This has a minimal resource overhead and
consumes only a small amount of network bandwidth."

The client doesn't take part in the consensus protocol. This seems to be the
OPs whole reason for choosing etcd over Consul and I don't believe the OPs
claim is correct.

~~~
iwalz
I personally think the OPs reason to use etcd is mainly about 2 parts: 1.) The
ecosystem it integrates in 2.) Only maintaining 1 SR technology

I think that with the upcoming kubernetes trend, many OPs guys are somehow
familiar with the CoreOS stack where etcd fits in very great. The hashicorp
toolchain is great, but I've seen many OPs guys only using terraform or vault
e.g. and don't build an integrated environment with nomad or atlas.

The CoreOS stacks I've seen are more integrated in platforms than the
hashicorp ones (can be a good and a bad thing at once).

~~~
bogomipz
>"The hashicorp toolchain is great, but I've seen many OPs guys only using
terraform or vault e.g. and don't build an integrated environment with nomad
or atlas"

I think this is one of the nice parts about Hashicorp releases though is that
they are building blocks and you can choose to use them as standalone pieces -
Consul, subsets Consul + Vault or of them in an integrated stack such Nomad
and Atlas.

I agree with you though if you are a CoreOS shop it certainly makes sense to
choose etcd.

------
zbjornson
(Aside from the issues with reason #1 for not using consul noted below...)

The author says that Atlas is essentially "required" for bootstrapping. We
simply use our cloud provider's "list-instances" command (with a filter) so
that we don't rely on any third-party and never contact the internet. This is
important to us for security and stability, and has kept our cluster running
with over 1k members. We don't want to go down when Atlas or etcd's discovery
service goes down.

~~~
iwalz
I haven't meant that you need to use atlas, but I have never seen a nice
automation implementation for a scalable consul cluster that doesn't use
atlas. I've also built something around atlas, mostly using hiera in puppet -
but since I used atlas and the etcd discovery, everything else looks like a
workaround.

What provisioning do you use to integrate with your cloud providers api?

~~~
takeda
We use consul in our infrastructure, and by default none of our internal hosts
have internet access, so we can't use Atlas even if we wanted it. We deploy
consul through salt, but information about our consul servers is extracted by
our internal CMDB which predates consul.

I'm curious how etcd is bootstrapping? I don't see a better way to do it than
using multicasting.

BTW: I absolutely hate that consul (and looks like etcd has this issue as
well) is using http for communicating. It's so inefficient to obtain updates
about changes these way. Zookeeper which everyone loves to hate (and it wasn't
even created for service discovery), did this so much better, you have a
single long standing connection where you subscribe what updates you want to
receive, it has much lower overhead and is simpler to code with.

Another thing that seems to be lost buy people who promote their service
discovery solution is that you don't need to be always consistent and
eventually consistent is perfectly fine. You don't really need raft or paxos
to do it.

~~~
iwalz
Since v3 they're using grpc, which means http/2 - since than, long polling is
much more efficient!

~~~
takeda
It still feels like a hack instead doing it properly. You're doing long pool
instead what you should do in first place, which is push.

Again, I did not work with etcd yet, but in consul because it's "RESTful" i
when you monitor multiple services you need to maintain multiple requests.

Edit: Reading more about http 2's frames and pipelining, looks like it's
possible to use it similar way it's done in ZK[1]. If GRPC allows that then I
suppose it indeed solves this problem.

[1] having a single long standing connection that's not closed after receiving
a response. The request frames could be used to place watches and response
frames would send the updates to the client.

~~~
ideal0227
etcd use bi-directional streams for watchers. One. TCP connections can
maintains multiple streams. No matter what you need to keep at least one
connection. ZooKeeper is not an exception.

~~~
takeda
That's good to hear. Seems like that would work the same way then.

------
jolynch
Nice, now that v3 includes some of these crucial features I might even be able
to merge the etcd provider for Synapse!

[https://github.com/airbnb/synapse/pull/58](https://github.com/airbnb/synapse/pull/58)

------
RRRA
I'm no consul expert, but their only problem i've seen right now is the fact
that ID is bound to an IP and that this IP is taken from the first network
interface.

I hope they can move to GUIDs or some other solution soon and facilitate
consul's use in "dynamic" (at leat from consul's point of view) IP
environment, say inside Docker, etc.

~~~
Leon
You can specify a custom ID. I use ID's such as AWS instance ID for clusters
in Amazon.

~~~
RRRA
So a single node Consul cluster, when rejoining itself with a custom ID, would
end up a leader of "itself" and not wait for the previous IP as a follower,
forever? :)

Which option would that be for a server config?

[https://www.consul.io/docs/agent/options.html](https://www.consul.io/docs/agent/options.html)

~~~
chucky_z
This was actually an issue in previous versions, in (I believe 0.7) after a
timeout (reconnect_timeout and reconnect_timeout_wan in options) a node will
be completely removed.

~~~
takeda
0.7.0 is the latest version of consul

~~~
RRRA
And it still is an issue

------
slackpad
For the zombie problem, Consul has always deregistered dead nodes (and their
associated services) automatically after 72 hours. Consul 0.7 added the
ability to define a health check attribute called
"deregister_critical_service_after" which will deregister a service
automatically if a check is failed for longer than this value. This helps
clean up services from processes that register themselves that might not get
shut down cleanly for whatever reason.

Details are here -
[https://www.consul.io/docs/agent/checks.html](https://www.consul.io/docs/agent/checks.html).

------
iwalz
I have updated the post to reflect #1 - thanks for pointing that out!

