

Consul.io – Service discovery and configuration made easy - rogerfernandezg
https://consul.io/

======
krat0sprakhar
For people interested in Distributed Systems the architecture[0] page is a
treasure trove. The explanation of the gossip(SWIM) and consensus
protocol(Raft) used is easy to understand and quite lucid.

It would be wonderful if all (most?) open-source projects have a such
beautiful documentation explaining their architecture. Thank you Armon and
team for sharing this with us!

[0] -
[https://consul.io/docs/internals/architecture.html](https://consul.io/docs/internals/architecture.html)

~~~
nhjk
Any suggestions on where to get an in depth intro to distributed systems in
general?

~~~
cristobal23
I enjoyed Berkeley's CS162 because its from the perspective of an operating
system. [http://bit.ly/11rTdEw](http://bit.ly/11rTdEw)

------
bascule
Consul looks awesome. The only thing I don't like about it is they chose to
use a homebrew encryption scheme for Serf's gossip protocol:

[http://www.serfdom.io/docs/internals/security.html](http://www.serfdom.io/docs/internals/security.html)

(If you're confused about how this relates to Consul, see this:
[http://www.consul.io/docs/internals/security.html](http://www.consul.io/docs/internals/security.html))

Worse, their justification for using this instead of standard schemes like
(D)TLS seems to be that they don't need the sorts of features transport
encryption normally needs because they lean on the protocol's state machine to
provide some of the features transport encryption would normally provide for
free, like replay attack avoidance.

Not only does homebrewing encryption almost always carry with it design and
implementation mistakes, but someone auditing the protocol not only needs to
understand the cryptographic design, but how it interacts with the rest of the
protocol's state machine.

My advice would be to scrap their homebrew encryption and use (D)TLS

~~~
armon
We did take a look at D(TLS) when implementing Serf and the associated
encryption mechanism. D(TLS) is much better suited when you are doing point-
to-point duplex communication. The gossip algorithm instead is doing peer-to-
peer (N-to-N instead of 1-to-1) in a half-duplex model. Using DTLS would've
been too heavy for our use case.

We did develop the algorithm in the open (see this gist:
[https://gist.github.com/armon/7159161](https://gist.github.com/armon/7159161)),
and made a call on the community to provide feedback to help improve the
design of the system.

The crypto system used is actually pretty vanilla based on AES-GCM. We
certainly didn't attempt to invent our crypto primitives, and instead stuck to
the best practices around the most modern systems.

~~~
jimktrains2
> The gossip algorithm instead is doing peer-to-peer (N-to-N instead of
> 1-to-1) in a half-duplex model.

Unless you mean you're doing multicast, you're still doing 1-to-1 connections.

> We did develop the algorithm in the open and made a call on the community to
> provide feedback to help improve the design of the system.

I wouldn't trust myself, let alone many unknown people who are just
volunteering their time and have no known credentials as a cryptographer.

> instead stuck to the best practices around the most modern systems.

Except for the whole "don't role your own crypto system" one.

There are just so many things that can go wrong and when crypto fails it can
do so very quietly. Additionally, if you ever want anyone to interoperate with
you it'll just be a PITA for them, and depending on who it is, a PITA for you.

The overhead of a proper protocol like (D)TLS isn't that much.

> "On our production frontend machines, SSL/TLS accounts for less than 1% of
> the CPU load, less than 10 KB of memory per connection and less than 2% of
> network overhead. Many people believe that SSL/TLS takes a lot of CPU time
> and we hope the preceding numbers will help to dispel that." \- Adam
> Langley, Google

~~~
armon
You are right, the communication is still unicast in nature. I should clarify
to say that there isn't a persistent 1-to-1 communication, the nodes we gossip
with are randomly selected on each interval. There is no connection or session
establishment between peers.

I guess it depends on your definition of roll your own. We didn't invent AES-
GCM or implement it. We are using the implementation shipped with the Golang
stdlib.

~~~
bascule
Nobody is questioning whether AES-GCM is a good algorithm or not, however you
are using AES-GCM as part of a hand-rolled transport encryption protocol, and
this is what's worrisome. Designing a transport encryption protocol is a
difficult endeavor, and it seems you have skipped most of the steps (e.g.
replay attack prevention) but suggest that it's irrelevant because other parts
of the protocol provide security (e.g. the SWIM state machine).

This makes your protocol difficult to audit: someone concerned about potential
attacks can't just look at your protocol in isolation, but has to factor the
underlying protocol state machine into the security of your transport
encryption protocol.

------
tigeba
A) Hashicorp, you are awesome.

B) This works really well in conjunction with registrator
[https://github.com/progrium/registrator](https://github.com/progrium/registrator)
which listens to Docker events and will automatically register your services
with Consul.

I occasionally run into an issue where a node will be briefly marked as
failing. This seems to clear up pretty quickly but I can't really figure out
what is happening when this occurs. The nodes are all in the same DC and on
their own private network. I also occasionally lose all running Docker
containers on a particular node. I don't think that is related to Consul, just
throwing that out there :)

------
jakozaur
Wonder how this compare to Smart Stack: [http://nerds.airbnb.com/smartstack-
service-discovery-cloud/](http://nerds.airbnb.com/smartstack-service-
discovery-cloud/)

E.g. you got webservices talking over HTTP. One goes bad, you want existing
clients not to talk to them.

Smart Stack: you are using HTTP proxy on each machine. You're code doesn't has
to be aware of who is talking to.

Consul: Not sure, but it sounds like server code has to be able to handle
events to change that.

~~~
armon
We compare Consul to SmartStack here:
[https://www.consul.io/intro/vs/smartstack.html](https://www.consul.io/intro/vs/smartstack.html)

With a tool like consul-template, you can have the same behavior that Synapse
gives you. [https://hashicorp.com/blog/introducing-consul-
template.html](https://hashicorp.com/blog/introducing-consul-template.html)

In the general case, clients of Consul don't need special handling code for
events, it's handled by Consul for you.

------
raspasov
Love the nice informative website.

However:

[1]
[https://www.consul.io/intro/vs/zookeeper.html](https://www.consul.io/intro/vs/zookeeper.html)

This did not unequivocally convince me in the benefits over ZooKeeper. On the
contrary, it makes it seems that ZooKeeper tries to do less, and I strongly
prefer simpler tools.

[2] [http://aphyr.com/posts/316-call-me-maybe-etcd-and-
consul](http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul)

Have the issues discussed in this article been fully addressed yet?

Overall, can you outline a use case where Consul is definitely better than
ZooKeeper?

~~~
armon
The issues discussed by aphyr have been fully fixed as of Consul 0.2, several
months ago. In the initial release of Consul, only the "default" mode was
available and has the now-documented caveat of a stale read possibility. The
various consistency modes are now discussed here:
[https://www.consul.io/docs/internals/consensus.html#default](https://www.consul.io/docs/internals/consensus.html#default)

With respect to ZooKeeper, there are different approaches. ZK provides a low-
level primitive on which you can build. Consul provides similar primitives,
but it ships with many features out of the box that don't require any
development effort. It's a "batteries included" approach.

Specific examples:

\- Real-time configuration with Consul + consul-template

\- DNS based service discovery

\- Scalable Nagios replacement

\- Dynamic HAProxy / Varnish configuration

\- Application configuration with Consul + envconsul

\- Triggering config management tools with the event system

That is just a handful of uses for Consul that don't require writing any code.
Doing similar things with ZK is possible, just requires a lot more work.

~~~
raspasov
Thank you for detailed answer!

------
bvanvugt
Is anybody using this in large production systems?

~~~
mitchellh
(Note: I'm one of the folks who works at HashiCorp)

We don't have permission at this time to share some of our big users (we're
working on approvals!), but I can say that that some very big companies you've
certainly heard of and probably use right now have deployed Consul across
every server. And Consuls deployment in just companies were working with is in
the hundreds of thousands of machines.

At this point were very confident in its stability and it's well proven at
very large scale.

If you're a big user and need references to other large users, email me and
I'd be happy to set that up.

~~~
ch
When you say hundreds of thousands of machines, is each one running a consul
agent, or do some just make use of answers provided via DNS or the remote api?

Without divilging too many details, what would be the ratio of agents running
'-server' to overall agents?

~~~
sandstrom
I'm no expert on consul, but I think it's architected such that you'll run a
few servers (say 3 or 5) and the rest will be agents.

All machines using consul will (generally) run the agent locally.

~~~
ch
You have the general topology right, but not all nodes necessarily have to be
agents. You could, forinstance, run an agent next to a bind server and
delegate the .consul. Zone to it. That way for nodes which only use DNS for
discovery can work I modified.

In general you are going to set up an N+M topology of n servers and n+m agents
-- but for a network of 100k nodes does consul scale to 100k agents? Plus what
goes unstated is how those 100k nodes are laid out in DCs and racks.

Anyway, this isn't to slag on consul. I think consul is like chocolate and
peanut butter.

I use it in my mesos cluster and just am genuinely curious how large it can
scale!

------
michaelmior
If @armon is still listening, curious if there's been any talk of maintaining
a log of the state of Consul. Along with consul-template, Consul provides a
great way to maintain application configuration. However, there's still
currently a need from external change tracking mechanism to facilitate
rollback.

------
somethingnew
Has this never been posted before?

~~~
praeivis
It was:
[https://news.ycombinator.com/item?id=7815993](https://news.ycombinator.com/item?id=7815993)

but half year ago and no community reaction.

~~~
FooBarWidget
No community reaction doesn't mean anything. HN is so competitive these days,
any submission HAS to get 5 upvotes within 10 minutes in order to reach the
front page, and only when it reaches the front page will enough people upvote
it. It's a catch-22.

------
ckdarby
I love Hashicorp :)

------
ericb
Does this do leader election?

~~~
wernerb
Yes, in quite the same way that etcd and ZooKeeper does it. Info here:
[http://www.consul.io/docs/guides/leader-
election.html](http://www.consul.io/docs/guides/leader-election.html)

I was looking it over and found how leader election is handled in general
fascinating. Leader election can be tricky if you have a lot of machines. This
example sets a watch on a single 'key'. But this would mean that all followers
would be triggered upon lock-release, meaning all your machines will pound
consul for a new lock request / leader election fight. To mitigate this, it
seems they implemented 'lock-delay'.

In ZooKeeper what you typically do is create a linked-list of watches so that
only the follower directly behind the leader gets triggered and so avoid
'disturbing the herd'.

In any case, consul seems great, and will probably use it!

------
incunix
well this is pretty cool isn't it!

