

Zero Downtime Frontend Deploys with Vulcand on CoreOS - robszumski
http://coreos.com/blog/zero-downtime-frontend-deploys-vulcand/

======
moe
The architecture is sound. But half-assing a loadbalancer impl (vulcand)
instead of embedding HAProxy is a terrible idea.

~~~
jksmith
So it's half assed if it's not HAProxy? How so?

~~~
moe
HAProxy is 14 years old, under active development by very competent people,
used in thousands of production deployments and many hardware appliances.

Pretty much every large site you visit every day has HAProxy in the stack.
It's one of the cornerstones of OSS infrastructure software.

Are you seriously asking how a 2 month old hack (vulcand) is half-assed in
comparison?

~~~
jksmith
If you're going to call something half-assed because it's not HAProxy then I
for one would consider it more productive if this was framed in terms of
metrics instead of something like "half-assed." Does HAProxy have a suite of
metrics that can serve as benchmarks for all the projects out there that would
dare to be less half-assed?

~~~
felixgallo
Why didn't you ask the question the other way around -- does vulcand have
anything to show that it is better in any way than the current best-practice
load balancer?

Reading the docs, the only thing that stands out is that it's tightly coupled
with etcd, which seems like a really bad idea architecturally.

~~~
alexk
I'm wondering why do you think that tightly coupling with Etcd is a bad idea?

Etcd provides advanced features for configuration and service discovery -
serialized updates, high availability and ttls for keys that's why it was a
reasonable choice for me to use it and not file system that does not provide
any of the above features out of the box.

One of the main reasons I wrote Vulcand is for experimentation - what would be
the next steps for proxy and load balancing out there?

One of the ideas - having a proxy that will embrace the dynamic nature of the
Docker enabled backends an provide full support for rapidly changing
dockerized landscape.

Another idea is to try out some ideas in error recovery/failure detection
algos, like preventing cascading failures:

[http://mailgun.github.io/vulcan-docs/#weighted-round-
robin](http://mailgun.github.io/vulcan-docs/#weighted-round-robin)

I don't pretend that Vulcand is somewhat robust and everyone should
immediately drop their HAProxy/Nginx solutions and start using Vulcand. I
hoped that would be clear from the README and is highlighted in Rob's post as
well.

In fact, I'm using Vulcand as well as Nginx and HAproxy, I'm just choosing
carefully when should I use one load balancer vs another.

~~~
felixgallo
First, let me say up front that I value your verve and appreciate your desire
to experiment and learn and push frontiers. Bravo on all that.

The problem with tight coupling -- especially across concerns and software
layers, as with 'etcd -> load balancer' \-- is that it's an immediate
withdrawal from the great Bank of Technical Debt.

Some time in the future, one of the following things will happen:

1\. Etcd will want to perform one unit of positive technical change, and
either that behavior will break your code, or you will be forced to fork
versions, or etcd will be forced not to make that change by the popularity of
your code.

2\. You will want to perform one unit of positive technical change, but etcd
will, for whatever political, architectural, or technical reason not wish to
support this change. You will either abandon your idea for lack of upstream
technical support or be forced to fork etcd.

The neglect of these incredibly common occurrences is itself now incredibly
common. It underpins, e.g., systemd development, which in turn appears to be
informing coreos and docker development, and since those are the new-shiny-
toys, a lot of people seem to think that's just the way we should all think
now, that the 'unix philosophy' is ripe for disruption(tm).

Problem is, invariably, coupling turns out to be the most important artifact
in all of computer science. It's worse than naming things, it's worse than
cache invalidation, it's worse than off-by-one errors. And the worst kind of
coupling is the kind of coupling that crosses stack boundaries for no
obviously good reason, because the amount and kind of reasoning that has to
happen at an operational level when that stops working in production is both
severe and very difficult to resolve.

So again: applaud the desire to learn, and experiment. The coupling: please,
please don't. The marketing of a test solution as something to consider using:
you don't need that to build your portfolio, just put the project up and say
it's a test thing. And preventing cascading failures with weighted round
robin, etc: [http://cbonte.github.io/haproxy-
dconv/configuration-1.5.html...](http://cbonte.github.io/haproxy-
dconv/configuration-1.5.html#4.2-balance%20url_param)

------
mahmoudimus
We have something very similar at Balanced called Rump (routing umpire). It's
written in python has a pretty cool rule based engine and a swappable backend.

It currently uses ZK and Chef, but it would be pretty interesting to use
something like Etcd/Consul since no one wants to maintain a ZK cluster.

If there are individuals who want to hack on it, shoot me a msg on twitter and
I'll see what I can do to speed up the process to open source it.

------
bkirkbri
What great timing! I was just about to roll my own using etcd, haproxy and a
bunch of perl duct tape.

Very excited to take a good look at vulcand.

~~~
robszumski
If you're still interested in doing this for any reason, check out confd [0]
for populating templates based on etcd values.

[0]
[https://github.com/kelseyhightower/confd](https://github.com/kelseyhightower/confd)

~~~
bkirkbri
Yes, confd was on my radar and looks promising. In my case, we've already got
a wad of perl duct-tape out there and adding etcd was a baby step. Using confd
would have been a later step.

Vulcand will likely allow me to replace the whole shebang in one step.

~~~
bketelsen
it doesn't look like vulcan supports HTTPS, so it's a no-go for anything
serious. Here's a post on my confd/nginx/docker setup for zero-downtime using
etcd. [http://brianketelsen.com/2014/02/25/using-nginx-confd-and-
do...](http://brianketelsen.com/2014/02/25/using-nginx-confd-and-docker-for-
zero-downtime-web-updates/)

~~~
bkirkbri
I actually learned about confd via your post. Thanks for that.

In our case, HTTPS isn't an issue but there are good arguments for using a
tried-and-true proxy. I see Vulcan as a quick way to test out docker
containers for our architecture.

In the end I'll probably use confd to rewrite a VCL file for Varnish. It's
just a matter of deciding how many containers and what level of
sidekicking/ambassadoring I'm willing to put up with to "properly" organize
things...

------
gabrtv
Top notch technical writeup. We've had our eyes on Vulcand for some time now,
though we'd really like to see native TCP support.

~~~
alexk
This is in pipeline!

TCP would be next step once I make sure the existing feature set is production
ready and robust.

------
jbrechtel
I'll ask about the reasoning instead of attacking the decision (since that
didn't get a response).

Why not embed HAProxy?

------
bjt
etcd, fleet, deis, and now this are getting me more and more tempted to move
from Velociraptor to the Docker ecosystem.

If Deis could run my uptests[0] before Vulcan gets called to re-route things,
that'd be awesome.

[https://bitbucket.org/yougov/velociraptor/src/2bcef54253a9e9...](https://bitbucket.org/yougov/velociraptor/src/2bcef54253a9e9238aee92de92b6d088a5d33c60/docs/uptests.rst)

~~~
gabrtv
Come find us in #deis on Freenode and we'll see if we can help.

------
jksmith
This is a golang project, no?

~~~
alexk
yep

