
GitHub Load balancer: HAProxy zero-downtime, zero-delay reloads with multibinder - logicalstack
http://githubengineering.com/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/
======
jolynch
Funny enough we're doing something pretty similar for our external lb tier but
instead of a custom proxy we're just using nginx proxying back to HAProxy.
Your solution is really nifty though because there aren't two accepts going
on.

In the thing that we're trying nginx terminates TCP (and SSL if needed),
proxies back to a unix socket where HAProxy is listening. This gives us the
best of both worlds: we can reload HAProxy all day long and socket listeners
are hitless, and we rarely need to reload the nginx config which comes out of
the box with hitless TCP sockets. But it's not quite as cool as this solution
because HAProxy and nginx both have to accept.

We haven't decided that we're going to do our internal load balancing this way
yet because the listening ports change so often, but if we do I'll probably be
working on
[https://github.com/airbnb/synapse/pull/203](https://github.com/airbnb/synapse/pull/203)
to do it. The PR makes it so that Synapse can manage both nginx and HAProxy
simultaneously (to deal with the extra system complexity of an additional
proxy). Or we'll try this cool thing out :-P

~~~
bogomipz
Can you remind me again why HA Proxy isn't hitless? Does it have to do with
nproc?

~~~
jolynch
When HAProxy reloads, it has to re-bind the sockets that it was listening on.
Due to a bug in how Linux implements listen port sharing (SO_REUSEPORT), new
incoming connections can get dropped for a very brief (~1ms) period while
HAProxy reloads. The Github article and the linked Yelp article both go into
detail on this.

The tldr is that in Linux right now there is no way to gracefully drain
connections from a listening socket. Many programs work around this by passing
the socket file descriptor from the old process to the new process, which is,
for example, how nginx works.

~~~
bogomipz
Right, the 3 way handshake can go sideways during that interim. Thanks.

