
Truly Seamless Reloads with HAProxy - kawsper
https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/
======
lobster_johnson
While this is fascinating, and Willy is brilliant as always, I always wondered
why HAProxy couldn't just, you know, reload the config.

Surely you don't need to fork: Just parse the new config, create the necessary
internal data structures, and let traffic flow into the new ruleset while
keeping all the sockets (except for those that are superfluous, and of course
let in-flight requests finish). Is it because HAProxy's internals weren't
designed to do that and that it would too big of a rewrite?

I always found Varnish's design very cool: It compiles the configuration
(which is a DSL called VCL) to C and loads it as a dynamically loaded library.
I don't know how it does hot reloads, but I believe it does do them
seamlessly.

~~~
JosephRedfern
The post kinda touches on this, and makes it clear that config changes isn't
the only situation that this would come in handy -- software updates are a big
reason too.

"Service upgrades are even more important because, while some products are
designed from the ground to support config updates without having to be
reloaded, they cannot be upgraded at all without a stop/start sequence.
Consequently, admins leave bogus versions of those software components running
in production because it is never the right time to do it."

~~~
annnnd
And if you have restarts without downtime, there is no need for configuration
reloads anymore. Why solve the same problem in 2 ways? It would just increase
a chance for bugs.

All in all, HAproxy is a brilliant piece of software and Willy is running it
in an exemplary way. Kudos!

------
ericd
I love HAProxy so much. In our architecture, it started with simple frontend
load balancing, but it ended up mediating almost every inter-server
communication, which gave us a great amount of flexibility in swapping
machines in and out, by giving each service load balanced virtual ip
addresses. Thanks for your great work, Willy.

------
DorothySim
I've used a simple iptables approach to redirect traffic to new Docker
container:

    
    
      iptables --wait --table nat --append PREROUTING --protocol tcp --dport 80 ! --in-interface docker0 --jump DNAT --to $new_target
    

Then removing tables for old one:

    
    
      iptables --wait --table nat --delete PREROUTING --protocol tcp --dport 80 ! --in-interface docker0 --jump DNAT --to $old_target
    

(repeat the same for ip6tables).

The same had to be repeated on system start but otherwise it worked flawlessly
and had zero-downtime.

------
fidz
I wonder, how does nginx and haproxy handle long/persistent connection
session? The connection itself can't be terminated since there is actual
client, established connection with a backend beneath. Will the reload be
failed? (Something like, "connection termination timeout; can't reload; try
later"). For web workers probably we won't see this; for most of the time, the
connection is terminated after request done.

~~~
jolynch
Both NGINX and HAProxy will hang around for as long as the connection is open
(up to the timeout). It's actually quite an issue when you're rapidly
reloading either proxy (you can run out of memory reasonably easily), but most
services that have long lived TCP connections also handle resets reasonably
well so you can typically just kill the old proxies and it'll be ok.

~~~
r4um
There is an option added in 1.7 which makes old process exit after a grace
period [https://cbonte.github.io/haproxy-
dconv/1.7/configuration.htm...](https://cbonte.github.io/haproxy-
dconv/1.7/configuration.html#hard-stop-after)

------
frik
Why has HaProxy still such a 20 year old website?

[http://www.haproxy.org](http://www.haproxy.org)

And then there is a blog on a new
[http://www.haproxy.com](http://www.haproxy.com) site.

Is there a comparision of HaProxy, Varnish, Squid, Traffic Server (ATS),
Nginx, lighttpd, etc for typical scenario.

~~~
ericd
Because it's an absolutely awesome tool, and it doesn't need a flashy site
with marketing graphics.

------
orthecreedence
This is great. Thanks, Willy!

I've used HAProxy on and off throughout a lot of my career. I'm currently
using it at my company as a way for services to talk to each other without
specifically knowing who is where. I wouldn't call it "microservices" but
probably similar: each server has HAProxy on it, and Ansible creates the
HAProxy config/hosts file so that, say, a worker server can grab [http://lb-
api:6666/some/resource](http://lb-api:6666/some/resource). lb-api is a host
that routes to 127.0.0.1 and HAProxy runs on port 6666 locally, parses the
"lb-api" host, and routes the request to one of the servers in the "api"
group. Any time we change any servers, we just run our haproxy playbook and
everything just flows.

As always, HAProxy is one of the few pieces of our infrastructure that "just
works" day after day.

~~~
toomuchtodo
Airbnb released a tool called Synapse that'll do most of this service
discovery and config rendering for you.

~~~
jolynch
Disclaimer: I help maintain Synapse.

I highly recommend this tool. Yelp has used it in production for years to
manage a fairly large PaaS (hundreds of services, thousands of containers,
constant churn); it's proven quite flexible and resilient.

Synapse is available on github [0], and we've open sourced our automation used
to create a highly available service router using Synapse as well [1][2].

[0] [https://github.com/airbnb/synapse](https://github.com/airbnb/synapse) [1]
[https://github.com/Yelp/synapse-
tools/tree/master/src/synaps...](https://github.com/Yelp/synapse-
tools/tree/master/src/synapse_tools) [2]
[http://paasta.readthedocs.io/en/latest/yelpsoa_configs.html#...](http://paasta.readthedocs.io/en/latest/yelpsoa_configs.html#smartstack-
yaml)

~~~
toomuchtodo
Really appreciate not only the work put into Synapse, but also the release and
maintenance of it. Its been very helpful for projects where I've used it.

------
was_boring
This is great. We use haproxy at my work and I like it, it does it's job, but
quirks like dns resolution only at startup, having to reload on config changes
and no seamless reloading stop me from loving it.

~~~
blibble
title of article says it now supports seamless reloading?

what's the alternative to reloading on config change? automatic detection and
reload on file change? personally I prefer the explicit action

~~~
Chronos
It still requires explicit action. However, the old way had a little dance
between the old process and the new process: the new process tells the old
process to start shutting down, the old process stops listening for new
connections, then the new process starts listening for new connections. That
left a gap where connections got rejected.

The new technique is for the old process to use a Unix socket to seamlessly
transfer ownership of the listening sockets to the new process. At no point
are the listening sockets closed, so no connections are rejected.

It's still a (potentially) new haproxy binary starting up and parsing the
(potentially) changed haproxy config because the user requested a graceful
restart.

~~~
vbernat
The new process listen to connections before the old process stop listening.
The problem is that the old process can still have new connections queued up.
They are lost when its sockets are closed.

------
jolynch
I'm so excited about this. We just finished rolling out a new seamless
strategy involving pairing NGINX with HAProxy which I am almost done with the
blog post for, but I envision this making our solution even simpler in the
future when it hits stable branches.

Absolutely awesome.

------
gwu78
Example line from the Openshift reload-haproxy script:

    
    
      old_pids=$(ps -A -opid,args | grep haproxy | egrep -v -e 'grep|reload-haproxy' | awk '{print $1}' | tr '\n' ' ')
    

Can we do this without grep, egrep and awk? Would this work?

    
    
      old_pids=$(exec ps -A -opid,args |sed -n '/sed/d;/reload-haproxy/d;/haproxy/{s/ .*//;p};'|tr '\n' ' ')

~~~
ploxiln
Yeah, the "ps | grep | grep -v grep" pattern is silly, there are a few better
ways. In this case just:

    
    
        old_pids=$(pgrep '^haproxy')

~~~
gwu78
"... is silly, there are better ways."

I see this usage often where it seems like

    
    
      grep pattern1 | grep -v pattern2 
    

can be replaced by

    
    
      sed -n '/pattern1/d;/pattern2/p'
    

or at least

    
    
      sed '/pattern1/!d' | sed '/pattern2/d'
    

or

    
    
      sed -n 's/pattern1pattern2//g;/pattern2/p'
    

But I must be missing something obvious.

For example look at the "grep -v" usage here:

[https://github.com/thomwiggers/qhasm/raw/master/qhasm-
arm](https://github.com/thomwiggers/qhasm/raw/master/qhasm-arm)

Is there something wrong with using

    
    
       sed '/^op:livefloat80:/d'
    

Moreover, in the last line, why not use

    
    
      sed 's/\$/#/g'
    

instead of

    
    
      tr '$' '#'
    

Apologies if I am missing the obvious.

~~~
gnaritas
All of your suggestions are more complicated than what you're suggesting
replacing. People chain simple commands together because they're a language
and it matches how they think of the problem. They're solving the problem with
simple commands and pipes, you're trying to solve it with regex and as few
commands as possible. All ways are valid but specific commands tend to be
easier to remember on the fly than trying to do it all with sed and regexes. I
use sed when I want to edit streams, not when I want to filter them. Tr is a
simpler replace than a sed regex.

------
greglindahl
As a comparison, nginx -- which is a quite different bit of software but is
sometimes used in a similar fashion to haproxy -- can be gracefully restarted
several times per second with no issues.

~~~
annnnd
What do you mean by "with no issues"? Do you want to say that it doesn't drop
connections? If that is the case, I would be curious to know more - but
cursory search doesn't support this claim. [0]

If you mean something else then HAproxy also supports "graceful restarts
several times per second with no issues". But the article is not talking about
that.

EDIT: added link.

[0] [https://serverfault.com/questions/523417/reloading-new-
nginx...](https://serverfault.com/questions/523417/reloading-new-nginx-
configuration-with-no-downtime)

~~~
greglindahl
I mean it didn't drop our user connections.

The link you give talks about persistent HTTP connections paired with a broken
http client.

~~~
annnnd
Actually the link I posted dealt with reloading config, not restarting (which
is something completely different - you can't upgrade binary that way). But
broken clients are everywhere, so you can't discount them. And persisten
connections too, while we are at it.

I did however find instructions how to properly restart Nginx without dropping
connections: [https://www.digitalocean.com/community/tutorials/how-to-
upgr...](https://www.digitalocean.com/community/tutorials/how-to-upgrade-
nginx-in-place-without-dropping-client-connections) Apparently it can be done
(and could even be automated), but the procedure looks very generic to me (not
nginx-specific). Is this what you did?

------
tamalsaha001
I wish they had prioritized HTTP/2 in HTTP mode.

