The solution you mention has some tradeoffs. Imagine that you have hundreds of services each of which have a port and have some long running connections. With your solution you can only restart the load balancer at the rate of the longest running connections or whenever ports run out, whichever comes first.
Also then you'd have to maintain the logic to do the port mapping, iptables switching, draining, etc ...
The post mentions they considered this option and they claim that they were worried about engineering risk; seems reasonable to be worried about that.
With your solution you can only restart the load balancer at the rate of the longest running connections
Well, perhaps I should have provided more detail. Our ansible deployment script actually counts the port up until it finds a free one (we keep a port-range reserved for this purpose). So if there are multiple changes in rapid succession then more than two haproxy instances may dangle around for a while.
The "port discovery" is a shell-task that registers the port to be used as a variable, which is then used by the templates of the haproxy and iptables roles.
The cleanup is done by a 15min-cronjob which kills all haproxy instances that have no connections in netstat and don't match the haproxy pidfile.
The HAProxies in question are dealing with at least one port per service, with dozens of services. If you were to remap ports, you'd end up with a lot of port mapping rules.
We also considered using different loopback IPs per HAProxy, so ports could stay consistent, but decided against it:
- We have other things (like scribe) listening on the same loopback IP address, so we'd either have to move those to different IPs or exclude them from iptables rules.
- We thought it could be confusing/misleading to see HAProxy listening on one IP/port, but connections being made to a different IP/port.
That's how we do it and the script isn't even complicated.
Use iptables to shift the traffic to the new instance, wait for the old instance to drain (netstat is your friend), and kill it.
Much respect to the effort that went into the yelp solution, but it does feel a bit brittle and over-engineered for a such a simple problem.