
Gracefully Restarting a Go Program Without Downtime - aberoham
https://gravitational.com/blog/golang-ssh-bastion-graceful-restarts/
======
ebikelaw
This kind of thing is foreign to me because I can't imagine writing an
application, to run in the cloud (as I always do) where I care if a single
instance has some downtime. I always rely on something at a higher level
proxying my traffic to the application, and when I need to push my application
I just shoot each instance in the face and start another one elsewhere,
usually on a different host:port. When I'm using the cheapest cloud capacity I
can buy my application is constantly preempted by higher-paying tenants
anyway, so I have to write my systems as if any instance will suddenly stop.
Once you have that problem you can forget about graceful in-place restarts
forever.

~~~
Thaxll
You know that some people work on servers that are stateful? Not everyone use
REST type of transport where a failed node is ok because the lb will route
traffic to another place. Your solution doesn't work for things like game
servers.

~~~
viraptor
Even stateful services like game servers could be organised to handle losing a
node. It's harder to do and you need some state replication in place.
(versioned state if you want to do upgrades that way) It may not be cost
effective to do so. But the possibility is there.

I worked on a plan to have live migration of multiparty phone calls between
data centres in realtime. The tech is available. But your planning outcome may
be: it's cheaper to make single node more stable and deal with the occasional
failure.

"Stateful" only means you need to care about the state being moved. Things
like Java hotswap, Erlang live code reload, and your custom solutions can
handle that.

------
crawshaw
What a coincidence seeing this topic today. Just yesterday I released some
similar code for upgrading Go servers in-place while maintaining sockets:

[https://github.com/crawshaw/littleboss](https://github.com/crawshaw/littleboss)

------
lmb
It is possible to pass file descriptors using os/exec, using the ExtraFiles
attribute [1]. No need drop down to os.StartProcess.

1: [https://golang.org/pkg/os/exec/#Cmd](https://golang.org/pkg/os/exec/#Cmd)

------
kjeetgill
Fantastic examples. I appreciate that they allude to the (canonical?) example
of how Nginx does zero downtime upgrades using SO_REUSEPORT. I had hard of the
trick of sharing the FD with a forked process but it's nice to a working
example.

------
erik_seaberg
It's theoretically interesting to hand off a listen socket, but I'd rather
take 1/n machines out of service than leave half of my memory unused on every
machine.

 _Edit:_ I also remember hearing some out-there stuff about handing off your
IP address and open connection state to another machine so the client doesn't
even notice…

------
ngrilly
Will Tarreau of haproxy has written an insightful post on the same topic:

"Truly Seamless Reloads with HAProxy – No More Hacks!"

[https://www.haproxy.com/blog/truly-seamless-reloads-with-
hap...](https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-
more-hacks/)

------
z3t4
I like to use a simple protocol: 1) Server receives shutdown signal 2) Server
tells all clients to disconnect, and also tells the load balancer to not send
more traffic 3) Client's connect to another server or waits for the server to
restart 4) server exit

~~~
sleepydog
How do you tell the load balancer to stop sending traffic? I've done two ways
in previous jobs:

\- Refuse all new connections. If you can change the program, this can be done
by closing the listening socket (I think) , but I just installed an iptables
rule.

\- use the load balancer's API to offline the server(at the time it was F5).

I liked the first solution because it was load balancer agnostic, but required
retries by the load balancer or client to avoid disruptions. Curious to hear
what other people did.

~~~
kyrra
See the lameduck section of this chapter of the Google SRE book. They do
frequent health checks on all running jobs. If a health check fails they stop
sending that job traffic. The concept of lameduck is you report yourself as
unhealthy but are still available to serve traffic, you wait before all of
your clients that were loaded balanced to you before you stop a sending
traffic.

[https://landing.google.com/sre/book/chapters/load-
balancing-...](https://landing.google.com/sre/book/chapters/load-balancing-
datacenter.html)

Health checks are part of GCPs load balancer.
[https://cloud.google.com/compute/docs/load-
balancing/health-...](https://cloud.google.com/compute/docs/load-
balancing/health-checks)

------
latchkey
This is an excellent write up, very easy to follow. Overseer [1] is a pretty
good implementation as well.

"Commonly, graceful restarts are performed by the active process (dark blue)
closing its listeners and passing these matching listening socket files
(green) over to a newly started process. This restart causes any foreground
process monitoring to incorrectly detect a program crash. overseer attempts to
solve this by using a small process to perform this socket file exchange and
proxying signals and exit code from the active process."

[1]
[https://github.com/jpillora/overseer](https://github.com/jpillora/overseer)

------
wejick
I'm using socketmaster[1] to do the graceful update or just normal service
reload. It is basically another process to facilitate socket exchange between
2 process. Just make sure that the service able to shutdown gracefully.

However this approach need the binary to be started from socketmaster (as
child process). I'm not sure what is the real cons tho, except the fact that
it alter SIGHUP behavior - which is fine in this case imo.

[1]
[https://github.com/zimbatm/socketmaster](https://github.com/zimbatm/socketmaster)

~~~
wejick
[https://github.com/paytm/grace](https://github.com/paytm/grace) The graceful
handle on the service side can use this package

------
epoxyhockey
Another reference project:
[https://github.com/facebookgo/grace](https://github.com/facebookgo/grace)

------
joevandyk
Seems simpler to do this at the load balancer level. That way you can use the
same strategy for any type of application.

~~~
SteveNuts
What if your Go application IS a load balancer?

~~~
ebikelaw
Even in that case you don't need graceful in-place restarts. If you have
multiple replicas of the load balancer and ECMP from higher on your stack, all
you need is for your balancer process to 1) remove itself as a destination for
new flows; 2) stop accepting new connections and complete all existing
connections; 3) shut down and restart; 4) add itself back to the set of new
flow destinations.

If you don't have multiple instances of your load balancer, that's weird.

~~~
kevin_nisbet
I don't believe this approach would work for a layer 7 load balancer (where
the connection is terminated on the host). This is in regards to the original
article, which is discussing how to maintain socket state through a process
upgrade.

When you remove the host from the routing table, the ECMP algorithm will
redistribute the traffic to the other hosts still in the pool. For TCP
connections, this will generate reset's, and drop any active connections moved
between hosts. Also, at times I've not gotten clear answers from router
vendors on whether they support a consistent hash in their ECMP
implementation, which means that connections through the load balancers not
being reloaded may also be rebalanced to other hosts and be affected by the
routing table change.

Even if we consider going down the stack to a layer 4 load balancer, outside
the scope of this article, it may still be desirable to combine approaches and
offer hitless reloads (a common case) and not just recover from a node failure
(a less common case).

The reason is, that you would need state replication of the load balancing
tables to support removing the load balancer from the routing tables, and
having another load balancer take over it's work while maintaining the
connections through the load balancer. This is under the assumption that the
service being load balanced requires an uninterrupted connection of course.

Disclaimer: I work with the author of the article and contributed to it.

~~~
ebikelaw
I've worked with ECMP routers where you can inform that router you'd like new
flows to be sent elsewhere while existing flows continue to be sent to their
established endpoints. JunOS for example has this feature.

~~~
AdamJacobMuller
Do you recall what option/platform that is? I assume that must have been
something like an SRX, not an MX/QFX/EX.

------
berdon
I made a blog post a couple years ago about doing this for a mud in Go using
the FD passing approach.

[http://austinhanson.com/slinging-mud-graceful-
reboots](http://austinhanson.com/slinging-mud-graceful-reboots)

------
gonyea
“modern systems software”

Lol, no. Modern, professional software development uses a load balancer to
manage a canary/red-black deployment strategy. Zero downtime deployments to a
live server, outside of telecom/finance, is a sign of an amateurish team.

------
spraak
Does anyone have any good articles about the same technique for Node?

