
Fixng Nginx 502 Gateway Timeout with proxy_pass During Deployments - winrid
https://blog.winricklabs.com/(2-08-2020)-fixing-nginx-502-gateway-timeout-with-proxy_pass-during-deployments.html
======
mfontani
The article mentions nginx returning 502 when a backend is down during a
deploy. This a problem in the cases where restarting the backend isn't atomic.

If the backend is able to perform a "reload" (vs. a slower stop/start or
restart) and/or replace the old instances with the new when they're up --
assuming that's an atomic operation -- nginx will not return a 502.

If one is rapidly iterating and prefers having a somewhat cleaner stop/start,
the advice can be helpful even though it masks/papers over a problem (the
backend being down).

~~~
winrid
Correct.

------
simplyinfinity
To me the most appropriate way (when not dealing with kubernetes, but rather
just vms) is to be able to mark specific server to not receive any new
requests, wait for the requests to stop coming in and then deploy.

IIS web server has this but not nginx :(

~~~
pritambaral
I use HAProxy for this. The paid version of Nginx also has this.

------
notyourday
Detecting a failure and minimizing the request wreck that it causes is
different from handling a deploy. The second one should cause no wrecks
because your deploy management system should be able to tell your proxy layer
to drain the target of deploy to allow the deploy to proceed when a target has
no traffic

------
kart23
I've never had this problem when doing reverse proxies through nginx. Theres
no delay when I restart node, the 502 goes away right after.

~~~
winrid
You have to make a request while the upstream is down. So do the restart, then
make a request right away. If your service restarts quickly it'll be hard to
reproduce.

------
mehrdada
Envoy is the new hotness in proxyland.

~~~
winrid
Nice to know. However, if I think if I was going to redo everything I'd use
Kubernetes+Istio?

~~~
hedwall
Istio uses Envoy for the data plane.

~~~
winrid
Oh cool. It shows I haven't dived very deeply into it yet. :)

------
viraptor
Is this "fixing"? The proper fix would be to use a seamless deployment by
either spawning a new vm with new version, then redirecting the traffic once
it's ready. Or by doing the same with a new version of the app server on the
same host and replacing the socket when ready.

Otherwise, the requests will potentially wait a long time to be handled while
the app starts up.

~~~
winrid
In this case it's mostly for experiments and stuff. And spawning a new vm etc
is slow for that kind of thing. This takes less than a second (I just merge to
master and it deploys).

