

Nginx with dynamic upstreams - Tenzer
http://tenzer.dk/nginx-with-dynamic-upstreams/

======
bmurphy1976
I spent a long time fighting this battle with nginx. I eventually came to the
conclusion that while it was technically possible, it just wasn't worth the
trouble (and yes, I tried various alternatives such as Tengine and OpenResty).

What I settled on was the following:

Ngninx for SSL termination in front of an haproxy load balancer. I wrote a
simple Python script (92 lines of code) that spits out a new haproxy
configuration and gracefully reloads haproxy whenever the DNS changes. That
has (after fixing a bug or two in my code) been FAR more reliable than any
other solution to this problem we've tried.

Bonus: haproxy is a superior load balancer with better runtime metrics and
health checking options than nginx.

~~~
adamonduty
What part of the battle couldn't you solve? Doesn't the author's setting of
the resolver directive work for you, or is there something I'm missing?

Also, the author doesn't mention it, but you can also set a custom cache
timeout with the valid=Xs option.

[http://nginx.org/en/docs/http/ngx_http_core_module.html#reso...](http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver)

~~~
bmurphy1976
It was not that reliable for us. We still had unexplained occurrences of IP
caching and it's easy to fat finger the configuration and lose the resolving
functionality. Anyway, as I mentioned haproxy has _vastly_ superior load
balancing capabilities. Nginx still has a lot of work to do to match haproxy
in this area.

------
Rapzid
I've been working with nginx quite a bit recently for use as a reverse proxy
in a containerized environment and have become quite disenfranchised with it.

For some reason it's the most popular reverse proxy for this sort of stuff,
however it's not particularly well suited for it. We have this issue. We also
have the lack of active upstream checks and any sort of upstream status
reports. Both are hidden behind the nginx Plus paywall which is absurdly
expensive in a micro-architecture world. There are various patches, Patches!?,
you can apply but the documentation isn't fantastic and this is an extra bit
of hassle we don't really deserve..

I think the open internet deserves a better reverse proxy TBH.

~~~
elithrar
Have you looked into HAProxy? Health checks, SSL termination and a reasonable
configuration DSL: [https://www.digitalocean.com/community/tutorials/an-
introduc...](https://www.digitalocean.com/community/tutorials/an-introduction-
to-haproxy-and-load-balancing-concepts)

~~~
Rapzid
I am acutely aware of HAProxy however the rest of the team is, no surprise I'm
sure, familiar with nginx. Likely I'll end up doing the usual
nginx->haproxy->backend chain again. HAPROXY appears to have an enterprise
now(it may have had forever I just have not noticed) but it appears to be much
more focused on value-add support and stack validation than hiding simple-to-
implement stuff behind a paywall and open-core. This is more of a frustration
with nginx which could be so much more for us if:

* They were not purposely holding back features so they can be pay-walled

* Re-compiling were not necessary to add functionality

* More cohesive modification ecosystem

I know I said "reverse proxy" which HAProxy is cleary the more superior, but I
didn't mean to limit my statement to reverse proxies. nginx does have a lot of
other functionality people rely on..

~~~
Tenzer
Dynamic modules are on the road map, according to this article:
[http://www.infoworld.com/article/2951849/web-
services/nginx-...](http://www.infoworld.com/article/2951849/web-
services/nginx-to-hop-on-http-2-train.html).

------
fica
Kong [1] (nginx + lua) is planning to have this feature for free. This is the
issue open for support for dynamic upstream [2].

[1] [https://github.com/mashape/kong](https://github.com/mashape/kong)

[2]
[https://github.com/Mashape/kong/issues/157](https://github.com/Mashape/kong/issues/157)

~~~
dekz
Unfortunately kong still has a hard dependency on Cassandra.

[https://github.com/Mashape/kong/issues/331](https://github.com/Mashape/kong/issues/331)

~~~
sinzone
Before the end of the year Kong will support Postgres.

~~~
dekz
That is fantastic news!

------
Cieplak
Here's an implementation that uses nginx's X-accel-* headers to perform
dynamic routing:

[https://github.com/bninja/rump](https://github.com/bninja/rump)

These are what the routing rules look like:

[https://github.com/bninja/rump/blob/master/test/fixtures/set...](https://github.com/bninja/rump/blob/master/test/fixtures/settings/include.conf)

------
jpmattia
Am I wrong for thinking that failure to honor the TTL in the first place is a
bug?

~~~
lisper
It's worse than a bug, it's a Really Bad Design (tm). Bugs are usually
unintentional, but this is a deliberate reinvention of a wheel (IP address
caching) that doesn't need to be reinvented.

But this is not just nginx's fault. It's also Bad Design on the part of AWS
because switching IP addresses this way means you can't keep a TCP socket open
to an ELB machine for more than 60 seconds at a time because you never know
when the routing rug is going to be yanked out from under you. This makes ELB
useless for anything involving a persistent connection. No websockets for you!

~~~
Tenzer
Regarding how the ELB works, then I think the AWS engineers have a different
idea on how to implement stuff. AWS is very much a dynamic platform and the
engineers seem to have embraced this when they came up with this solution.

It doesn't necessarily mean you can't use websockets through an ELB, it just
means that you would need to be able to handle reconnects, but that shouldn't
be a new challenge for any system relying on connections being open for long.
Also, the load balancer servers doesn't switch every 60 seconds, you can have
connections running for a lot longer than that. I would also assume the load
balancers keep handling connections for a while after they were taken out of
the DNS rotation, in order to make sure DNS caches are updated before the IP
addresses stops working.

~~~
JimmyL
AWS support has said that ELBs will continue to accept connections (and use
the correct backend) for at least an hour after the CNAME stops resolving to a
particular IP.

~~~
lisper
Ah, I didn't know that. That makes a big difference.

But there's still a significant hole here: is there any way to get notified
that this has happened other than polling the DNS?

------
corobo
Thank you for this, it's a decent start for me personally. I was previously
using the DNS director in Varnish 3.x to do this which was removed in 4.x. The
longer time goes on the more risk it is to be running older software so this
has been a great help.

Oddly enough I'd never really considered nginx for the job despite using it to
reverse proxy elsewhere. Sometimes you just need a poke in the right direction
:)

Now I just need to figure a way to specify *.internaldomain so that nginx
resolves www.example.com.internaldomain - where www.example.com is grabbed
from the requested Host: header

~~~
Tenzer
It's good to hear it was of use to somebody :)

I know you can use a map in Nginx to do what you ask for, as long as you have
a list of domains already:
[http://nginx.org/en/docs/http/ngx_http_map_module.html#map](http://nginx.org/en/docs/http/ngx_http_map_module.html#map).
I can only imagine it also being possible to make fully dynamic, I just don't
have a clear way of doing it in mind right now.

~~~
sprobertson
Lua + Redis is a nice way to do this, I have a small project
[https://github.com/spro/simon](https://github.com/spro/simon) that does
dynamic routing and load balancing based on Hostname -> IP:Port sets in Redis.
Adding a new route is as simple as:

    
    
      sadd backends:hnapi.dev 192.168.0.42:10555

~~~
Tenzer
Neat! I want to get started on some Lua scripting as well, probably first to
just get a bunch of metrics out of Nginx via StatsD. There's access to a whole
bunch of numbers via Lua that you otherwise can't get out from the stub_status
page.

------
brryant
Another way around the rewrite is to simply proxy_pass with the $uri:
[https://gist.github.com/sansmischevia/cf425d5ffe09f824cb27](https://gist.github.com/sansmischevia/cf425d5ffe09f824cb27)

~~~
Tenzer
Sure, that's also what I cover in the post. The latter part about rewriting is
only relevant when you want to have the service behind an ELB to be located at
something else than "/".

------
mattdeboard
Can you not assign static IPs to ELB instances? This might be a stupid
question...ELB is one of the older AWS resources that I've never really
touched since Nginx is so powerful & easy to set up.

~~~
Tenzer
No, you can't. You can instead create your own EC2 server, give it a static IP
(well, "Elastic IP") and then hope your own server doesn't go down. This is
the reason why I prefer to use an ELB since I never seen stability issues with
it.

