
Migrating Dropbox from Nginx to Envoy - SaveTheRbtz
https://dropbox.tech/infrastructure/how-we-migrated-dropbox-from-nginx-to-envoy
======
e40
_Also note that we’ll cover the open source version of the Nginx, not its
commercial version with additional features._

It always kills me when very successful companies don't buy software from
other companies.

I remember being at a lunch with a prospective client that really loved our
technology. About 1/2 way through, he said he really would love to purchase
our software, but the CEO doesn't allow them to use anything but OSS. What
they make? Non-OSS software.

Just blows my mind.

~~~
holografix
This also rubbed me the wrong way. As an individual I think that shows selfish
and opportunistic behaviour and it raises a red flag about that organisation
in my mind.

However, for profit companies are not here to do what’s “correct” they’re here
to make money for its investors. If I had decision making abilities at Nginx
I’d be conducting a comprehensive review of the free OSS offering and
redacting the features and overall value with extreme prejudice.

Dropbox never paid because it COULD not pay. If you have an enterprise, paid
version of your OSS product it has to be impossible for an enterprise to use
it for free.

~~~
dragonwriter
> If you have an enterprise, paid version of your OSS product it has to be
> impossible for an enterprise to use it for free.

Why? Most enterprises, especially ones that aren't tech firms, are going to
shell out for enterprise support even if there are no additional features.
Crippling the community version doesn't necessarily help enterprise sales, it
can reduce overall mindshare reducing enterprise traction or, worse yet, mean
that a third-party downstream edition with richer open-source features becomes
dominant and it's creator gets “your” enterprise support contracts.

~~~
chii
> shell out for enterprise support even if there are no additional features.

i don't feel this to be true.

Also, an enterprise that's large would want some features that are irrelevant
to a small shop. For example, single-sign-on integration with various
providers.

~~~
aliswe
I do though ... Imagine a manager being dependent on a system he ~~bought~~
_installed for free_ , which he didn't buy support for, and is now
malfunctioning. It's his fault. And this is what I've seen in practice as
well.

Just think about the commercial success of SUSE Linux?

------
ram_rar
I feel so old now. There was a time, when I used to discuss with senior
engineers @ Yahoo! to use NginX over Apache. Nginx was the hot thing,
popularizing C10k [1]. Now in my current team, I have junior devs in my team
pushing for Envoy over HAProxy/Nginx setup.

Is this trend happening primarily because devs are pushing for GRPC over REST?
What benefits does Envoy offer over Nginx, if you're still a REST based
service. I am not fully convinced of operational overhead that NGINX brings.

[1]
[https://en.wikipedia.org/wiki/C10k_problem](https://en.wikipedia.org/wiki/C10k_problem)

~~~
Matthias247
The sibling comments point towards the difference in configuration if you take
the "out of the box" product. But there is also a vast difference in how code
is organized, in case you ever have to touch it.

From my point of view Nginx feels "old". It's a C codebase without a great
amount of abstractions and interfaces, and instead having a bunch of #ifdefs
here and there. Unit-tests and comments are not to be found. Build via
autotools.

Envoy looks as modern as it gets for a C++ codebase - apart from maybe the
lack of using coroutines which could be interesting for that use-case. It uses
C++14, seems to be extremely structured and documented, has unit-tests, uses
Bazel for builds, etc.

So I think the experience for anyone being interested in working on the code
will be very different, and people that prefer the style of project A will
have a very hard time with the style of project B and the other way around.

~~~
MrBuddyCasino
> uses Bazel for builds

Is this unanimously good? I've heard both praise and horror, never used it
myself.

~~~
SaveTheRbtz
One of the senior engineers once said to me that "Bazel is like a sewer: you
get back what you put in."

Bazel requires __a lot __of upfront effort but the power of (a
programmatically accessible /modifiable) dependency graph and a common
build/test system across all the languages is very hard to underestimate.

~~~
MrBuddyCasino
> very hard to underestimate

Are you sure?

------
rdli
Really great post. I'm glad the post in particular mentioned community,
because I think in the end this is the huge advantage Envoy has over NGINX.
NGINX, could, in theory, resolve all technical issues raised in the post. But
the fundamental tension between the open source and commercial versions cannot
be resolved.

(Disclosure: We use Envoy as part of Ambassador, and so of course we're big
fans!)

------
dmicher
I know some people might find it a little controversial, but I’m super excited
about our load balancing future and that we probably have the biggest Envoy
deployment in the world now. When we moved most of Dropbox traffic to Envoy,
we had to seamlessly migrate a system that already handles tens of millions of
open connections, millions of requests per second, and terabits of bandwidth.
This effectively made us into one of the biggest Envoy users.

~~~
user5994461
Well, a single server doesn't really need to do more than 10Gbps or 100k
connections. Going above is a "simple" matter of managing horizontal scaling.

What I wonder about is how do you distribute the traffic on the higher level?
I imagine there are separate clusters of envoys to serve different
configurations/applications/locations? How many datacenters does dropbox have?

I was running a comparable setup in a large company, all based on HAProxy,
there was a significant amount of complexity in routing requests to
applications that might ultimately be in any of 30 datacenters.

~~~
SaveTheRbtz
We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR
is:

* First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients.

* User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!)

* Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS, now Katran[4]) that encapsulate traffic and DSR it to L7 balancers (previously nginx, now mostly Envoy.)

Overall, we have ~25 PoPs and 4 datacenters.

[1] [https://dropbox.tech/infrastructure/dropbox-traffic-
infrastr...](https://dropbox.tech/infrastructure/dropbox-traffic-
infrastructure-edge-network) [2]
[https://dropbox.tech/infrastructure/intelligent-dns-based-
lo...](https://dropbox.tech/infrastructure/intelligent-dns-based-load-
balancing-at-dropbox)

[3] [https://www.dropbox.com/peering](https://www.dropbox.com/peering) [4]
[https://github.com/facebookincubator/katran](https://github.com/facebookincubator/katran)

~~~
dilyevsky
Katran - nice! Any issues with it at all? Do you use it with xdp capable
hardware or just normal driver offload?

~~~
SaveTheRbtz
It works beautifully. We use driver offload (i40e on the Edge.)

------
eric4smith
It's interesting almost no web server provides an easy way to deal with multi-
tenant multi-domain architectures in a good way that includes automatic SSL.

Caddy is the closest, but still not near enough.

There is this small segment of the market that we operate in that requires
thousands of TLS connected domains to be hosted behind a dynamic backend. It's
services like Tumblr, Wordpress.com, or any other hosting service where you
can get a "custom domain" to point to your own blog or site.

NGINX - No.

Apache - Nope.

Caddy - Can do (but need lots of workarounds)

Envoy - Nope.

Everyone focuses on a few hand-coded domains and no automatic TLS. Maybe this
part of the market is too small anyway. Sigh.

~~~
mholt
Several companies use Caddy for exactly this purpose. Fathom Analytics for
example uses it for their custom domains feature. Caddy can even reactively
provision certs during TLS handshakes. It's a native feature. Why does it
require lots of workarounds?

~~~
sladey
Is there any mature integration to achieve this with Kubernetes?

~~~
mholt
WIP:
[https://github.com/caddyserver/ingress/](https://github.com/caddyserver/ingress/)

------
shay_ker
> C++14 is not much different from using Golang or, with a stretch, one may
> even say Python.

That's... definitely a stretch.

------
spacewander
One of my friends brought me up this post in the morning. The post is awesome
and inspirational (caused a discussion in our chant group), though I can't
agree with some trivial points.

> Nginx performance without stats collections is on part with Envoy, but our
> Lua stats collection slowed Nginx on the high-RPS test by a factor of 3.
> This was expected given our reliance on lua_shared_dict, which is
> synchronized across workers with a mutex.

The `a factor of 3` is quite large to me. Maybe you put all your stats in
lua_shared_dict? You don't need to synchronize the stats every time. Since the
collection regularly happens in per-minute frequency, you can put the stats as
Lua table, and synchronize them once per 5/10 seconds.

It look like that the compared Nginx is configured with a system which has
been survived for years and not up-to-date. The company I worked with used a
single virtual server to hold all traffic and routed them with Lua code
dynamically. And the upstream is chosen by Lua code too. There is no need to
reload Nginx when a new route/upstream is added. We even implemented 'Access
Log Service' like feature so that each user can have her favorite access log
(by modifying the Nginx core, of course).

However, I don't think this post is incorrect. What Envoy surpasses Nginx is
that it has a more thriving developer community. There are more features added
into Envoy than Nginx in the recent years. Not only that, opening discussion
of Nginx development is rare.

Nginx is an old, slow giant.

~~~
rolls-reus
> The `a factor of 3` is quite large to me. Maybe you put all your stats in
> lua_shared_dict? You don't need to synchronize the stats every time. Since
> the collection regularly happens in per-minute frequency, you can put the
> stats as Lua table, and synchronize them once per 5/10 seconds.

Any pointers on how to achieve this for someone just starting out with lua and
openresty? I have the exact same thing (lua_shared_dict) for stats collection,
would love to learn a better way.

~~~
spacewander
You can look at [https://github.com/knyar/nginx-lua-
prometheus/pull/75](https://github.com/knyar/nginx-lua-prometheus/pull/75) for
inspiration.

------
pmlnr
If anyone was wondering, this is solely for proxying, not for oldschool web
server functionalities, eg. static file serving.

~~~
SaveTheRbtz
We've actually started experimenting with converting our static file serving
to "proxying to S3 + caching." This is simpler from deployment and development
perspectives (for companies that do not have a distributed filesystem, like
Google with its GFS):

* for deployment we do not need to maintain a pool of stateful boxes with files on them and keep these files in sync.

* for development, engineers now have a programatic interface for managing your static assets.

------
caiobegotti
I'm positively surprised that Dropbox (at least from what I understood from
the post) didn't require lots of changes or patches on top of the upstream
codebase of Envoy to migrate their traffic!

~~~
SaveTheRbtz
We did require some of them[1]. Esp. painful were Transfer-Encoding quirks,
and some dances around old HTTP/1.0 backends and request buffering.

Compared to NGINX though, it was relatively easy to push these fixes upstream.
Community is very welcoming to outside contributions.

[1] [https://dropbox.tech/infrastructure/how-we-migrated-
dropbox-...](https://dropbox.tech/infrastructure/how-we-migrated-dropbox-from-
nginx-to-envoy#-issues-we-encountered-)

~~~
veshij
We do have some local patches as well (mostly for integration with out own
infrastructure - stats collection, some RPC specific stuff). As SaveTheRbtz
mentioned we encountered some issues with non-RFC clients, corner cases which
were not exposed when envoy is used in "trusted" environment, etc., but all
our fixes are now in upstream, so next migrations will be way easier both for
us and for other envoy users.

------
weitzj
I did not quite get how they configure envoy? Did they write their own control
plane? Use ambassador/Istio/Gloo?

~~~
veshij
We have a mix of static and dynamic configuration. We started with almost
everything defined in the configuration and implemented our control plane only
for endpoint discovery service. Over the time we implemented more and more
features there (certificates, tls tickets, route and vhost configuration,
etc). We decided to write own implementation on control plane - actually the
core part is pretty simple and easily expandable.

------
mperham
Did you consider using commercial nginx? If so, what made you decide against
it?

~~~
xmichael0
The price is really insane for Nginx commercial.

~~~
mperham
As an Enterprise software vendor myself, I can assure you: everything is
negotiable at Dropbox’s scale including very deep discounts.

------
xet7
How does Envoy compare to Caddy 2 ?
[https://caddyserver.com](https://caddyserver.com)

~~~
SaveTheRbtz
To tell you the truth, we didn't consider it. From what I can get from the
architecture docs[1], it can be a decent platform for apps, but might not be
the best choice for a general purpose ingress/egress proxy (at least for now.)

[1]
[https://caddyserver.com/docs/architecture](https://caddyserver.com/docs/architecture)

~~~
mholt
It is a great choice for a general purpose proxy. (That's kind of the point.)

~~~
atonse
But they mentioned that they wanted to use C++ instead of go to get even that
extra performance out.

I use Caddy a lot and it's perfectly fine for my scale, but at dropbox's
scale, maybe go wouldn't be enough for the ingress part?

~~~
mholt
I just lament the increasing deployments of programs written in memory-unsafe
languages to the edge, in general.

I am more curious what makes the author think Caddy "might not be the best
choice for a general purpose ingress/egress proxy" (there were no other
qualifications to that statement, but no evidence to support it either).

~~~
mperham
Yeah, to its credit, the article brought it up but then kinda hand waved away
"envoy had many more security issues than nginx". Having a huge load of C
library dependencies in a user-facing service seems like a bug these days.

Part of reducing dependencies in my own software was a conscious decision to
minimize future CVE exposure.

------
Havoc
Sensing a bit of a trend here. Didn't another major player recently make the
same switch?

~~~
Thaxll
It's a drop in the water compared to Nginx usage.

~~~
SaveTheRbtz
That is indeed true. But, I remember the time when we were rolling out nginx
back in 2000's and exactly the same thing was said about Apache.

------
varbhat
[https://h2o.examp1e.net/](https://h2o.examp1e.net/)

This is also good web server.Configuration is done in yaml. Also,it claims to
be very fast.

------
user5994461
A shame they picked nginx in the first place, it has all the stats and
critical features behind the paid edition. HAProxy is always a better choice
for load balancing.

Besides that, it looks like the move was significantly driven by GRPC and
profobuf. No surprise here, GRPC really doesn't work well over HTTP. Once a
company start using the google stack, they have to move to more of the google
stack to make it usable.

~~~
SaveTheRbtz
Our technology stack is very gRPC friendly, so developer experience is
actually better with it, than without (though this is very subjective.)

As for the middleboxes, using gRPC-WEB[1] allowed us to switch Desktop Client
App to gRPC even behind firewalls/IDSes that do not speak HTTP/2 yet.

As for the HAProxy, Dropbox used to use (circa 2013) it specifically for
loadbalancing, but we eventually replaced it with our Golang proxy. That said,
recent HAProxy improvements (v2.0+) make it quite an awesome dataplane and an
excellent loadbalancer!

[1] [https://github.com/grpc/grpc-web](https://github.com/grpc/grpc-web)

------
polskibus
Thank you for the thorough comparison. Could anyone chip in whether a recent
haprozy version would be a better choice than nginx and /or envoy in a similar
case?

------
MuffinFlavored
Can somebody speak to why dynamic upstreams included in a file paired with
`sudo service nginx reload` for prod deploys stopped scaling?

~~~
SaveTheRbtz
It is easy enough for simple cases (and we used it for quite a while, until we
moved to using Lua for that.) For more complex scenarios you will have new
`server` blocks, certificates, tls tickets, log files / syslog endpoints, so
the automation will end up interacting not with just a single dynamic upstream
file but with rather large amount of system interfaces. Control-plane ends up
being distributed between config generation, filesystem state, service
configuration (e.g. syslog.)

On a more practical note, each nginx `reload` will double the number of
workers, almost doubling memory consumption and significantly increasing CPU
usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So
there is only that many reloads that you can do in an hour.

~~~
DarkWiiPlayer
nginx is not well suited for constantly reconfiguring your infrastructure on
very hot servers. This is a problem when you expose such infrastructure
configurations to users (think cloudflare), but otherwise you can just
mitigate this problem by having a sane deployment strategy.

------
sroussey
One thing nice about OpenResty (nginx) and their Lua support is that it plugs
in at TLS negotiation. Does Envoy?

~~~
SaveTheRbtz
Can you describe your use-case?

If you are talking about the ability to select a certificate on the fly via
`ssl_certificate_by_lua_block`[1] we are not aware of such functionality. If
you are missing something, I would highly encourage you discuss it with the
community on a github!

From Oleg Guba, Traffic Team TL, co-author, and person driving the deployment:

* ListenerFilters + NetworkFilters are flexible enough, that some of the custom logic could be just moved to the config.

From Ruslan Nigmatullin, our head Envoy developer:

If you are talking more about a custom verification code there is already
couple of ways to do that:

* Client TLS auth Network Filter: [https://www.envoyproxy.io/docs/envoy/latest/configuration/li...](https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/client_ssl_auth_filter#config-network-filters-client-ssl-auth)

* Alternatively, if you are writing C++ extension you can use Network::ReadFilter, Network::ConnectionCallbacks.

[1] [https://github.com/openresty/lua-nginx-
module#ssl_certificat...](https://github.com/openresty/lua-nginx-
module#ssl_certificate_by_lua_block) [2] [https://github.com/openresty/lua-
resty-core/blob/master/lib/...](https://github.com/openresty/lua-resty-
core/blob/master/lib/ngx/ssl.md)

~~~
sroussey
Wordpress and others use this to load certain on the fly. When you are a
multidomain host this matters a lot.

You don’t just load up a million cents as files and restart the server (though
I do know a company that does something like this, but man, quite brittle).

------
dandare
Is Condoleezza Rice still working for Dropbox?

~~~
teichmann
She’s on the board of directors:
[https://www.dropbox.com/about](https://www.dropbox.com/about).

------
Snelius
And finally we got nginx as legacy now, lul.

------
DarkWiiPlayer
Seems like they could have switched to openresty instead and saved quite a lot
of effort in their migration, but oh well, they probably just couldn't handle
the 1-indexing /s

~~~
anonymoushn
Seems like they were already using openresty. Having used openresty
professionally, I appreciate that it provides ways to write code to solve a
lot of the problems outlined in TFA, but solving the problems out of the box
is significantly better.

------
techntoke
Who does Dropbox compete with these days? They have pretty much the highest
prices for the least amount of value. The only reason I see them mentioned
here frequently is their connection with Y Combinator.

~~~
rmoriz
I've admired Drew and the early Dropbox team for getting things done and
shipped even when compiled python GUIs was edgy as the initial Rails version
of Twitter was. But they shipped and validated the market. Now adding all
those fancy and cool tech mentioned in the blog post will increase the
complexity by a lot but it's not clear what are the real benefits. Does a
decreased number of machines running really justify the migration and addition
of complexity? Maybe they have some new products in the pipeline that built
upon the new stack. Or they waste their time. We will see.

~~~
sanxiyn
> Does a decreased number of machines running really justify the migration and
> addition of complexity?

Of course it does. I am not sure why you think it doesn't.

~~~
rmoriz
servers are a commodity and cheap compared to runtime complexity

~~~
sanxiyn
Yes, but it's a lot of servers.

~~~
rmoriz
How many? 5000 servers? As a public company it should not really matter at
that size. Developers are way more expensive.

