
Managing two million web servers - timf
http://joearms.github.io/2016/03/13/Managing-two-million-webservers.html
======
klibertp
OMG, guys, this is getting really strange. Half of the commenters here read
the word "process" and jumped to their own conclusions, possibly true in
general, but obviously wrong in the case of Erlang.

It bears repeating: Erlang processes _are not_ OS-level processes. Erlang
Virtual Machine, BEAM, runs in a single OS-level process. Erlang processes are
closer to green-threads or Tasklets as known in Stackless Python. They are
extremely lightweight, implicitly scheduled user-space tasks, which share no
memory. Erlang schedules its processes on a pool of OS-level threads for
optimal utilization of CPU cores, but this is an implementation detail. What's
important is that Erlang processes are providing isolation in terms of memory
used and error handling, just like OS-level processes. Conceptually both kinds
of processes are very similar, but their implementations are nothing alike.

~~~
collinmanderson
Thanks. That wasn't really clear from the article.

------
frik
With the same speak, you could say Facebook mangages billions of PHP web
servers, though no one speak like that. (PHP has a shared nothing
architecture; HHVM works simlar to the Erlang VM, if one can say so)

~~~
jontro
Exactly, also I do not understand the comparison with apache. Apache can be
configured to spawn one process per connection. You also could call a thread a
"server" in it's own, so it's just a matter of playing with words.

Also if an apache process dies it will not crash other requests

~~~
signa11
> Also if an apache process dies it will not crash other requests

can you have 2m Apache processes on your machine ?

~~~
jontro
Well that would depend how long you run it for. They don't have to be
simultaneous. It wouldnt make much sense to recycle the process for each
request however.

MaxConnectionsPerChild defines how often the process is recycled

~~~
klibertp
> They don't have to be simultaneous.

Now you're just moving a goalpost. We're talking about Erlang processes - you
_can_ have millions of them simultaneously.

> It wouldnt make much sense to recycle the process for each request however.

It does make a great deal of sense from the fault-tolerance perspective. It
also helps security.

I have an unpleasant feeling that you don't know much about Erlang and its
philosophy. The "one process per connection" architecture has many benefits;
it's a canonical way of getting concurrency and parallelism on Unix systems,
for example. The problem is the way OSes handle and schedule processes, it
simply doesn't work with a very large number of them. Erlang implements its
own kind of processes, which enforce the same constraints os OS-level
processes without taking megs of RAM each and without being pressed after
spawning a few thousand processes.

~~~
jontro
What I'm saying is that it's misleading to call one process a web server when
they really are saying "How we handle 2 million simultaneous web requests".
Also I would be very interested in reading such a writeup. There are many
different approaches to dealing with it (i.e. on thread per connection / one
process / poll based approch). I certainly can see the benefits of having it
in lightweight threads called processes in this context.

~~~
klibertp
> What I'm saying is that it's misleading to call one process a web server
> when they really are saying [...]

No. It's simply a terminology used in Erlang, there's nothing misleading in
it, once you know the definition. I mean, there is very specific thing called
"server" in Erlang:
[http://erlang.org/doc/man/gen_server.html](http://erlang.org/doc/man/gen_server.html)
and also "processes" are described here:
[http://erlang.org/doc/reference_manual/processes.html](http://erlang.org/doc/reference_manual/processes.html)

What's highly misleading is using more general definitions of such words
instead of Erlang-specific meanings. This is a conversation about Erlang and
it makes sense to use the terms accepted in Erlang; you can't blame anyone for
the fact that some of these terms have meanings different than you'd expect.

~~~
Vendan
It's misleading because there are now 2 different definitions of the exact
same word at work. It's a computer running a process running an Erlang VM that
is then running multiple Erlang processes. It is completely reasonable to be
annoyed at that. Many languages have a concept of threading in userspace
rather then kernel space, but they don't just call it threading, they give it
another name so that it can be distinguished from the existing concept. For
example, stackless python's tasklets, Go's goroutines, green-threads.

~~~
klibertp
Well, you can't really expect to hold a meaningful conversation about a
language without knowing, at least, its basic concepts. With Erlang
"processes" are front and center; you read about them in every tutorial and
reference, in most blogposts, basically everywhere Erlang related, ad nauseam.
So you're guaranteed to understand what kind of processes we're talking about
if you have any familiarity with it at all.

And there is a good argument in favor of calling them "processes".
Conceptually, both are providing a very strong separation/isolation and
cleanup guarantees, which is not true for _any_ of other constructs you
mentioned. The range of operation you can perform on a process: spawning,
monitoring, sending signals - is the same for both OS and Erlang processes.
You can very much think and use both kinds in exactly the same way, other than
rather excessive memory usage by the OS-level kind.

I actually have a somewhat related talk tomorrow, about - in essence -
replacing Celery in a Python project with Elixir. Integration via ErlPort
makes it natural to think of Python worker processes as simply (Erlang)
processes - you spawn and link and monitor them just like you would normal
(Erlang) processes. It even works with Poolboy out of the box.

~~~
Vendan
On the other had, the _exact same_ comparisons hold true for threads vs. green
threads. They are extremely similar, they perform almost identically to
threads in many languages, and yet, green threads have their own name, as they
are a different thing. I can understand that Erlang is built around calling
them just "processes", but that's, in my opinion, a bad idea. To me, the fact
that Erlang developers insist on co-opting the name "process" seems arrogant.

~~~
phamilton
Time out. Let's go back to the late 80s, when Erlang was written. It ran on
Ericsson's switches. The platform needed to describe a unit of concurrency
that provided execution and memory isolation. Process fits that definition
perfectly.

Fast forward to today. Look at projects like LING, where the Erlang VM runs
directly on Xen, without a traditional operating system. Processes in Erlang
are 100% analogous to OS processes.

Also look at something like Virtualbox or VMware. Is your process running in a
VM less of a process than the OS process powering the VM itself?

------
stephen_mcd
I really love the idea of explaining the actor model as tons of tiny little
servers compared to a single monolithic server. I tried to make the same
comparison recently when I talked about adding distributed transactions to
CurioDB (Redis clone built with Scala/Akka):
[http://blog.jupo.org/2016/01/28/distributed-transactions-
in-...](http://blog.jupo.org/2016/01/28/distributed-transactions-in-actor-
systems/)

~~~
StreamBright
What was the motivation to rebuild it in Scala?

~~~
stephen_mcd
If each key/value in the DB is treated as distributed (as per the actor
model), a lot of the limitations Redis faces in a distributed environment are
solved. I wrote a lot more about it here:
[http://blog.jupo.org/2015/07/08/curiodb-a-distributed-
persis...](http://blog.jupo.org/2015/07/08/curiodb-a-distributed-persistent-
redis-clone/#curiodb)

------
mpweiher
Beautiful way of putting it. Also very close to Alan Kay's vision of "object
oriented"

"In computer terms, Smalltalk is a recursion on the notion of computer itself.
Instead of dividing “computer stuff” into things each less strong than the
whole – like data structures, procedures, and functions which are the usual
paraphernalia of programming languages – each Smalltalk object is a recursion
on the entire possibilities of the computer. Thus its semantics are a bit like
having thousands and thousands of computer all hooked together by a very fast
network." \-- The Early History of Smalltalk [1]

I also personally like the following: a web package tracker can be seen as a
function that returns the status of a package when given the package id as
argument. It can also be seen as follows: every package has its own website.

I think the latter is vastly simpler/more powerful/scalable.

What's interesting is that both of these views can exist simultaneously, both
on the implementation and on the interface side.

[1]
[http://gagne.homedns.org/~tgagne/contrib/EarlyHistoryST.html](http://gagne.homedns.org/~tgagne/contrib/EarlyHistoryST.html)

~~~
krylon
> Also very close to Alan Kay's vision of "object oriented"

Yes! I remember reading a quote by Alan Key about how pretty much every modern
OOP language gets it right entirely wrong, because OOP is supposed to be - in
Kay's view - about classes or inheritance, but about sending messages to each
other.

If one thinks of processes as objects and sending messages as "method calls",
it is very object-oriented indeed.

~~~
mpweiher
Yep. Except: please don't think of them as "method calls" :-))

"Smalltalk is not only NOT its syntax or the class library, it is not even
about classes. I'm sorry that I long ago coined the term "objects" for this
topic because it gets many people to focus on the lesser idea.

The big idea is "messaging" \-- that is what the kernal of Smalltalk/Squeak is
all about (and it's something that was never quite completed in our Xerox PARC
phase). The Japanese have a small word -- ma -- for "that which is in between"
\-- perhaps the nearest English equivalent is "interstitial". The key in
making great and growable systems is much more to design how its modules
communicate rather than what their internal properties and behaviors should
be." [1]

My contribution towards this is Objective-Smalltalk[2], where I am working on
making _connectors_ user definable. So far, it seems to be working.

[1] [http://lists.squeakfoundation.org/pipermail/squeak-
dev/1998-...](http://lists.squeakfoundation.org/pipermail/squeak-
dev/1998-October/017019.html)

[2] [http://objective.st/](http://objective.st/)

~~~
krylon
Yes, that must have been the one I was thinking of! :)

I was very happy a couple of years back when I was building a toy project in C
and Lua to come across a Lua library called concurrentlua[1] that implemented
Erlang-style message passing (without pattern matching, though, IIRC). It even
allowed you to send messages across machines.

[1]
[https://github.com/lefcha/concurrentlua](https://github.com/lefcha/concurrentlua)

------
andy_ppp
I get the feeling that people reading this and saying "it's just kind of like
a pool of PHP FastCGI instances or Apache worker pools" etc. Do not understand
that Phoenix + Elixir can serve minimal dynamic requests about 20% slower than
nginx can serve static files. This is very very fast.

It also leads to better code due to being functional, lots of amazing
syntactic sugar like the |> operator and the OTP can easily allow you to move
processes (these basically have very little overhead) to different machines as
you wish to scale. Pattern matching and guards are also incredible.

I really do not want to write anything else!

~~~
jerf
And I think Erlang programmers sometimes, if not frequently, forget the Erlang
is not magic, it runs on the same CPUs as everybody else with the same access
to memory protection, same assembler, etc., and consequently, it can't
actually do anything that other many languages can't do too. (And the
languages that can't do it only can't do it because they've somehow locked
themselves out of it.)

Yes, it's a great default to be shared-nothing, and it's great to have a VM
that supports this, the differences in affordances are important and I think
Erlang was a milestone language. Very serious about that. But when it comes
down to it, the _practical_ difference between nginx (as in, a finished piece
of software that exists right now, not the hypothetical space of future C
programs) and an Erlang webserver is not much.

So when the Erlang community tries to rewrite the definition of "server" to be
"an Erlang process", it is not an unreasonable response to point out that
there are plenty of other web servers that have similar levels of isolation
that just happen to be written in other languages, and that we don't run
around saying "Oh, my nginx has two hundred thousand web servers in it!"

This is bad advocacy, and I'd really suggest that Erlangers stop trying to
defend this point. It's not a defensible position. There's no reason to try to
redefine "server" to be "the number of isolated processes", because even if
you do, Erlang will not have some sort of unique claim to being able to run
lots of such processes. Any two "processes" that don't write into each other's
space and can't crash each other are "isolated", even if some implementation
work had to be done to get it that way. And of all things, webservers are
_the_ definitive programs that have had that isolation bashed on and banged on
to the nth degree; I wouldn't be surprised that nginx's isolation is more
tested than Erlang itself's.

~~~
jimbokun
"it can't actually do anything that other many languages can't do too."

Yes, we all are familiar with Turing equivalence, but its an exceedingly
pedantic point when discussing programming languages.

~~~
jerf
"it runs on the same CPUs as everybody else with the same access to memory
protection, same assembler"

I'm not referring to Turing equivalence at all. I'm 100% discussing practical
considerations here.

------
akkartik
This article got me to go figure out precisely what Erlang processes are.
Tl;dr - they aren't OS processes. So it is still conceivable that an error in
Erlang can bring down all your web servers.

[http://stackoverflow.com/questions/2708033/technically-
why-a...](http://stackoverflow.com/questions/2708033/technically-why-are-
processes-in-erlang-more-efficient-than-os-threads)

~~~
ghayes
Well, not if you distribute your erlang processes across several physical
nodes.

~~~
jbardnz
You mean just like you can with Apache or Nginx?

~~~
felixgallo
No. Not at all like those.

~~~
niij
Can you please expand on why?

~~~
vertex-four
The processes can communicate with processes on other servers in exactly the
same way that they can communicate with processes on the same server.

If you have one traditional application server and want to do some form of
cross-user interaction - let's say chat - you can do that trivially, put it in
a queue for that user in a global map. Now when you outgrow that server, you
need to rewrite all your code to understand the concept of users being on
other servers or use an external message queueing system.

In Erlang, all of this is built in by default - if you write code for the OTP
framework (the standard library for dealing with messaging, process
supervision, etc), all you need to do is connect the two servers together and
point them at the same shared user->process mapping process (which you have to
build whether you're dealing with one server or 20, as there's no global data
otherwise).

Of course, if you have absolutely no direct interaction between users, it's
trivial to scale anything - fire up a new server and direct some portion of
your traffic at it. Erlang's trick is to make it that easy even when you do
have direct interaction between users. And of course that works for even
backend workloads - if you have a backend server that your frontend servers
talk to and need to scale it, if you've coded in Erlang and put as much logic
as possible in per-connection processes, you're probably a significant chunk
of the way there.

~~~
ww520
What does any of these have to do with the equivalence between Apache/Ngix and
Erlang when running multiple servers of them to avoid crashing all?

~~~
vertex-four
You can't use traditional servers to scale in lots of cases without a lot of
additional development work. You can use Erlang servers to do so. Therefore,
they're not the same - Erlang covers a broader range of use cases.

------
rodionos
The title is somewhat misleading. I clicked expecting to read how someone is
managing 2 mln web server instances such nginx or apache. I was curious what
kind of company would claim that.

~~~
yelnatz
This is Joe's blog, he co-created Erlang 30 years ago.

So he's not really affiliated with any company, but rather a voice for Erlang
itself.

~~~
rodionos
It's not a clickbait of course, it's just that the term 'web server' and
'manage' made it easy to misinterpret the subject. Typically, by managing a
web server we mean administrative tasks involved in configuring and running a
multi-threaded http daemon. Perhaps "2 million web server processes" would
have been better.

~~~
Vendan
Even that would be confusing, as "Erlang process" != "process" for anyone
that's not an Erlang programmer(or who doesn't parse "process" as the Erlang
variant by default)

------
DougWebb
In the late 90s I implemented the same concept for a web application written
in Perl. (It's still running today.) There were three tiers to it:

Tier 1: a very small master program which ran in one process. It's job was to
open the listening socket and maintain a pool of connection handler processes.

Tier 2: connection handler processes, forked from the master program. When
they started they would load up the web application code, then wait for
connections on the listening socket or for messages from the master process.
They also monitored their own health and would terminate if they thought
something went wrong. (ex: this protected them from memory leaks in the socket
handling code.) When an http connection came in on the socket, they would fork
off a process to handle the request.

Tier 3: request handlers. These processes would handle one http request and
then terminate. When they started, they had a pristine copy of the web
application code (thanks to Copy-On-Write memory sharing of forked processes)
so I knew that there was no old data leaked from previous requests. And since
they were designed to terminate after a single request, error handling was no
problem; those would terminate too. In cases where a process consumed a lot of
memory it would get released to the OS when the process ended. We also had a
separate watchdog process that would kill any request handler that consumed
too much cpu, memory, or was running much longer than our typical response
time.

This scaled up to handling hundreds of concurrent requests per (circa 2005
Solaris) server, and around six million requests per day across a web farm of
4 servers. That was back in 2010; I don't know how much the traffic has grown
since then but I know the company is still running my web app. This was all
_very_ robust; before I left I had gotten the error rate down to a handful of
crashed processes per year in code that was more than one release old.

BTW, while my custom http server code could handle the entire app on its own,
and was used that way in development, for production we normally ran it behind
an Apache server that handled static files and would reverse-proxy the page
requests to the web app server. So those 6 million requests per day were for
the dynamic pages, not all of the static files. That also meant that my web
app didn't have to handle caching or keep-alive, which simplified the design
and makes the one-request-then-die approach more viable.

------
z3t4
I would like to see the code for the chat or presence server. I have a hunch
it will look different depending on the experience of the programmer.

I'm especially interested in how they manage state. Because when you do not
have to manage state, everything becomes easy and scalable. With state I mean
for example a status message for a particular user.

~~~
yelnatz
The Phoenix Framework (web framework for Elixir) already solves this for
you.[1]

It's called Phoenix.Presence. They used a combination of CRDTs with heartbeats
to implement it.

You are right, it is a hard problem because of the distributed nature of
Erlang/Elixir. That's why Chris provided a framework level solution for it.

[1] [https://youtu.be/XJ9ckqCMiKk?t=921](https://youtu.be/XJ9ckqCMiKk?t=921)
(Erlang Factory SF 2016 Keynote Phoenix and Elm – Making the Web Functional)

------
kennydude
> why does the Phoenix Framework outperform Ruby on Rails?

Ruby is known to be a slow language. Most things will easily outperform it

~~~
vemv
Language performance rarely impacts web applications performance.

Actual bottlenecks are the database, and your webserver/architectural choices.

------
smaili
Could someone explain how in the context of the article, "process" differs
from a "thread", in say Java or Python? Or are they one in the same?

~~~
markbnj
All interesting replies, but as I am not an erlang programmer could someone
briefly explain the threading model? Whatever the unit of code packaging is
ultimately it all has to get scheduled onto a core. On what platform is
2,000,000 threads a realistic scenario? If you don't have 2,000,000 threads
you don't have 2,000,000 web servers. You might, for example, have 2000 web
servers each using an event-driven framework to handle 1000 concurrent
requests. Is that closer to the erlang processing model?

~~~
phamilton
A single preemptive event driven framework is probably a closer match.

BEAM has a very efficient multi core fair scheduler. All 2M process share time
on all available cores and are handled by a single scheduler. Because it's
preemptive, starvation problems one might see on an event loop are avoided. A
CPU intensive process can run alongside low latency IO processes and they are
all fairly scheduled.

------
siscia
A little while ago I wrote an extremely short introduction to distributed,
highly scalable, fault tolerant system.

It is marketing material for my consulting activity anyway some of you can
find it interesting.

The PDF is here: [https://github.com/siscia/intro-to-distributed-
system/blob/m...](https://github.com/siscia/intro-to-distributed-
system/blob/master/intro_to_distributed.pdf)

The source code is open, so if you find a better way to describe things feel
free to open an issue or a pull request...

------
rasengan
I can't help but think this is madly in-efficient with cache misses and the
like.

~~~
phamilton
Why?

~~~
mkhpalm
I think what he's trying to say is that silver bullets don't exist. Only pros
and cons.

~~~
phamilton
I'm specifically asking about caching.

~~~
wang_li
Presumably he's talking about processor caches. A shared nothing means that
every time you "context" switch to another "process" you're going to have to
reload all your cache lines in the L1 D-cache.

~~~
phamilton
In Erlang you have a "reduction count budget" of 2000 reductions. This is
fairly low, less than 1ms of execution, but during that time you have
exclusive use of a CPU. At the end of your budget, you might be preempted, or
you might get another window. So you take a bit of a hit to cache, but it's
not like you are infinitely context switching. In practice it works fairly
well.

------
sandra_saltlake
processes do not share memory, but threads may be,

------
yandrypozo
does anybody know how to undo an upvote here in HN ?? This reading was a
terrible waste of time :(

~~~
tvon
You cannot alter your vote.

------
jondubois
You don't need to "crash the server" in response to an error from a single
user - It is sufficient to just close the connection and destroy the session.

I doubt that erlang spawns millions of OS processes because that would be
extremely inefficient due to CPU context switching. So in reality, all erlang
is doing behind the scenes is closing the connection and destroying the
session... It's not actually crashing and restarting any processes... You can
easily implement this behavior with pretty much any modern server engine such
as Node.js and tornado.

~~~
gcr
How is that different than just throwing an uncaught exception? Most web
toolkits will catch that, log it, and 500 the client for you.

~~~
statictype
Your web service is stateless - for those cases it's fine. However with this
model, you are offloading all your state to another layer - likely the
database layer.

With Erlang (or rather, the Actor model), you can build state-full pieces of
code that are also resilient to crashes that cascade across the entire system,
by having a standard pattern of restarting individual actors from a known
clean state through a clean supervisor hierarchy.

