
Wordpress.com serves 70k requests/second using Nginx - diego
http://highscalability.com/blog/2012/9/26/wordpresscom-serves-70000-reqsec-and-over-15-gbitsec-of-traf.html
======
jacques_chester
I've been administering Wordpress blogs for almost a decade now; hosting for
... 5? 6 years? I'm not sure any more.

Anyhow. I started with Apache. Apache probably works fine if you load the mpm-
oh-you-didnt-know-about-mpm-so-sorry module.

My next stop was lighttpd. A fine piece of software, with a persistent bug
that caused it to drop into a 500 state if FastCGI instances were ever
unavailable.

So now I'm on Nginx. And with some fancy footwork in the rewrite rules to
support WP Super Cache, it can serve static files and gzipped HTML straight
off disk.

Unsurprisingly, in this configuration, it runs like the clappers.[1][2]

One thing that always surprises me is seeing configurations where Nginx is
merely a proxy or load balancer into a bunch of Apache instances. Are you
nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later.

[1] At least, it used to do this, until I updated everything and it silently
stopped working the way it used to (which I only discovered while outlining my
current config in comments below). I mean, thanks to memcached and closer
servers and running Percona on a separate server it's _still_ pretty fast, but
this change bugs me and you can bet I'll be fixing it later today after work.

[2] Two optimisations I've dropped are the MySQL query cache and a PHP opcode
cache.

The query cache because I don't think it buys me enough versus using memcached
(just a gut feel, I've got no hard numbers to back me up) and because it often
turns into a contention point in read-heavy MySQL instances. Also, by dropping
it I can free up more memory for InnoDB.

The opcode cache because they tend to be flaky. I've had bad experiences with
all three major ones. If you ever pop Wordpress up on a profiled PHP instance
you find that its time is spent overwhelmingly on waiting for MySQL,
concatenating strings and then streaming it out to the webserver, for which
proper on-disk/in-memory caching is the answer. Time spent loading and
compiling .php is minimal by comparison, so why put up with the hassle?

I don't use varnish because the whole philosophy is to let the OS select what
to cache in memory, and ... well I already do that. Plus getting ESI to work
on frequently updated objects like "recent comment" widgets is a hairy pain in
the arse and I just can't be bothered.

~~~
smcnally
What's Percona's role in this environment?

~~~
jacques_chester
It's slightly nicer than stock MySQL and I can more easily get Percona to help
me if it goes bang (it has far more stats available than stock MySQL, for
instance).

Some of my sites have sufficient write activity that using XtraDB (basically a
slightly souped-up InnoDB) is a smarter option than MyISAM. For searching
people can just use Google, it does a much better job than MySQL's inbuilt
fulltext search.

I don't know if they've fixed it, but AFAIK tables with full text fields used
to cause MyISAM joins to go to disk, even if the full text field isn't
included in the query. As you can imagine, this sucks. Maybe that's been
fixed.

------
gregcmartin
I was one of the core engineers @ layeredtech who managed the servers and HA
for wordpress.com when they launched in 2006.

If I remember correctly we were using DNS round robin and haproxy -> apache ->
mysql all on freebsd systems wow have things come a long ways since then also
it's incredible the sustained growth of Wordpress after all this time. good
memories... congrats Matt on all your success.

~~~
barsxl
WordPress.com hasn't run on FreeBSD since the TextDrive days which was way
before Layered Tech, before me and before WordPress.com was open to the
public. We are 100% Debian today, but have used Ubuntu in the past. We never
really used HAProxy either. I posted about out load balancer choices back in
2008 - <http://barry.wordpress.com/2008/04/28/load-balancer-update/>

------
ck2
Those must be cached pages from dozens of servers on the backend.

There is no way in heck it's realtime queries.

I can tell when authors are in the WP backend on the server just by looking at
the server load because it's crippling.

The wordpress bottleneck is not nginx vs apache vs whatever, it's the problem
of loading hundreds of files for any kind of page render (even to just
authenticate for ajax, etc.) and over a hundred db queries in many cases.

A cache-miss in WP is a terrible, terrible thing.

~~~
piers
Don't forget the version of Wordpress that you download is very different from
the version that's on Wordpress.com

~~~
diego
It's actually not that different, and of course they make as much use of
caching as possible.

So does my self-hosted wp blog, for that matter. If you are using WP Super
Cache correctly, you can deal with significant amounts of traffic. I've seen
peaks of hundreds of simultaneous users on my blog, with a server load < 1\.
The server is a modest aws small instance.

~~~
ck2
Single author mini-blogs without logged in users are super easy to cache.

But multi-author blogs with thousands of logged in users is a nightmare with
wp.

I suspect 90%+ of wp blogs are in the mini-blog category though.

~~~
TillE
Why would multiple authors make a difference?

~~~
ck2
Every time an article is created, saved or published, wp uses hundreds of non-
cachable queries.

Every time an article is published, it causes the cache to be deleted for not
only that article but related pages, which means all those pages have to be
rendered again.

For one author, that can be managed. Many authors, the cache is constantly
being defeated.

~~~
diego
What matters is the ratio of reads per write. Blogs with multiple authors have
a higher ratio than blogs with single authors.

------
stickhandle
Two takeaways - (1) The article is bumf wrt the Wordpress.com setup. NGINX
plays a role, but the thousands of servers for http, db, memcache, load
balancing etc is integral to this discussion (and not mentioned). A typical
reader couldn't relate if it did. (2) Directed at a more typical reader, use
NGINX for hosting your WP.org blogs, company sites etc. __Properly configured
__, your performance to cost ratio will be terrific. Lastly, on a personal
note, it pains me when caching plugins are mentioned as the main part of the
solution. No need for these folks. Use the NGINX cache instead and never worry
again about a buggy/outdated cache plugin (looking at you W3TC) again. Remove
that dependency and you will be happier and possibly better looking.

------
mstank
Article didn't mention that they also use a CDN (EdgeCast). Helps make the 70k
requests/second a lot more bearable.

~~~
barsxl
We serve about 150k req/sec from the CDN. That (obviously) isn't included in
the 70k number from the article. The 70k is only what we are serving from the
origin.

------
misiti3780
"Ability to reconfigure and upgrade NGINX instances on-the-fly, without
dropping user requests."

\- how do you do this ?

~~~
j2labs
SIGHUP

~~~
nodesocket
`service nginx reload` will do the trick as well.

~~~
mfjordvald
Only for reloading the config, not for upgrading the binary.

------
sowhata
And how many requests/sec per CPU?

They have like 300 servers and who knows how many cores.

What kind title is this? The target High Scalability reader must be presumed
to be a complete fool.

~~~
HyprMusic
According to the article they have 2 thousand servers. These are Dual Xeon
5620 4 core CPUs with hyper-threading.

So that averages out to: 70000 / (4 * 2000) = 8.75 requests per second per
CPU.

That seems like quite a low number, I would presume that the load would be
shared amongst less servers than this and the others would be used for
replication/redundancy.

~~~
cowsaysoink
I don't think that is right, it says that automattic has 2000 servers but they
also run things like gravitar, akismet, and vaultpress. Which makes me think
that using that number is completely wrong. (Also these are just the load
balancers not the back end, which could be taking up a large amount of that
2000 server count)

From the article it seems more like they have ~100, maybe less, for the load
balancers which comes out to 175 requests/second per cpu which is getting a
little more reasonable.

~~~
sowhata
If this is just about load balancing, then they could probably get better
performance from haproxy, right? Anyone should be able to exceed 200 req/s
with haproxy. In fact, you should be able to get around double that or more.

The point is the title is useless. It tells you almost nothing. That blog is a
joke.

~~~
cowsaysoink
Well it is average req/s so peak should be much higher, and they probably can
average much more than what they are doing now in case of spikes and other
such things.

But yeah the title is pretty much useless.

------
tsahyt
This figure is _not_ from a single instance though. When running a single
instance of WP I've always found it to be fantastically slow. The greatness in
the described stack really lies in Nginx. It's really not WordPress serving
70k requests here, it's some decent load balancing that achieves the effect.
The title is somewhat misleading.

~~~
ocharles
Yea, I was trying to find some information on exactly how many servers are
responsible for content generation. If this is about 100 machines then yes,
70k/s is pretty damn impressive. If it's 1000 machines, then I'm a little less
impressed.

------
zerop
We dropped nginx because of no native support for session affinity. SSL
support has been introduced in HAProxy very recently.

~~~
Negitivefrags
Is the ip_hash directive not good enough for what your looking for?

~~~
zerop
I was not aware of ip_hash. But as per documentation it uses first 3 octets as
hashing key, this may not work for me.I want to use nginx on LAN. It will
throw all clients to same backend server.

[http://nginx.org/en/docs/http/ngx_http_upstream_module.html#...](http://nginx.org/en/docs/http/ngx_http_upstream_module.html#ip_hash)

~~~
Negitivefrags
It should be fairly trivial to patch your copy of nginx to use the entire IP
address as the hashing key.

------
haight_geek
1 instance of G-WAN serves 500k requests/second with an typical application
using Oracle NoSQL (or Berkeley DB).

