
250x Speed Improvements with Microcaching (and No New Code) - taybenlor
http://fennb.com/microcaching-speed-your-app-up-250x-with-no-n
======
patio11
A national radio campaign is likely to bring hundreds or thousands of visitors
spread over hours, not hundreds of thousands of visitors spread over seconds,
so I probably would not take any particular action to harden a site in
anticipation of it. It is a poor use of engineering resources and adds
technical risk with no corresponding benefit. (n.b. Pasting code you got from
a blog post, particularly code marked as _kinda broken_ , is not a risk-free
endeavor! I love nginx, don't get me wrong, but paste in snippets from two
different blog posts and _watch the sparks fly_ if you don't understand how
nginx handles, e.g., location priority.)

~~~
adgar
Not to mention that he's compromising on accuracy. Yeah, 1 second is tiny, but
why give it up if you don't have to? Especially those of you reading who want
to try this on a potentially more dynamic site.

Edit: downvoted with no explanation, cute. And readers here think they're
above Reddit standards.

There's no reason to compromise the integrity of a system to _any_ extent
before you have to. As others have demonstrated, that need was clearly not met
here.

~~~
gruseom
_downvoted with no explanation, cute. And readers here think they're above
Reddit standards._

Downvoting with no explanation is arguably acceptable here. There's
disagreement about it, but people do it all the time, including me when I
don't have time to type in a comment or feel like doing so would add more
noise than signal (like, probably, right now). Complaining about being
downvoted, though, definitely is not.

~~~
adgar
How can explaining why you downvoted, when nobody else has, add more noise
than signal, unless your opinion is not worth sharing in the first place? And
if your opinion is not worth sharing, why is it worth a downvote?

Let's be honest - we downvote here because we disagree, just like very other
site.

~~~
gruseom
That's a good enough question that you've provoked me into figuring out what I
think about this.

Adding words takes up space. A downvote doesn't. Therefore a downvote is more
lightweight, therefore there are times when it's more appropriate than a
comment. It's an in-between gesture, partway between silence and
verbalization. I like that subtlety. It's an especially good way to express
disagreement with something one feels is not only wrong but also somehow
debasing. A comment doesn't have to be rude to be debasing, it can just be
mediocre or somehow crass, nor is one always able to say exactly how. Those
are cases where fostering more discussion is unlikely to do good and it's
better to hold one's tongue. But downvoting still gives a signal to the
original commenter as well as to other readers that someone's not ok with what
was said. I think that's meaningful, and also that it's fairly rare to see
people abusing that signal. It does happen, but more often when someone thinks
they don't deserve it, it's kind of obvious to others that really they do. So
when it happens, the best way to respond is with a touch of self-honesty.
Maybe you'll come out roses but just maybe you'll notice something worth
correcting.

One other thing about downvoting. I think what makes it controversial, and
also interesting, is that it's an emotional expression. (Upvotes are too, but
downvotes are stronger.) That explains why people get upset about it, have
strong opinions and so on. But it also explains why it's a good thing to have
around. There are very few emotional channels available to us that don't
require proximity. It also explains why it's wrong to say that downvotes are
inferior and should be replaced by comments. Imagine if people were forced to
put every emotional gesture into words. It would be impossibly ponderous. And
we'd end up talking about nothing else.

------
wheels
> _This is fine, up until the point where you get on HN and Reddit at the same
> time_

Incidentally, you don't actually need much to handle that. Our web server is a
wimpy 256 MB VPS and we've had (Wordpress) blog entries hit the front page of
HN and Reddit simultaneously and weather the storm without missing a beat. An
appropriately setup Apache + Wordpress SuperCache does the trick just fine.
(Hint: The default Apache configuration isn't "appropriate".) You're not going
to hit anywhere even close to 2k requests per second on the front page of
those two.

~~~
timdorr
It depends on the content. If it's something that will be visited just by
HN/Reddit users: Sure, you're not going to get 2k/sec directly from them. But
if it's something that's going to get
shared/tweeted/retweeted/blogged/reblogged/etc., then you will be looking at
huge traffic numbers. It just depends on how viral your content is and how far
it spreads. It's the long tail that matters.

~~~
wheels
You've _really_ got to hit the jackpot to get up to 2000/request per second
(assuming you don't already have a high traffic site). Assuming about 20
requests per pageview, that's more than a quarter million pageviews per hour.

I've had things go moderately viral (thousands of tweets / retweets) and you
don't get anywhere near that. Pre-tuning for hitting the jackpot is in most
cases going to be premature optimization.

~~~
asto
Premature optimisation is premature until it's not. Caching is an easy boost
and is among the first things I enable on a wordpress or drupal install.

------
jbyers
Nice technique. The number of distinct dynamic pages you expect to get
hammered still must be regenerated within that second. With a longer window,
some wp-admin or logged-in-user detection, and a third-party comment service,
I could see this being a standard nginx wordpress configuration.

~~~
mhd
What comment systems are you thinking about? With Disqus the rest of the page
shouldn't really matter at all, whether it's dynamic, cached, static etc…

Depending on one's error tolerance, maybe dynamic configuration would help
(check if you're hit heavily and/or check if you're on HN/slashdot/reddit,
then modify the config accordingly).

~~~
IanMikutel
What about the Facebook comment system?

~~~
nl
Same thing. Any Javascript-based comment system should be ok (because the
comments are all served from a 3rd party site).

It is when you are using internal wordpress comments you need to be careful.

------
mopoke
I like this solution and am definitely tempted to give it a go.

Anyone got any thoughts on the best way to do this on a page with
personalisation? (and this is really simple personalisation - one section of
the page changes depending on whether you're logged in or not).

My solution would probably be to have the personalised section load as an
async request but then you'd need to make sure that the async request can
handle the same load as the microcached content.

Any other ideas?

~~~
_phred
You can put any nginx variable into the proxy_cache_key setting, including
cookies and query strings. I've used this to cache localized versions of each
page on a site based off of a "language" cookie. Async is a good approach for
what you need, but it's good to know what nginx's caching mechanism can do.

------
zzzeek
Microcaching at the page level is of course a great idea for a dynamic app,
but only works if the content being served is identical for all users - in
which case why not just use static pages. Oh right, because we only know how
to use Wordpress. Ditching wordpress for a static generator should be the
preferred route, if possible. (use disqus or similar for comments).

The vast universe of truly dynamic apps that we write in Rails or Python or
whatever usually have page elements that are specific to the user's session -
"Welcome John Smith" and all that (edit: oh i see he mentioned that at the
bottom). So page-level caching isn't feasible there, unless like in the case
of disqus you're using javascript to inject personalized content from another
server. But for a really interactive web application where coarse grained
solutions like this aren't feasible, I'm still a proponent of page-component
level caching, something you normally do in your app layer, not the web server
layer.

------
michaelbuckbee
This seems pretty similar to using Varnish or Squid as a reverse proxy (though
likely easier to setup).

~~~
taybenlor
Similar sort of approach. A lot less work to set up, especially if you're
already running nginx.

------
simonw
I really like the idea of setting a cookie that bypasses caching for a few
seconds - I've heard the same technique used by Facebook, who set a cookie
that ties you to the MySQL master server rather than the slave after you
perform a write action so that you'll see your update without waiting for
replication lag.

It relies on using the Max-Age Cookie argument though, and I was under the
impression that IE doesn't implement that correctly. Anyone know what the
status of IE and max-age cookies is?

~~~
Fenn
Hey Simon,

Hmmm, you're right about the IE issues (<http://mrcoles.com/blog/cookies-max-
age-vs-expires/>). I haven't looked into how possible/hard expires would be in
nginx config, but you could probably hack it in with
<http://wiki.nginx.org/HttpUserIdModule>

~~~
simonw
The problem with the expires argument is that you have to set it to a specific
date, but I don't think you can be sure what timezone / clock setting the
user's browser has - which means that it's pretty much impossible to reliably
set a cookie that expires in a couple of seconds time.

Hopefully I'm wrong - I'd love to know if there's a workaround for that issue.

------
todsul
The big issue here is the application type. This only works well for a very
particular type of application. That is, a highly dynamic site that is NOT
dependent on user logins.

1) If the site is only moderately dynamic, you can just use plain Nginx and
set fastcgi_cache to a few minutes or hours. Much less load on the server. I
like to keep things simple, I wouldn't even bother with Apache. Porting
rewrites to Nginx is super simple.

2) If the site is customised on a per-user basis, 'microcaching' will break
the site and have disastrous consequences. Every user will see the system
customised for whichever user primed the cache.

My primary website is user based. That means this 'microcaching' concept wont
work at all. It would be catastrophic.

That's where Varnish comes in with ESI. I really wish I didn't have to use
Varnish. It's slower than Nginx, it adds another layer of complexity, and in
testing, it seems slightly flaky. But what Varnish+ESI allows is caching of
parts of my page that aren't user specific. I.e. header, footer, etc.

If you want to see my test results of Nginx vs Varnish, see
<http://todsul.com/nginx-varnish>

~~~
chlee
I thought about incorporating Varnish into my side-project as well. However,
Varnish seems to be a [potential] hassle when you account for VCL, ESI, and
etc. It also adds an extra layer or indirection and complexity as you've
mentioned.

With regards to caching user specific parts of your page, have you considered
template caching [1, 2]? This way you can apply different caching policies to
different parts of your pages w/o using varnish and ESI.

With that said, why would one choose Varnish + ESI (or reverse proxy in other
words) over template caching + memcached/redis/riak? Can someone explain the
pro's and con's of both approaches?

[1] <http://www.makotemplates.org/docs/caching.html> [2]
<http://jinja.pocoo.org/docs/extensions/>

~~~
todsul
Yes, we previously used template caching because it was built into Symfony1
(an MVC PHP framework). Symfony2 now leaves caching to the HTTP Standard, plus
integrates Akamai's ESI. Symfony specifically recommends Varnish for live
projects.

Memcache certainly has its place, but for me, Varnish is a plug-n-play
solution that doesn't require code changes. Even with ESI, if you've already
set HTTP Cache, you just need to set s-maxage per component. That's it. Then
of course there's all the Varnish load balancing features that I haven't used.

If my site didn't use logins, I'd definitely just go for Nginx +
fastcgi_cache. People argue that Apache can do the same with the right tuning.
But there's something to be said for a 5 minute VPS build by a complete novice
that can serve 20,000 requests per second using about 10 lines of config code.

I think this all becomes much more complicated for seriously large websites.
That's where memcache/redis et al. join the party, potentially with Varnish.

------
jbarham
Given that I'm currently personally setting up a caching cluster to host
www.melbournecup.com (aka "The Race That Stops a Nation"), being able to
gracefully handle a huge spike in traffic is something that is very much on my
mind! :)

The site itself is developed in Django and so far I'm just planning on putting
a bunch of Varnish caches (behind a load balancer) in front of the Django
server. I'm using the very nice Django Varnish app
(<https://github.com/justquick/django-varnish>) in the Django instance to
automatically purge pages from the cache as they're updated.

I'm deliberately trying to keep the setup as simple as possible, but the goal
is to have a fast site and fresh content.

Tips from others who have handled similar traffic loads would be very welcome!

------
splitrocket
I've been doing essentially the same thing with my wordpress install. Went
from a few hundred reqs/second on a cheap linode to over 4k reqs/second. (I
think the limit was the benchmarking tool, not nginx) I've got the nginx
config if anyone is interested.

~~~
splitrocket
Here it is: [http://alchemi.st/nginx-wordpress-network-and-fastcgi-
cache-...](http://alchemi.st/nginx-wordpress-network-and-fastcgi-cache-the-
ultimate-guide/)

You can change the caching ttl to meet your needs.

------
brown9-2
Is it fair to compare those ab benchmarks when the concurrency value is
different for each (4 and 500)?

Would be curious to see how the original config handled 500 concurrent
requests.

~~~
splitrocket
My experience doing something very similar is that the original config would
not handle 500 concurrent requests. It's a fair comparison insofar as the
uncached app probably couldn't serve that many concurrent requests in a
reasonable amount of time.

------
eli
I think I'd rather just install Varnish than try to reinvent it in my ngnix
config. As a bonus you'll get faster serving of static assets too.

------
ck2
The only way to make wordpress fast and responsive is to bypass wordpress
entirely.

You don't need to do anything complex - just install wp-super-cache, set a
long timeout, and most importantly add the .htaccess rules it generates to
bypass wordpress entirely and serve the cached static files directly.

------
mike-cardwell
The few times I've been on the front page of Slashdot it has eclipsed the
traffic that I've had from being on the front page of Reddit. Hacker News
barely causes a blip.

------
crikli
Is this using nginx in lieu of Apache or nginx in front of apache, acting as a
reverse proxy or similar?

------
bbrizzi
Anyone know of an equivalent in Apache?

------
rymedia
Very impressive.

------
nirvana
"If you have personalized pages (ie: majority logged-in users) this approach
isn't going to work. "

I've considered this problem, and am working on a solution for nirvana[1]. The
biggest challenge to this project has been to take a language (coffeescript)
that is sequential and run it in a distributed environment, without the
programmer having to know distributed programming. One of the techniques I'm
applying is making a response (in this case, a web page) the result of a
collection of components, which are rendered separately in the same context.
(EG: The context is the headers of the request, plus the user record if the
user is logged in, etc.)

So, the request comes in, the components are loaded from the cache, they are
executed (in parallel) all with a copy of the state, their results are
aggregated and that result can run thru templating to produce a webpage that
is returned.

The idea then becomes, instead of executing the code for every component in
every request, if the component has no context specific requirements (e.g.: it
is the same for every user, it's a static element, or it's dynamic, and but
doesn't need to be generated every time) .. then it can be flagged as
cacheable. The caching would also have a staleness factor (Eg: 1m, 5m 10m).[2]

My hope is that you can have pages that are custom per user, but that also
contain heavy impact results (say a graph produced by an expensive operation),
where the results come form cache, the static components come from cache, but
the user specific parts are dynamically generated each request.

This component approach not only lets the code be rendered in parallel, and
often not even rendered, but instead pulled from cache, but it should allow
for more convenient re-use of common elements and features across a site.

I hadn't considered caching for just 1s, though. Will have to think about
that.

[1] Nirvana is CoffeeScript web development backed by erlang and Riak.
Instantly distributed coffeescript. It will be open source, hopefully soon.
Follow @nirvanacore on Twitter if you're interested in being notified.

[2] Planned. There are some implications of this that will require tradeoffs,
so initially it may just be a flag of Yes/No for "Cache for up to 1 minute."
or some value like that.

~~~
buro9
Varnish supports ESI. What you've described is ESI.

Just put Varnish in front of your servers and used the Expires headers of the
responses to cache stuff accordingly. If you include user information in the
cache hash of the personalised chunks of the page (i.e. in the URL or cookie)
then you can cache that for a short time too.

I do apply short term caching to a lot of things, mostly to protect against
double requests for the same thing when there might be an expensive query
behind it.

~~~
Cherian
I believe nginx has something similar called SSI

<http://wiki.nginx.org/HttpSsiModule>

~~~
buro9
Everything has had SSI. In fact the way that the BBC used to make their web
site in PERL on Apache used SSI for their "Most Read" section.

ESI is different... Edge Side rather than Server Side.

The benefit is that you don't need one server capable of doing everything. You
can have a separate web service capable of returning the "Most Read" section
on different technology, different machines, elsewhere on the network... and
ESI will interleave the output of these web services at the edge of the
network.

Varnish supports ESI, and as one of the main reasons to use Varnish is caching
you can reasonably expect (and be correct) that Varnish will cache ESI
sections.

So Varnish will receive a document, cache the whole of it, and use an entirely
different cache policy on a section of it.

It's all swings and roundabouts. I just have a strong preference towards not
making any part of the solution complex by trying to make it do anything more
than it need do, especially when other layers can be added that do that
specific thing (caching in this instance) better.

