

Ouch - MaysonL
http://www.antipope.org/charlie/blog-static/2010/11/ouch.html

======
RiderOfGiraffes
Please could you make the submitted title a little more informative? Perhaps:

    
    
        Ouch!  (First hand experience of a massive load spike)
    

Thanks.

------
xiongchiamiov
> 4\. Tune Apache now — you won't get a chance to do it when the stampede
> arrives!

Or take the easy route and ditch Apache.

I'm not saying that Apache can't be great (people who know much more than me
use it), but if you need to quickly serve up a ton of requests without
bothering to tune this and tweak that, there are better choices, in particular
for static pages.

Cherokee's my preferred, but lighttpd, Nginx, thttpd, etc. are all good, too.

~~~
mechanical_fish
Ditching Apache is only "easy" if (a) your blog software isn't designed around
Apache-isms like .htaccess and mod_rewrite, and (b) you have time to find and
comprehend the documentation for your alternative server.

Nginx is great, for example, but the docs are primitive compared to those for
Apache, which has been around forever and which has spawned thousands of hints
and snippets all over the web.

EDIT: At this point I should probably say out loud that for accelerating
static pages it might well be simpler to get a reverse proxy, e.g. Varnish,
running in front of Apache than to actually ditch Apache.

~~~
cstross
Let me add: I am paid to write books, not paid to ride herd on servers.
Apache's what I know from days of yore; switching to a new platform means
taking an opportunity cost hit.

Having said that, I'm looking at Varnish (once the stampede dies down) ...

~~~
tomfakes
If you add Varnish, you'll still need Apache.

If you switch to nginx, then you probably won't need varnish at all and can
replace Apache.

The nginx config is much, much, much easier to work with than Apache.
Transferring complex rewrite rules is the hardest thing to get right when
moving over, but everything else is much easier.

Smart lobsters use nginx

~~~
pronoiac
I compared changing the Apache MPM, Varnish & nginx for a high-traffic, low
memory (256M) server.

* Changing Apache would have required changing other packages - PHP wasn't thread-safe.

* Varnish looked fragile. I wish I still had my notes about why I thought that.

* Nginx in front of Apache was relatively quick to setup & really great in practice - restarting an order of magnitude less often.

If the nginx config file would be useful as a starting point, I offer it.

Edit: formatting.

~~~
aaronblohowiak
Why is varnish fragile? It is a bit odd to configure, but that seems like a
set and forget kind of a thing.

~~~
pronoiac
The Varnish configuration is hairy - manually tweaking cookies to cache images
for everybody while avoiding duplication, denormalizing headers so that IE &
Firefox can both get the benefit of compression, dealing with pragma & cache-
control, etc.

(Compression & other varying content was a headache - separately caching &
serving gzip, deflate, & unpacked versions, without any built-in compression.)

Those tweaks looked time-consuming to set up, hard to test, & liable to break
quietly. A buffer in the way that simply & quickly gets a response from (&
then frees up) Apache, then slowly feeds the response to the client, provides
most of the benefit; nginx fills this role well for me.

------
pero
I understand this is HN and all, but has anyone even bothered to take a peak
at the EXTRAORDINARY content that was behind the spike?

In a nutshell, it is a transcript of a British parliamentary session in which
the speaker claims a shadowy yet totally legitimate organization--"Foundation
X"--is willing to donate billions to the British government no strings
attached. It is extremely engrossing.

~~~
zoomzoom
I still don't know if this is to be taken seriously. What exactly could be
going on?

~~~
fredoliveira
Sky news is reporting on it too, and while it looks like it could be a work of
fiction (it sure reads like one - Dan Brown comes to mind), it apparently
isn't.

[http://news.sky.com/skynews/Home/Politics/Foundation-X-
Lord-...](http://news.sky.com/skynews/Home/Politics/Foundation-X-Lord-James-
Of-Blackheath-Tells-Sky-News-Group-Could-Pay-Off-Billions-Of-UK-
Debt/Article/201011115795267)

------
CharlesPal
I've also experienced similar spikes before. It's important to added that
everything will slow way down... that includes SSH access.

Even if you think "I'll just jump in there and correct the issues if/when the
time comes" you may not be able to if SSH access slows down to a crawl.

Like the post says, the best course of action is to be prepared ahead of time
and not trying to make tweaks while the traffic spike is happening.

~~~
patio11
And, for God's sakes, turn off Apache KeepAlive. It kills more blogs than
cancer.

~~~
andfarm
Or switch to nginx, which eats keepalives like candy.

------
mhd
What's the current name for that? Do we still call it "slashdot effect"?

~~~
wazoox
Sometimes it's the reddit effect, or similar. However "slashdotted" should
remain understandable to anyone.

~~~
RyanMcGreal
Though in this particular case, the site seems to have been Gaimaned.

~~~
wiredfool
And boingboing, and reddit x2, here, and a bunch of newspapers in the uk. I
couldn't go anywhere yesterday without tripping over that link.

He got slammed.

If you want a picture of the future, imagine an internet stamping on a single
server— for a day anyway.

------
param
" I don't have an exact breakdown of what just happened, because my logfile
analyser is still churning, from 2am last night.That's because the Apache
logfile today is over 500Mb it's normally about 150Kb. " The post is timed at
11:31.

I think another lesson learned is that the log analyzer needs to be analyzed.

~~~
pronoiac
If the server's under really hard load - memory or I/O or both, with the swap
thrashing - I'd be unsurprised.

I've never checked to see how my stat analyzers fare on half a gig of logs.

