Well, server optimization has been my primary hobby for the past month, and I'd like to make some clarifications about a few things the article says.
Saying that Lighttpd supports "CGI scripts, Ruby on Rails, Pylons, TurboGears, Django, Quixote, webpy, Mongrel, Catalyst, Mason, Joomla, AWStats, and so much more" is kind of misleading. Lighttpd can do a few things: it does fast cgi (fcgi support explains about 75% of the examples in that list), it can serve static files, and it can proxy requests to other servers. Lighttpd doesn't "support Mongrel", because Mongrel is another server. What Lighttpd does is use something akin to mod_proxy to send certain requests somewhere else, that somewhere else might be a Mongrel cluster that is running on port 8080 (hence it "supports" Mongrel).
That proxying service is where the majority of Lighttpd usage has been. A typical efficient setup these days (YouTube does along these lines) is to have lighttpd accept incoming requests and if the request is for a static file then it serves the request (because it is very lightweight (i.e. minimal) it is more efficient than Apache at serving static media), and if it is for dynamic content the request is passed to Apache, which then passes it back to Lighttpd, which passes it back to the requester (this means that "heavy" Apache threads will be dedicated to a request for a shorter period of time since it only has to communicate/wait for Lighttpd instead of the actual requester).
Lighttpd on its own isn't necessarily the best solution (an example from my experience is that mod_python and Apache become more efficient at serving dynamic content than fcgi and Lighttpd as the numbers of requester per second get very high). Usually a combination of a light frontend server and Apache in the backend can provide the best performance (again this is at high numbers, at lower numbers lighttpd is probablly equally or more efficient).
As a final note, I'd really recommend Nginx over Lighttpd if you are looking for a light http server to serve static content, support fcgi, and/or proxy to other servers. The biggest reason? Lighttpd leaks memory and has to be restarted occasionally because every couple of days (under heavy usage) it dies. Not a great feature in a production server.
The big win of lighttpd/nginx is that they're single-process [1] multiplexing servers. Apache spawns dozens (or hundreds) of processes/threads and relies on the OS to distribute resources. This critical difference is why they're so much more efficient and generally faster. The epoll mechanism in Linux has widened the gap even further.
Brad Fitzpatrick created a software load balancer in Perl that handles all the requests for LiveJournal based on this principle.[2]
1. A small number of processes may be used to take advantage of multi-processor machines.
It seems every 6 months or so, one of these single process, select based servers comes out as the "apache killer", only to fall prey to a later successor. Anyone else remember boa?
That's not really true at all. BOA did almost nothing but serve static files and there was no consistent development. Same is true of thttpd and all the rest.
Lighty and Nginx are becoming full-featured multiplexing servers. It's a lot of work to add features when all the code must be super reliable and non-blocking. No serious open source effort has ever been made. Zeus has been the only option for years, but it's commercial and expensive. They've made millions filling this void.
Saying that Lighttpd supports "CGI scripts, Ruby on Rails, Pylons, TurboGears, Django, Quixote, webpy, Mongrel, Catalyst, Mason, Joomla, AWStats, and so much more" is kind of misleading. Lighttpd can do a few things: it does fast cgi (fcgi support explains about 75% of the examples in that list), it can serve static files, and it can proxy requests to other servers. Lighttpd doesn't "support Mongrel", because Mongrel is another server. What Lighttpd does is use something akin to mod_proxy to send certain requests somewhere else, that somewhere else might be a Mongrel cluster that is running on port 8080 (hence it "supports" Mongrel).
That proxying service is where the majority of Lighttpd usage has been. A typical efficient setup these days (YouTube does along these lines) is to have lighttpd accept incoming requests and if the request is for a static file then it serves the request (because it is very lightweight (i.e. minimal) it is more efficient than Apache at serving static media), and if it is for dynamic content the request is passed to Apache, which then passes it back to Lighttpd, which passes it back to the requester (this means that "heavy" Apache threads will be dedicated to a request for a shorter period of time since it only has to communicate/wait for Lighttpd instead of the actual requester).
Lighttpd on its own isn't necessarily the best solution (an example from my experience is that mod_python and Apache become more efficient at serving dynamic content than fcgi and Lighttpd as the numbers of requester per second get very high). Usually a combination of a light frontend server and Apache in the backend can provide the best performance (again this is at high numbers, at lower numbers lighttpd is probablly equally or more efficient).
As a final note, I'd really recommend Nginx over Lighttpd if you are looking for a light http server to serve static content, support fcgi, and/or proxy to other servers. The biggest reason? Lighttpd leaks memory and has to be restarted occasionally because every couple of days (under heavy usage) it dies. Not a great feature in a production server.