

Stack Overflow Architecture Update - Now At 95 Million Page Views A Month - ZeroMinx
http://highscalability.com/blog/2011/3/3/stack-overflow-architecture-update-now-at-95-million-page-vi.html

======
carson
For those who aren't reading it the ServerFault blog
<http://blog.serverfault.com/> is a treasure trove of information. The work
they are doing and blogging about on the systems side is something a lot of
people take for granted.

------
moe
Hm. I love Stack Overflow but architecture only starts to become interesting
at 4-5 times their traffic.

95M/mo translates to a mere 36/sec avg. A single web- and db-server will
handle that without breaking a sweat, although you of course want some more
machines for peaks, redundancy and comfort.

~~~
citricsquid
36/sec but then they're doing 800 http requests per second, not sure what
could cause such a big difference (lots of ajax stuff?) but I think 800/sec is
an interesting thing to see them deal with, that's something in the region of
1bn a month, which fits your requirement for interesting-ness :p

~~~
moe
_which fits your requirement for interesting-ness_

Sorry, the threshold of interestingness for serving static assets is even
higher. ;)

A single nginx on a moderate host will barely warm up before ~10k reqs/sec.
The network tends to be the bottleneck here. Anyways, ~800/s should be doable
from a $10 VPS.

~~~
dillona
I don't think they're paying all this money for the fun of it

~~~
moe
Hm, I'm not sure I follow. What money do you mean?

~~~
alextgordon
They have _10_ web servers. If they could replace all that with a VPS or two,
they would.

~~~
moe
According to the article only 3 are dedicated to StackOverflow which sounds
about right to me.

------
rabc
Really cool to see how they moved from Windows-only to other technologies like
Redis.

They dind't jump into the hype at first version, they started using other
things just now when they really needed that.

------
razvan346
T9space.com up to 42 Million pageviews a month: <http://i.imgur.com/Vggvl.png>
10 App servers dynamically converting websites to mobile versions + 1 MySQL DB
+ Haproxy LB + 1 Lucene Search

------
citricsquid
I use stackoverflow as the benchmark for success on my own* sites, we're still
winning. We did 145m for February!

~~~
spoiledtechie
whats your website?

~~~
citricsquid
It's not even related, I feel bad now because it's irrelevant to the post, it
just happens that the first time Stackoverflow announced their traffic I
mentioned ours and Joel (I think?) said something about it, so now I always
use them as a benchmark. Anyway, minecraftforum.net and minecraftwiki.net, I
don't own them any more (reason for the * -- curse.com acquired them).
<http://www.quantcast.com/minecraftwiki.net> &
<http://www.quantcast.com/minecraftforum.net> combined 145m for feb

------
ralphc
I understand the tradeoffs between Windows and Linux and choosing one or the
other for a particular task, buy why run Ubuntu Server on one machine and
CentOS on others? Are the tools that Linux specific that it's worth keeping
track of two versions, two package managers, etc?

------
keyle
I question the truth in that article and whether they really use Razor
already. It's barely out, so is MVC3 and I haven't noticed any changes that
would suggest that they would have converted it all from MVC2 to MVC3. It's a
major task.

They might have some Razor in there because it is backwards compatible but to
have it all razor is a stretch.

~~~
Jamiecon
We're using ASP.NET 4 and MVC 3 in production. New features in both made it
very worthwhile to invest the time in conversion.

We are not using Razor yet, but it's feasible they are. Take a look at this:
<https://github.com/telerik/razor-converter>

------
bluesnowmonkey
> All pages accessed by (and subsequently served to) annonymous users are
> cached via Output Caching.

I'm surprised they don't cache output on logged-in pages as well, using ESI or
some kind of client-side replacement. There's not that much personalization
going on. Most page content is the same between users.

~~~
Jamiecon
I would imagine that they calculated it would be cheaper to scale up their
application servers and backend caching than pay for the development resource
to facilitate this change.

------
btipling
Interesting on the switch from the Windows/MS stack. Is there not a decent No
SQL option for a windows stack?

~~~
michaelbuckbee
I'm not sure what switch you are referring to, they've been using HAProxy and
Lucene for quite some time.

Also, what would a No SQL data store give them (for central data storage - not
caching) that they aren't already handling with SQL? Keep in mind that they
also have relatively easily partionable data (if necessary they can break off
any StackExchange site and database to their own dedicated hardware).

~~~
mattbriggs
redis is nosql, and they are using it on linux

------
nopal
How did the code have to change to read from/write to Redis?

~~~
gecko
I doubt very much. They're basically using Redis as a much more powerful
memcached. SQL Server still is the ultimate owner of all StackOverflow data,
last time I talked to the team.

~~~
nopal
I guess I just haven't seen memcached integration in .NET. All of the caching
examples I've seen use the framework caching components.

I was also wondering if they'd developed any generic custom code to fetch from
Redis or fetch from SQL/insert into Redis, and how it fits into their
application logic. (e.g. does it happen before they make a LINQ to SQL query,
or does it intercept LINQ to SQL queries and return results from Redis).

~~~
mythz
If you're interested, here is a C# Redis Client:
<https://github.com/ServiceStack/ServiceStack.Redis>

With downloads for windows: <https://github.com/dmajkic/redis/downloads>

Here's an example of an OSS 1-page mini-StackOverflow, written only using
Redis: <http://www.servicestack.net/RedisStackOverflow/>

And this page has caching web services examples (at the bottom), with source
code of non-non-invasively dropping in a Redis cache provider using an IOC.

<http://www.servicestack.net/ServiceStack.Northwind/>

If you want to learn more about Redis, here is an article on how to build an
application using only POCO's and Redis:
[https://github.com/ServiceStack/ServiceStack.Redis/wiki/Desi...](https://github.com/ServiceStack/ServiceStack.Redis/wiki/DesigningNoSqlDatabase)

With more info on the wiki:
[https://github.com/ServiceStack/ServiceStack.Redis/wiki/Desi...](https://github.com/ServiceStack/ServiceStack.Redis/wiki/DesigningNoSqlDatabase)

~~~
nopal
Great, thanks!!

------
grandalf
It's fascinating that a MS loyalist like Joel would end up using so much
linux/oss software.

I wonder if Joel regrets not going 100% open source. I guess the cost of
Microsoft licensing for all that is still very cheap if viewed as the price of
being featured in Bing searches.

~~~
spolsky
This might possibly be the most ridiculous comment I've ever seen on the
internet, let alone Hacker News.

* I am not a Microsoft Loyalist

* I don't make the technical decisions. Why would I? I'm the person who knows the least

* It is not necessary to use Microsoft licensing to be featured in Bing searches... that is possibly the most preposterous thing I have ever heard in my life

* Microsoft licensing is a rounding error, with the possible exception of SQL Server licenses

* Your bizarre personification of Stack Overflow as "Joel" is charming and quaint and completely obnoxious to the 20-odd people at Stack Overflow that actually do the work instead of sitting around browsing hacker news like I do

~~~
keithwarren
That you would reply to this - makes me respect you even more

~~~
grandalf
And I you.

