
StackExchange System Architecture - adamnemecek
http://stackexchange.com/performance#
======
y0ghur7_xxx
So they have 185 req/s for all their Stack Exchange sites. Let's say half of
that is on Stack Overflow. That's 93 req/s and they do it with one DB (and one
hot standby) with 384GB of RAM. I don't know many sites that big, so I think
that the majority of people that say "the DB does not scale" work all for
online properties with more that hundreds of millions of users, or are
optimizing for a problem they will never have.

~~~
andrea_s
On the other hand, the StackExchange workflow allows for very aggressive
content caching - they probably wouldn't be able to maintain the same
performances if they needed to generate content on a finer level.

edit: a few additional details - from a few years ago, alas
[http://highscalability.com/blog/2011/3/3/stack-overflow-
arch...](http://highscalability.com/blog/2011/3/3/stack-overflow-architecture-
update-now-at-95-million-page-vi.html)

~~~
y0ghur7_xxx
That DB does 343M queries/day. If the average page requires 10 queries to
render, that's 397 pages/second the db would be able to return data for.

This math is obviously an average, but even if you halve that number you are
still very much above everything most of us web devs are ever going to build
or even see.

~~~
andrea_s
Absolutely! I was not demeaning their expertise, just saying that their
workflow is helpful in reaching such impressive performances with seemingly so
little hardware...

------
samirageb
It's probably worth noting a few things about StackExchange's uniqueness for
those that may not know.

StackExchange runs on physical hardware and they have spent a considerable
amount of time optimizing for bare metal performance. Their team is unique in
that they embraced hardware from the start vs. many teams today that want
hardware abstracted away. They have more maintenance overhead around hardware
management, but don't experience lost IOPs due to IaaS (AWS/Azure) overhead.

Their current environment required overcoming tremendous technical hurdles on
earlier versions of SQL Server (these might be general RDBMS limitations as
well). Luckily they were able to get the Microsoft SQL team engaged to get
them through this.

Finally, their team was world class. BrentOzar, SamSaffron, MarcGravel (and
others) are highly respected members of the SQL & .NET community.

It's easy to look at their setup and say "Wow, that's not a lot." and overlook
the circumstances & talent required to achieve such an efficient system. I'm
not sure many teams would pursue this architecture if they knew the effort
(and luck) involved.

~~~
BrentOzar
> Their current environment required overcoming tremendous technical hurdles
> on earlier versions of SQL Server

Jeff Atwood's goal was always "performance is a feature," and that means
rendering your pages faster than anybody else's. With that in mind, you're
always going to have technical hurdles to overcome on any platform - because
you want to get faster results than anybody else is getting.

I remember when I first got involved, and Jeff told me something along the
lines of, "This slow query runs in ~500ms, and we want it to run in ~50." That
statement alone is a huge leap over what a lot of RDBMS users say - usually
when people refer to a slow query, their unit of measurement is whole seconds,
not milliseconds. They were serious about performance from the get-go.

> Luckily they were able to get the Microsoft SQL team engaged to get them
> through this.

Nothing against Microsoft - I love 'em dearly and make a great living with
their tools - but Stack's success is much more due to their own internal
team's dedication to performance. When we open bug requests with Microsoft,
the time to resolution is typically measured in weeks or months. During that
time, Stack Exchange's team has to come up with creative, impressive
workarounds. They're the sharpest SQL Server users I know.

> Finally, their team was world class. BrentOzar, SamSaffron, MarcGravel (and
> others) are highly respected members of the SQL & .NET community.

Awww, shucks, but I'm not brown-nosing Stack when I say that their current
team is ridiculously good. Their Site Reliability Engineers know more about
SQL Server than most career DBAs I know.

> I'm not sure many teams would pursue this architecture if they knew the
> effort (and luck) involved.

It sounds like you're implying that other architectures are faster by default
and with less effort, and I would disagree there. I haven't seen any big sites
(big being top-100-in-the-world[1] type stuff) where the persistence layer was
set-it-and-forget-it. Scale - especially millisecond performance at scale - is
seriously hard work on all platforms.

[1] [https://www.quantcast.com/top-sites](https://www.quantcast.com/top-sites)

~~~
yuhong
I wonder how many hotfixes MS ended up creating for example.

~~~
Nick-Craver
We've only gotten 2 that I can remember in 4 years here. One was a regression
in .Net 4.0 with high load hashtable inserts (which was previously fixed in
3.5 SP1) that they had to re-fix. The other was a SQL server issue in 2014
CTP1 we were running with an availability groups edge case on high volume/high
latency secondaries.

Unless we're testing beta software, the MS stuff generally works and works
very well. We of course work with them on feature requests that will make life
better - some happen, some don't. I'm on the opinion you should try and have
this relationship with _any_ vendor or open source project. We're trying to
make what we use better, and not for just us.

Nick Craver - Stack Exchange SRE

------
mjibson
Nick Craver has many more details on his blog:
[http://nickcraver.com/blog/](http://nickcraver.com/blog/)

~~~
mjs
In particular, [http://nickcraver.com/blog/2013/11/22/what-it-takes-to-
run-s...](http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-
overflow/)

------
nfm
It always puts me on edge when people choose to write their web-app in a
particular language or framework 'because performance'. The C# VM is by no
means slow but the fact that StackExchange is running off 9 front-end servers
under pretty light load is testament to the fact that for regular web app
workloads, it's just premature optimization to pull out the 'performance'
card.

~~~
saosebastiao
This points me to the exact opposite conclusion. Why wouldn't I want to write
my webapp in an efficient language and framework? If a tiny bit of extra
effort means I can keep my app on a single EC2/RDS pair for a long enough to
last until my first hire, I think that's worth it. It means I don't have to
worry about setting up and tuning and maintaining load balancers, memcached
servers, etc. Sysadmin/Devops work is a large enough distraction from the
product that I would prefer not to have it at all. Efficient software
frameworks/languages give me the breathing room I need to focus on the
product.

~~~
stouset
Any choice of language is going to satisfy your requirement of one app server
prior to your first hire (for most everything not involving huge amounts of
data processing). If your code performs poorly enough that this isn't the
case, the problem isn't the language — it's _you_ , and no language would have
saved you.

That said, there are so many benefits of running multiple servers behind a
firewall off the bat, I can't fathom why people don't do this. It more or less
enables trivial failover for many classes of problem, and deploys are no
longer as large a source of instability: deploy by launching a new box, route
a small amount of traffic to it at first, then route all of it. If there's an
issue, route to the previous app servers.

Compared to the time wasted by getting the architecture wrong off the bat (and
months of pain while you wait for there to be enough time to do the work,
which never happens), this is a massive win.

------
alexchamberlain
Very interesting overview. It shows that the majority of a (Q&A) site can be
run on "just" 9 servers - we're not talking scaling to thousands of servers
here.

I'd be interested to know what DBMS they're running, and it's very interesting
that they've moved the tag "indices" out to their own cluster. I'd also be
interested to know whether they run out of 1 datacentre and how/if they
mitigate that risk.

~~~
adamnemecek
[http://meta.stackexchange.com/questions/10369/which-tools-
an...](http://meta.stackexchange.com/questions/10369/which-tools-and-
technologies-are-used-to-build-the-stack-exchange-network)

~~~
alexchamberlain
TL;DR Microsoft stack and 2 datacentres: NJ and Oregon.

------
noise
Am I the only one that finds it annoying to have each layer's stats in
different units? We have: TB/mo, reqs/sec, queries/day, reqs/min,
searches/day.

------
guiomie
I see their programming stack mentions ASP.Net MVC. Are you guys planning to
move to vNext? Which version of IIS are you guys using? Is F# something also
used?

~~~
Nick-Craver
Yes, we'll move to vNext when it's a bit more stable (likely before RTM).
We're already compiling with Roslyn for localization for example.

------
PhilipA
I remember a podcast with Scott Hansleman when they just used 1 server (an
still served tons of req/s). I think it was this one:
[http://www.hanselman.com/blog/HanselminutesPodcast134StackOv...](http://www.hanselman.com/blog/HanselminutesPodcast134StackOverflowUsesASPNETMVCJeffAtwoodAndHisTechnicalTeam.aspx)
If I recall correctly Hansleman was quite impressed with the performance they
could get from just the one server.

~~~
Maarten88
At the PDC conference 2008 Jeff Atwood RDP'd into the live production
webserver that was Stackoverflow as a guest speaker in the presentation by
Phil Haack on the then-new ASP.NET MVC framework, see
[http://channel9.msdn.com/blogs/pdc2008/pc21](http://channel9.msdn.com/blogs/pdc2008/pc21)
ca 49:40 into the presentation.

------
Kiro
What's a tag engine server?

~~~
ayrx
Looks like a custom written service. I quote from
[http://highscalability.com/blog/2014/7/21/stackoverflow-
upda...](http://highscalability.com/blog/2014/7/21/stackoverflow-
update-560m-pageviews-a-month-25-servers-and-i.html)

The Tag Engine is entirely self-contained, which means not having to depend on
an external service for very, very core functionality. It's a huge in-memory
struct array structure that is optimized for SO use cases and precomputed
results for heavily hit combinations. It's a simple windows service running on
a few boxes working in a redundant team. CPU is about 2-5% almost always.
Three boxes are not needed for load, just redundancy. If all those do fail at
once, the local web servers will load the tag engine in memory and keep on
going.

------
jilljennV
This seems to be an orphaned page. If you click on “team”, “performance”
disappears from the menu.

Apparently, it got leaked :)
[http://meta.stackexchange.com/a/244932](http://meta.stackexchange.com/a/244932)

~~~
mwsherman
It’s not a secret exactly, obvs it has a public URL. It’s quietly been there
for a couple of months.

We kinda wanted to have a 2.0 version of the page with live data before
promoting it. This one is static with #’s we sampled at the time.

------
beck5
Does anyone have more information on the websockets? I assume they all go
through haproxy to their 9 web servers, which is a decent 29k sustained
connections per server.

~~~
icebraining
This might help: [http://brokenhaze.com/blog/2014/03/25/how-stack-exchange-
get...](http://brokenhaze.com/blog/2014/03/25/how-stack-exchange-gets-the-
most-out-of-haproxy/)

------
wuliwong
By no means am I an architecture expert but I've wondered if these statistics
would look much different if stackoverflow had more images and video?

~~~
tinco
They would simply have additional file servers, could be any amount depending
on how many video/images.

They would possibly have to be in separate datacenters though, since shipping
large amounts of data over seas can be inefficient and possibly expensive.

I'm not sure at what point that starts to be a necessity, they're big but it's
still a niche market. And their content is very sparsely visited (i.e. it
wouldn't happen often that 80% of their audience would view a single
question).

------
johansch
Wow 185 req/s :) . Why do you even need more than a single modern cpu? All
that cacheability.

------
jhildings
Doesn't it feel like a bit small environment? I was imaging at least 3x more
servers

------
juliangregorian
It would also be interesting to know how much it costs them in Microsoft
licensing fees.

~~~
BrentOzar
> It would also be interesting to know how much it costs them in Microsoft
> licensing fees.

They started with BizSpark, Microsoft's program for startups that gives you
free licensing for a few years. After that, you pay maintenance on what you've
got, and buy new things as you add new servers.

~~~
bsg75
StackExchange is a well-known-ish case of "startups can run Windows too". I
wonder if they get any leeway on licensing for the PR?

~~~
zamalek
> I wonder if they get any leeway on licensing for the PR?

For production? No. Development machines? They used to.

The best you can hope for these days as a showcase ISV/startup is more direct
access to devs, and you'll probably only get that if you are proofing a new
system of theirs. Remember, the most likely gospel is that Microsoft is
expensive to run - so dropping the licensing cost "for PR" is actually a bad
idea.

~~~
BrentOzar
> > I wonder if they get any leeway on licensing for the PR?

> For production? No. Development machines? They used to.

Development licensing for SQL Server (and most of MS's products) is done
through the MSDN program, and it's dirt cheap, and doesn't require any leeway.
Anybody can buy MSDN licensing.

For production, everything is negotiable. It's not unusual to get special SQL
Server licensing deals when you buy in bulk.

(That's me walking the line between what I can say publicly, and what I
can't.)

------
curiously
how did they get servers with insane amount of ram?

~~~
WatchDog
You can get servers with much more ram than 384GB. Here is a relatively
inexpensive motherboard with 24 DIMMs that will go up to 768GB.
[http://www.newegg.com/Product/Product.aspx?Item=N82E16813151...](http://www.newegg.com/Product/Product.aspx?Item=N82E16813151260)

There are other solutions to take it much further than that.

~~~
curiously
holy crap! so no more need for multiple clusters of servers, just up the ram
and disk space.

~~~
iancarroll
This is of course why there are two options, vertical and horizontal scaling.
Eventually you have to stop upgrading one server and split it up.

