
StackExchange Performance Stats - ksec
https://stackexchange.com/performance
======
f6v
I’m underwhelmed by the absence of 139 micro services(written by 11 one-pizza
teams), each with its own database instance.

~~~
qes
I serve near StackOverflow levels of traffic. Similar # of page views and more
than 5x the egress bandwidth. Also .Net.

Also running a traditional monolith style system. One main SQL DB. Another for
some logs. Redis. Blob storage. KISS.

We have various miscellany and a dozen+ web apps for public website, internal
admin, content editing, monitoring, on-the-fly image resizing, etc. but even
the API's serving those live in one web app, and our high traffic data &
content service is just one app. Nearly everything in one solution and repo.

Tbf, we've been utilizing chromium more over the past year and I think we're
going to have to split that out into its own service.

We do a lot of data-driven image generation and used a couple .Net imaging
libraries for that until last year. Since then we've been building new data-
driven content solely in HTML/JS and screenshotting through chromium to
generate static images (we serve the same content in animated HTML and JPG and
used to build each separately).

------
ed_elliott_asc
Stack overflow is awesome

Stack overflow runs on .net

Stack overflow squeezes lots of performance out of very little hardware or
cost

Stack overflow doesn’t do that much

Customers haven’t added feature after feature until it resembles a half dog
half cow hybrid

Stack overview developers spend time working out how to scale for their
specific use case rather that theoretical use cases

I honestly believe that by sticking to a singular narrow focus stack overflow
has achieve magnificent things. It isn’t because of .net it is in spite of it.

Well done stack overflow but unless you can stop the business or the users
with extra requests or expanding the product you cannot replicate it.

------
matt2000
I love to read about how StackExchange does things, it's such a stark contrast
to the cloud maximalists. I'm not saying there aren't advantages to each camp,
it's just nice to have a solid example on the "vertical scaling can actually
get you pretty damn far" side of things.

~~~
ed_elliott_asc
One day stack overflow will be offline for weeks and then people will have an
answer to this.

~~~
zamadatix
It's 20 servers, many of which are hot spares. What about self hosting that is
going to take weeks and why hasn't it happened in the last decade?

------
kburman
I have worked in a project which required double infrastructure of what SO is
using and render page at > 200ms. Our db size was 1/4 of stackoverflow but we
don't even have 0.1% of content was stackoverflow offers.

~~~
GordonS
What stack were you using?

The only time I've personally seen render times that high is with large,
complex databases in bad need of indexes and/or restructuring, and with hybrid
cloud stuff, where cloud services are using an on-prem API or database.

~~~
f6v
Do live queries for frequently accessed data without caching and you can go
into seconds to render a page. I once came across a Postgres full-text query
without an index. The page took a minute to render, I kid you not. And that
page was arguably crucial to internal operations.

------
jmaygarden
There’s really no need for all of the speculation in this thread. Jeff Atwood
documented the StackOverflow development process as it happened on his blog.
Here’s the first announcement from 2008:

[https://blog.codinghorror.com/introducing-stackoverflow-
com/](https://blog.codinghorror.com/introducing-stackoverflow-com/)

------
atdt
At first I was surprised that they don't use edge caching: it seems that all
requests get proxied to a backend server, which use just two redis servers as
a shared cache. A global internet site can't get good performance this way:
RTT between NYC and Sydney is >200ms, so even if your application is lightning
fast, it's going to feel slow to users in many parts of the world.

Digging around a bit, it seems that they do use Fastly for edge CDN:
[https://nickcraver.com/blog/2017/05/22/https-on-stack-
overfl...](https://nickcraver.com/blog/2017/05/22/https-on-stack-overflow/).

Still, it's an impressively lean stack.

~~~
EE84M3i
Potentially they can't get much edge cache offload for HTML/API content
because many people are logged in and/or they do page customization even for
non-logged in sessions and pushing logic to the edge isn't worth it.

~~~
atdt
Probably you're right!

A webpagetest from Sydney shows pretty dismal performance:
[https://www.webpagetest.org/result/200801_FJ_dd5f2a76d77bcaa...](https://www.webpagetest.org/result/200801_FJ_dd5f2a76d77bcaaac4ee45d5967ef4e7/)

~2 _seconds_ for first contentful paint. That's pretty dismal, IMO, and shows
the limitation of their current setup.

~~~
ksec
>That's pretty dismal, IMO, and shows the limitation of their current setup.

Well they cant beat the speed of light. What could they do ?

~~~
lixtra
Cache their content on different continents and live with only eventual
consistency. But as long as there is no competition, why should they
complicate things.

------
BossingAround
Wow, it's a bit surprising that they use C# under the hood. I wonder what was
the cause for that decision; back when I started learning Java, C# was said to
be the slower of the two.

Honestly, I never wanted to get into C# as much because typically, it also
means getting into Windows (from personal experience, companies using C# are
heavily invested in Windows, and I love using Linux for everything).

Do companies nowadays develop and deploy C# on Linux boxes?

Edit: Just a reaction to a number of responses, I know C# is now fully open
source. That doesn't mean companies using C# develop using Linux and deploy on
Linux. C# being open source means very little to me when all deployments are
on a proprietary platform.

~~~
GordonS
C# dev here, mainly working in enterprise size companies who are Microsoft
shops. We usually deploy on Windows for on-prem services, and on Linux for
cloud services (some on VMs, some on containers, some serverless).

I'm more of a Windows guy for desktop, but always prefer Linux for server.
Nowadays dotnet is a first class citizen on Linux and MacOS, through dotnet
core.

On performance, there has been a big focus on perf by the dotnet team over the
past years - I'd be shocked if Java could beat dotnet on just about any
metric. If you know what you're doing you can write allocation-free code, and
you even have hardware intrinsics at your disposal - I recently ported a
hashing algorithm to C# and got the performance pretty close to parity with
native C code (almost _memory speed_ ).

C# doesn't get much love on HN, but IMO it's a truely fantastic language.

~~~
cheerlessbog
Their recent blog post gives a flavor of some of the performance work in the
platform:

[https://devblogs.microsoft.com/dotnet/performance-
improvemen...](https://devblogs.microsoft.com/dotnet/performance-improvements-
in-net-5/)

------
rolls-reus
Is stackoverflow essentially a monolith? I remember reading about the single
database setup several years ago, I'm surprised that they have been able to
keep scaling vertically.

~~~
algorithmsRcool
I think that is the thesis of StackOverflow's devs. You can vertically scale
easier and much further than what most people think.

No groud up re-architecture, no microservice complexity, no crazy high cloud
bills for exotic high power VMs. Just upfront capx for your own big iron and
the skill to run it.

~~~
vii
Amazon is offering 24TB RAM high memory machines on AWS. The Stackoverflow
setup is far away from Big Iron :)

In terms of opportunities to add complexity: I'm surprised there is no mention
of a data-platform or any model training for ranking, etc.

To reduce components, it might be possible to combine ElasticSearch and MSSQL
into just PostgreSQL which has awesome text indices. As MSSQL performs very
well and presents a somewhat esoteric SQL dialect this could be an expensive
project.

~~~
zamadatix
The specs seem to line up with their upgrade notes from 2015 so it was a bit
more "big iron" when made. Not that it's exactly under stress at the moment by
any means but that's just further proof big iron can get you really far.

------
bzb3
They should rewrite everything in python so they need 10 times as much
hardware. The economy is stagnating

------
vyrotek
And here's where they were in 2016

[https://nickcraver.com/blog/2016/02/17/stack-overflow-the-
ar...](https://nickcraver.com/blog/2016/02/17/stack-overflow-the-
architecture-2016-edition/)

~~~
heipei
I've been patiently waiting on on update for this post and others on the
Stackoverflow architecture and hardware on his blog. Nick, if you're reading
this, we're dying for on updated post ;)

------
skunkworker
This is all for a single site (location) though right? I wonder how many
backup AZs they have and how they’re replicating across zones.

It’s surprisingly less infrastructure then I was imagining.

~~~
p_l
Two datacenters, they used to have each site run in 2 not-completely-full
server racks (or maybe 4, been long time since I read the blog post of their
switch gear upgrade to 10 gigabit).

AFAIK they still keep to two datacenters, and the second one is a backup. All
of SO sites run from the same few racks.

------
ksec
I have always thought StackExchange as the Gold Standard of CRUD Apps.

I would be happy if Ruby Rails could do it in 3x the rendering time ( i.e
slower ) with 2x Resources / Server.

------
alkonaut
I knew about most of their architecture, but what puzzles me is: why are some
servers vertical boxes and some horizontal?

~~~
Youden
It could be blade vs. regular rackmount but I think it's just a design choice
since the other ways of arranging 9 web servers aren't as visually appealing.

------
rurban
I'm esp. impressed about the C# Json library Jil, they are using. Many useful
tricks. After the first deserialization it internally optimizes itself to
speed up further deserializations of the same format.

------
simonebrunozzi
So, no cloud, no AWS. All in house?

------
syspec
Didn't know stackexchange made such use of websockets, where are they using
them?

~~~
haneefmubarak
They use websockets for notifications (upvotes, comments, etc).

------
redleggedfrog
I couldn't tell but are those CPU percentages real time?

------
li4ick
I wonder about the future of Stackexchange, now that every community has it's
own subreddit/discort/etc.

~~~
Can_Not
those two things are terrible at the Q&A format, search, and SEO, so probably
not much to wonder about.

------
MuffinFlavored
11k peak SQL queries @ 15% peak CPU usage across how many cores/processors?

------
recuter
300-450 rps for the web server tier seems very low. I wonder why that is.

~~~
f6v
9x300x60x60x24x30=6998400000

~~~
recuter
Sure, I meant I'm surprised one server handles so few requests per second
each.

~~~
ksec
In a blog post a few years ago they explained they could have fitted all 3000+
Req/s in a single server, but its ~99% percentile would have been ~100ms+. So
in the name of performance and sub 20ms rendering time they decided to it with
more servers.

------
jchook
Surprised to see such low peak ops/sec on HAProxy

~~~
rogerdonut
I'm curious to know what you mean by this? It's not like it can't handle more
that is just what their peak workload was. With only 18% CPU usage at peak
it's clear that it could handle much more.

~~~
jchook
I guess I'm mostly curious how they handle websocket load balancing.

------
DonCopal
Much performance! Yet they don't even have a button to select all of code in a
codeblock.

------
brainzap
What about the CDN?

