
Why Stack Exchange Isn’t in the Cloud - johns
http://blog.serverfault.com/post/why-stack-exchange-isn%e2%80%99t-in-the-cloud/
======
nupark2
Given the introduction, I thought we were going to see a fantastic breakdown
of cost estimates; capex and opex expenses, 95% percentile bandwidth pricing
vs. Amazon's fixed rate per-byte pricing.

A real analysis would have even dug into the question of sysadmins vs
engineering staff, and the potential for "the cloud" to allow programmers to
actually program infrastructure, eliminating internal sysadmin staff
completely (and outsourcing all the remaining sysadmin work to Amazon).

Instead, we got bizarre platitudes about how they "love computers". How
boring. Early in my career, I spent a lot of time in the hot/cold, blaring fan
world of data centers -- sometimes late at night, when a hard drive failed or
a switch's fans went out. Anyone that thinks that it's pleasant to spend time
in a data center is insane; whether or not it makes sense in terms of cost was
the answer I'd hoped to get from this article.

~~~
brudgers
It was a real analysis that went beyond just counting the beans. As a
privately held company, getting rid of passionate people doesn't make the
stock go up, increase anyone's bonus, or benefit their long term goals - and
blogging about their values certainly cannot hurt in attracting top notch
talent.

~~~
nupark2
If you remove any rational analysis based on productivity, developer time, or
finances, then we're only left with a question of personal taste -- and it's
impossible to argue against personal taste.

An evaluation based on personal taste is useless. If I say "I like orange
because it's a great color" -- then either you already like orange, in which
case my analysis does you no good, or you don't like orange, in which case my
statement can simply be ignored.

The same applies here, in that the article could have been coalesced to a
single sentence: "We don't use the cloud because we don't want to. We like
dorking around with hardware and data centers, just like someone else might
link tinkering with their car or collecting bottle caps"

Possibly interesting -- but not useful.

~~~
ssmoot
Instead of 12 physical cores, 96GB of RAM and a 2TB SSD Array pushing 1M IOPs
on dedicated hardware for my PostgreSQL database servers, I'd need 1TB of RAM
in an AWS box because I'll be lucky if I can even break 10K in IOPs.

Does the price make sense then?

I have yet to see any significant AWS deployment that doesn't feel like it
could be done better, more reliably, and much more cheaply as a co-located
setup.

~~~
reuser
You can't really create a massive co-located setup on demand for big jobs,
then tear it down ... use the right tools for whatever job you're doing. EC2
being more expensive for your Postgres deploy doesn't mean it is an expensive
toy

~~~
ssmoot
I don't think I could say it better:
<http://twitter.com/#!/dhh/status/133656040561577984>

If you really have a need to setup and teardown a bunch of cores for the
occassional large batch, nobody's questioning that the cloud is an economical
way to do that.

But I've _never_ had to do that. Not in a way that would make the development
time involved economical anyway. Wait two hours for this once-in-a-blue-moon
processing job to finish, or spend a day setting up a process to handle such
jobs quickly in the future?

It'd really take something exceptional (again, with low IOPs demands) to have
that make much sense unless I was already hosting in the cloud and had
invested money in making such a task quick and cheap.

------
MichaelGG
What a silly argument. "If you just want to use someone else’s computers, it
means you don’t love computers — at least not every aspect to them."

Uh, don't they use Dell? If they love them so much, why not build their own?
Why stop there? I love software and programming languages, but it doesn't mean
I'm going to use my own compiler or run a company on my own libc.

~~~
KyleBrandt
There is always the question of degree and practicality. So in terms of libc /
and custom compilers that is perhaps ad absurdum. There really wouldn't be any
practical benefit taking it to that far for most companies. However, people
that love programming languages often probably wish it was practical and that
they had an excuse to write compilers at some point.

As far as Dell goes, I honestly have mixed feelings at this point. When I
started at the company there was less than 10 people, so the idea of having
centralized firmware updates etc seemed practical. I'm not sure I really feel
this way any more -- I go back forth. But sticking with Dell seems to make
sense at this point so we have more uniformity in hardware and management.

~~~
dkersten
The practicality argument doesn't apply when they already say they're doing
more than necessary _because they love computers_ and if you don't that you
_don't really love computers_.

When someone says that you don't love computers because you did whats most
practical or cost effective for you, how can they say that they love computers
when they do or don't do something else because its most practical or cost
effective to them?

If it saves my time, is cheaper or easier to use amazon rather than running my
own data center, that doesn't mean I don't love computers. If they love
computers so much that practicality is out the window, then why aren't they
using their own operating systems and libc's?

~~~
jodrellblank
Clearly there's no difference in effort between a day installing and
configuring a Dell server and several thousand person years of writing a
solid, scalable operating system, so your question is perfectly reasonable.

~~~
dkersten
I understand that what I'm saying is a bit of a stretch, but when they claim
people don't love computers for not doing that they're doing, they may as well
take it to the extreme. People have written operating systems, maybe these
people should say everybody else doesn't love computers.

I mean, obviously I don't expect anyone to build everything they use. My
argument was aimed at the "you don't love computers" statement and not at the
fact that they don't write their own OS.

------
quanticle
The reason I'm still wary of the cloud is that the abstractions are still too
leaky (to borrow Joel Spolsky's turn of phrase). When you start abstracting
away core system calls (like fsync), things work great 99.99% of the time. But
when that .01% bites you, it bites _hard_. We don't expect core system calls
(like fsync) to fail. And when they fail, the fact that our code is two or
three levels levels of abstraction higher means that we often have no way of
fixing the issue. The cloud will be at a disadvantage to "real servers" as
long as its abstractions are leaky enough to be distinguishable from real
hardware.

~~~
wmf
In what way does fsync work differently in the cloud?

~~~
bad_user
Because there's an extra virtual layer between you and the hard-disk, fsync()
does not do what it says it does, which means that a database cannot actually
guarantee that a transaction was successfully completed.

In case your traffic is huge, this can cause big problems that are very hard
to fix. And it did cause problems for Reddit, as EBS is notoriously awful in
this regard (and others, like big latencies on access).

Basically when working at big scales, nothing beats having complete control
over your infrastructure. It may be tough and time consuming, but at least you
can identify the problem and fix it.

It is my opinion that something like Google's search cannot be built on top of
Amazon's AWS.

~~~
asharp
Saying that because fsync() doesn't work in AWS that it won't work 'in the
cloud' is much like saying that because you can't take a cheap sedan to the
bush that a 4wd can't possibly exist.

Google can't exist (well) on top of AWS as AWS is designed for the deployment
of (qusi)stateless applications. Saying that you can't build a data intensive
application on top of it is like saying that your machinegun sucks at grating
cheese.

OVM was designed to run a search engine origionally (darkmatr, now defunct due
to lack of comparitive profitability), so you could quite easily build google
on top of it. It wouldn't surprise me if google would work really well on top
of that stack.

------
jinushaun
Poorly written argument full of red herrings. From what I can tell, their
answer to the "why aren't you in the cloud" question is that they "love
computers so much that they want to do it all themselves because it's so fun".

~~~
nephyo
If I asked you why you were eating pizza and you responded "because I like
pizza" would you still consider that a red herring argument?

All they are saying is that they are not in the cloud because they don't want
to be in the cloud and because they quite like not being in the cloud. It
works for them. Why is that expression at all controversial?

If they'd labeled their blog "Why being in the Cloud is a bad idea" or "Why
being in the cloud would cost Stack Exchange a lot more" or "Why nobody should
ever use the Cloud" then I could see the reason for the dispute. Then their
arguments would be very unconvincing.

It is certainly valid for Stack Exchange to make a choice because it fits
within their culture and then explain that that is why they are doing it. You
might think it's a dumb decision and that's fine. But they clearly are not
trying to convince you that they've made the best possible decision. They're
just saying that good or bad, they've made the decision that they want to
make. And it's clearly a decision that has been working well for them so far
anyway. So why do people keep bugging them about it?

~~~
jinushaun
You're correct about alternative titles for the blog post. However, the
assumption is that on a technical blog, readers would expect technical answers
--at least I did. You're not going to tell your boss that the next product
should be developed in Java because you like Java. They want concrete reasons.

------
jtbigwoo
I suspect that this is part of the problem with many big manufacturing
companies these days. Ford was a leader for so many years was because their
leadership was passionate about engineering and manufacturing. Henry Ford was
an engineer for the Edison Company before he was a manager and is famous for
obsessing about every detail of the manufacturing process. That sort of spirit
infected the company up through the 1960's and 70's when the MBA's took over
and everything was reduced to a column of numbers.

It appears that many early tech companies (TI, HP) have headed in the same
direction. Maybe 30 - 40 years is the limit before the accountants take over.

~~~
mechanical_fish
A Ford bought today is less expensive, considerably safer, more efficient, and
more reliable than a Ford from the days of Henry Ford.

Meanwhile, today's manufacturing companies routinely make objects – like the
processor driving the machine that you are reading this on – that are orders
of magnitude more complex than the most advanced technology of 1959.

And all of this was done by companies that employ lots of accountants. I'm not
sure why you think that accountants and engineers don't routinely coexist, or
that Henry Ford didn't employ plenty of accountants back in his day. Being
able to manufacture good stuff in bulk at reasonable prices is a major
exercise in accounting, and always has been.

~~~
Donwangugi
The current CEO of Ford, Alan Mulally is an engineer by trade. I attribute a
lot of their current success to him.

------
espeed
One key reason I suspect is that you haven't been able to get SSDs in the
cloud so their scaling up approach would be more difficult. Having your own
servers means you can "NewEgg your way out of it"
(<http://news.ycombinator.com/item?id=3243133> :-).

------
Joakal
Past article 'Coding Horror: Scaling Up vs. Scaling Out: Hidden Costs'
[http://www.codinghorror.com/blog/2009/06/scaling-up-vs-
scali...](http://www.codinghorror.com/blog/2009/06/scaling-up-vs-scaling-out-
hidden-costs.html)

Although it's not quite the cloud, one of the founders prefer vertical stacks
due to several reasons including licensing costs.

~~~
asharp
You scale out hardware layer by finding optimal hardware and scaling out on
that.

The major problem with their 'scale out' chart is that that particular
hardware is far from optimal hence costing far more then it should.

------
david_a_r_kemp
I'm growing to understand "the cloud" not as AWS/Rackspace/Heroku/whatever,
but as a mindset where: 1) you're not tied to hardware 2) your application
scales horizontally 3) you can easily (and dynamically) add (computational)
resources as needed, preferrably managed by your application. 4) you build
fault tolerance and redundancy into your application

This article basically takes the approach that "the cloud" is AWS.

------
ww520
OT: I've heard claims that dedicated servers beat cloud VPS (i.e. AWS) in
cost. But most of the dedicated servers I've seen cost $100 and up for a
month. Is there a comparison on the cost/computing_power with benchmark for
review?

~~~
asharp
What you care about at any reasonable scale (ie. the point at which costs
matter) is $/serviced request. That usually ends up being dominated either in
$/cpu cycle or $/iop. The current problem is that most cloud platforms suck
iops which makes them far more expencive then they really should be.

Cloud vs. dedi vs. colo in terms of raw prices though, depends entirely on
your price of money and the price you can negotiate with your provider.

------
coob
So for cultural reasons?

Is it not perhaps because there's much less competition for .NET stack cloud
platforms, therefore less quality?

~~~
hughesdan
Amazon has very good .NET support. So does Rackspace. So does Azure
(obviously). In my personal experience the quality of cloud platforms has been
comparable for both Linux and Windows environments. Is there any evidence that
.NET cloud platforms are of lower quality? I'm genuinely interested. Please
elaborate.

~~~
reuser
Personally, I doubt there is a real quality problem with those platforms. But
running .NET on those platforms looks unnecessarily expensive to me. If I were
married to Microsoft (as SO is) then I would avoid 'the cloud' for that
reason.

Given that one wishes to use Windows/.NET, I feel the decision follows pretty
naturally from an analysis of costs - no vague argument about culture
necessary

~~~
hughesdan
I would say cost (referring to license fees) is more relevant in deciding
whether or not to choose the .NET stack in the first place. You're not going
to avoid license fees by avoiding the cloud.

Their culture argument is a funny one. But I have no beef with it. If owning
and tinkering with the hardware makes them happy, more power to them.

As an aside, I've been using Azure lately for my .NET projects. MSFT has done
a nice job with that platform and don't get enough credit for it IMHO. I
especially like SQL Azure.

------
mathattack
I think we are overanalyzing here. This is like a car company that maintains a
garage to attract people who like to look under the hood. It's taking a
strategic people decision when hard data isn't easily available. Everything
about sys admin job descriptions seems like a digression.

------
eduadecastro
Indeed the cloud has no soul

------
jaequery
It's because they are a performance hungry company. I thought it was quite
obvious by now.

------
lhnn
Why is everyone attacking this article? So what if their reason is subjective?
They want people who enjoy working on the whole stack. They have fun doing
that, and that's all the reason they need.

They aren't saying that "people who use the cloud don't care as much". It's
just their thing.

They're the masters of their own company, and as such, can do things however
they like. One of the benefits of a private organization.

~~~
chollida1
I think the reason most people are disappointed is that from the title they
expected the article to be something that it wasnt.

I personally thought I going to get a break down of the pros and cons of being
cloud hosted vs running your own servers.

The article ended up being completely non technical or quantitative and boils
down to "we like to build our own computers".

Chalk it up to high expectations that weren't met :) That's not the authors
fault.

