

Hardware is Expensive, Programmers are Cheap - WalkingDead
http://lastinfirstout.blogspot.com/2009/01/hardware-is-expensive-programmers-are_29.html

======
patio11
Isn't the real story of this "incompetence is expensive"? Seriously, that
application design is worse than the worst implementation we've ever had in a
failed outsourcing project. (I make Big Freaking Web Apps for Japanese
universities at the day job.)

Up until recently universities pretty much had to overspend on hardware,
though. (Virtualization/expandable-on-demand cloud computing may eventually
get around to changing it, but the customers aren't ready for it and the
programming costs to take advantage of it dwarf the benefits for our clients.)

Most of the important systems that need to scale have very, very bad usage
patterns from a hardware buyer perspective: for example, take course
registration. (Edit to clarify: They are probably not talking about course
registration, because there is no way in heck that peak is only 50% more than
the steady state.)

At a university with 10,000 students, about 360 days out of the year you can
run the course registration on a laptop while it is being used to play World
of Warcraft. Then there is course registration season, at which point your
peak concurrency goes from 1 user per hour to generally a _multiple_ of your
student population all signed in at once. (Because, no matter what you do,
they will open multiple windows/connections/etc because "the site is so slow,
come on duuuuude, why the heck is this POS always so slow?")

All the accesses are dynamic. Most of them have writes attached. You have to
get caching right because if you overbook a class and tell 15 students that
they're confirmed a seat in a room which sits 12 because your cache got stale
for three minutes, your customer gets yelled at, and they will turn around and
yell at you. The end users are also typically incompetent at using the system
(typically 1/4 of them have never used it before) and they will perform an
impromptu fuzz test on it.

(Oh the stories I can't tell, sadly.)

~~~
TJensen
I used to work at a company providing enterprise systems for higher education.
Fall registration was panic season; we were in fire-alarm mode for most of
August and September.

Those were good times. :)

------
danielrhammond
Sometimes Hardware is Expensive, and so is time though.

I think often times in a startup, especially one thats bootstrapped, its easy
to get caught up in trying to optimize everything too early on to scale for a
million users before you have your first thousand. Sometimes you have to
optimize for your development time first and product development goals, and
leave the optimization till you get a bit of runway.

~~~
alecco
> Sometimes you have to optimize for your development time first and product

> development goals, and leave the optimization till you get a bit of runway.

That's a typical way to postpone and fall into a trap. Not all optimizations
take huge amounts of time. And certainly there are some architectural
decisions you can take early on that can save you a lot of wasted time later.
The most common is to prevent bottlenecks.

For example, it's not necessary to make a clustered DB from launch day but
coding and managing the UI and middle-ware to be ready for a switch to
clustered DB usually takes very little impact. If you don't do this and the
site becomes successful, re-engineering your whole site to support a new DB is
usually close to impossible or extremely expensive. The site ends up in the DB
hardware feedback trap just like in the article.

Edit: format fix.

~~~
billswift
Architectural decisions also limit what you can do later. Generally, the more
focussed/optimized a system the less flexible. To the extent you KNOW exactly
what the system should do, it should be optimized from the beginning; but, as
PG points out repeatedly, most startups change direction at least once after
launch.

------
stcredzero
In other words, figure it out for yourself in your particular situation. Do
the back of the envelope calcs. The reason why articles like this and the
opposing view at Coding Horror are posted is because authors want to draw
attention to management with preconceived notions.

------
bayareaguy
Looking at things along the hardware/programmers are expensive/cheap axis
(take your pick) is short sighted and highly subjective. Setting aside
political considerations, the real thing decision makers should focus on are
the organization's underlying time and efficiency constraints and
unfortunately these things are most often overlooked, misunderstood or
inaccurately represented.

------
jasonkester
This sounds like an edge case to me. Now that it's no longer 1998, most of us
don't spec out $500,000 boxes for our stuff anymore. A $5,000 box will get you
a long way for just about anything you need to do.

I suspect that the author is looking at a problem that would require tons of
hardware regardless of how well it was optimized. Evidence of this can be
found in the fact that even after tons of optimization, his $1.5M setup is
still running at 20% load steady state.

Our single <$5,000 box handles about 4M pageviews per day without moving the
cpu above 5% steady state. That's the sort of baseline I'm used to from the
Microsoft stack, so it causes me to question whether the author is really
looking at a mainstream case.

~~~
kscaldef
> Our single <$5,000 box handles about 4M pageviews per day without moving the
> cpu above 5% steady state. That's the sort of baseline I'm used to from the
> Microsoft stack

You realize how meaningless a statement like this is, right? You just can't go
around talking about "pageviews" as if they were some uniform measure of
workload.

~~~
sho
Yeah, I was about to mention that. 4M a day is about 50/sec; if it's nothing
but static pages you could serve that on a Pentium 1 without breaking a sweat.
I've been giving away P4s recently, so their value is effectively zero - they
can probably do it at under 5% too.

The problem is obviously when you're _not_ serving static pages.

------
radu_floricica
This is not a problem of hardware vs software. It's a problem of vendor's
money vs your own. Of course he doesn't care. Actually, the more you spend on
hardware the cheaper the software seems.

~~~
patio11
_Of course he doesn't care._

My day job sells integrated solutions to Japanese universities. We would care
very intensely how much the hardware costs in a circumstance like this. (Which
we wouldn't be in, because we try not to deliver software which is an
abomination against all that is good and holy in terms of database use, but
still.)

The math is simple: the university typically has a budget of X million yen to
Get This Done. It doesn't matter what the line items are on our invoice -- we
can't charge them more than X million yen.

Given that constraint, what do you think we want to charge them? Software
license fees for our solutions, where our margin is anywhere from... crikey, I
can't tell you the numbers, but "high to higher". Software license fees for
third party providers like a certain Enterprise Database, where the margin to
the reseller (us) is from low to medium? Or the margin resellers get on
hardware, which compared to our software is small enough we could use it to
remove food from between our teeth?

~~~
radu_floricica
I mean, once the contract is signed. I know how sucky it is to go into such a
relationship with a vendor. Custom apps are always risky... and hardware costs
are one of the risks.

In your situation, actually in most situations when the client can compare the
costs for both software and hardware before, it's logical to optimize the
software.

------
cbetz
there is a fundamental difference between software for sale and software as a
service when it comes to this debate. the economic efficiency math is much
different for licensed software because more than one company is using it.

~~~
edw519
"the economic efficiency math is much different for licensed software because
more than one company is using it"

So is the risk.

Nothing drives customers crazier than a slow system because _someone else_ is
having a busy day.

I spent my first 10 years decoupling applications from each other because
independence was more important than economies of scale. Now we're swinging
back the other way. Hopefully, we will have learned something this time.

------
zandorg
Yeah, if you can make software run 100 times faster with lots of profiling and
hand optimisations, that $300 laptop is basically a $30,000 laptop.

~~~
dasil003
Of course you need to consider what those optimisations will do to maintenance
and future development costs.

------
vaksel
I think the problem is that the costs add up slowly. Your site starts getting
slow? You just throw another dedicated server at it, increasing your cost
300-400 bucks a month.

------
Tichy
Hm, why not find something to do with the remaining cycles?

~~~
alecco
In a production database!?

~~~
Tichy
I was assuming in a modern "cloudy" setup, processing power and storage is all
kind of fluent. Or if it isn't yet, might be worthwhile to make it so? They
could set it up all with virtual machines and stuff?

