
Paul Buchheit: The secret to making things easy: avoid hard problems - paul
http://paulbuchheit.blogspot.com/2007/04/secret-to-making-things-easy-avoid-hard.html
======
Sam_Odio
_Want an example of "simple but less reliable"? Here's one (in one or two easy
steps): 1. All updates go in to memcached, but not the database 2. (optional)
A background process occasionally copies entries from memcached to the db.
Without this, the values will be completely lost when memcached restarts._

Paul - here's an interesting take on your recommended solution: Use solid
state ram and forget about step two. You get fast a read/write, without the
risk of data loss if the machine restarts.

Here's a pretty cool board that will take DDR SDRAM, and supply it with a
constant source of power (imitating solid-state memory):
<http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=1>

Those who insist on a database, could even store the database on this board.
Best of both worlds.

Here's an insane graph showing the database transaction rate for the board vs.
a conventional hard drive: <http://techreport.com/reviews/2006q1/gigabyte-
iram/index.x?pg=7>

~~~
aristus
Solid-state drives greater than 128GB are in OEM testing now. Expect server
economics to turn upside-down in about 12 months, when persistent storage seek
times get 100X faster.

~~~
paul
Yeah, solid-state storage is clearly the future for most apps. Flash is over
10x cheaper than DRAM though, so it'd be nice to see some high-performance
flash drives as well. It's also nice to not have to worry about losing data
during a prolonged power outage.

~~~
aristus
No need to wait, really. Addonics' SATA/CF adapters for your $80/8GB flash
cards cost $35. Even cheap compact flash does random seeks 40x faster than
platters. Throughput is a little slow but what the hell, for 115 bucks? If I
were building a server farm right now I'd try those things out.

~~~
mattculbreth
This is a cool thread.

So please spec out a couple servers for us. How would you get a bunch of
memory (32GB, 64GB even) into one of them and still have it be a good web/app
server?

~~~
aristus
My current favorite is the Altus 600 from Penguin Computing. Cheap chassis
with dual sockets and 16 RAM slots. More slots means cheaper chips. 2GB chips
for that board are about 300 bucks. Throw two 2.2GHz Opterons and 2x 300GB
drives in there. Add an SATA controller card, 4 SATA/CF adapters and CF cards
(in the CD-ROM bay), and you've got a nice machine: 1U, dual processors, 32GB
of RAM, 600GB of platter, 32GB of flash for about USD$7500.

Expensive, yeah. 5 grand of that is RAM. You could get away with 4GB of RAM
and still have a fucking fast web/app server as long as your active dataset
fits on the flash.

~~~
paul
$215:
[http://www.newegg.com/Product/Product.asp?Item=N82E168201343...](http://www.newegg.com/Product/Product.asp?Item=N82E16820134302&ATT=20-134-302&CMP=OTC-
pr1c3watch)

Brings the memory cost down to around $3440.

~~~
aristus
Wrong speed but yeah, that's the ballpark. I used Crucial to price the RAM.

------
brlewis
An even better secret to making things easy: only try to solve problems that
exist.

This is a nonexistent problem: "The problem? Most databases are still storing
my name using the same techniques that they did 15 or more years ago -- they
seek the disk head to some specific location and then read or write my name to
that location. These databases rely on the one operation that DID NOT
dramatically improve in the past 15 years! They still perform like it's 1991."

I don't think PostgreSQL is the only database that uses write-ahead logging.

~~~
paul
Post some numbers. How many updates per second can you do?

~~~
brlewis
I don't know. If that becomes a problem I'll put the write-ahead log on a
faster disk to buy time to look into it.

~~~
mattculbreth
Why can't you just succumb to peer pressure and get rid of that database? :)

~~~
brlewis
I'm not sure. First I'm going to get rid of that operating system. It has lots
of features that aren't relevant to my web app.

~~~
paul
It's somewhat amusing, but also mildly depressing, to see these remarks
despite the fact that I explicitly stated (in bold, even), that "I'm not
advising you to ditch your database!"

And yet somehow that gets translated into "get rid of that database."

~~~
brlewis
Sorry it's mildly depressing. Unfortunately, any thoughts you bring up related
to a polarizing issue are going to end up being framed in terms of the
polarization. That's just the way it is; some things will never change.

On the up side, you are going to learn a lot about essay writing, and about
how concepts do/don't get through to people. These are very valuable skills.

------
mattjaynes
Loved this point:

"The thing is, a lot of those difficult problems are irrelevant for 99% of
products. For example, "real" databases can handle transactions that are too
large to fit in memory. That was probably a really important feature in 1980.
Today, you can buy a computer with 32GB of memory for around $5000. How many
GB transactions do you suppose Twitter performs? My guess is zero -- I suspect
that their average transaction size is closer to 0.0000002 GB (messages are
limited to 140 characters)."

