

Ask HN: Just replace my DBA: painless db admin and scale without the cloud?  - jacobscott

The recent launch of FathomDB is the latest of many articles on HN about databases, scalability, and the cloud.<p>What is the state of the art -- in terms of scalability, management, monitoring -- for people who need to keep their data in house? Maybe X where X:FathomDB as Xobni:Gmail? Or are we stuck hiring a DBA?
======
xenoterracide
There is no easy out, if you really need to scale, regardless of
infrastructure you are going to need someone who understands databases. If
you're lucky you might be able to find someone who can do more than one job.
e.g. dba+sys admin. Just make sure that they really do know both things.

~~~
thwarted
Yeah, this kind of question seems really close to "I'm a business guy, all I
need is a tech guy to implement my idea", which implies there's some way to
avoid having to know how to do something that is core to the business.

~~~
jacobscott
I want to make it very clear that I'm not a business guy.

Think of it in the context of building a database-backed enterprise app that
customers run in house. You don't want to ship a DBA to every customer...

~~~
thwarted
For that you use an embedded database system. Many enterprise apps come with
embedded database systems, or require a major database system to be installed
in tandem, and these things are targeted at medium sized companies that don't
have DBA specific to the the department that is using the app.

------
jacobscott
If it helps, I asked a more concrete version of this question focusing on
PostgreSQL on stackoverflow:
[http://stackoverflow.com/questions/556325/postgresql-
managem...](http://stackoverflow.com/questions/556325/postgresql-management-
and-monitoring)

------
iamelgringo
The state of the art _is_ hiring a DBA. They get paid to be up on all the
database monitoring, scaling, managing and management tools. This is why DB
consultants charge $200+ an hour. It's pretty arcane knowledge, and for a
business of any size, the database quickly becomes the lifeblood of the
organization.

This is also the reason that databases like Couch DB or Amazon Simple DB or
Google's App Engine are so appealing. They all have the promise of reducing db
management headaches, at the cost of sacrificing features.

------
DenisM
Try a commerical database, such as Microsoft SQL Server (FD: I work there as a
developer). To bring up a recent example, it can build indexes without locking
the whole table so you wouldn't need to jump throught the hoops like the
FriendFeed guys just did with MySQL. Unless your time is free it will cost you
less to buy SQL Server than to write your own "online index thingie" on top of
MySQL.

And if your time _is_ free, then you are probably a startup and you can get
SQL Server and bunch of other MS tools for free via
<http://www.microsoft.com/BizSpark/>

MySpace runs on SQL Server:
[http://www.sqlservercentral.com/blogs/steve_jones/archive/20...](http://www.sqlservercentral.com/blogs/steve_jones/archive/2009/02/18/myspace.aspx)
and if it's good enough for them it will likely be good enough for your site
as well. I'm not sure if the scale-out story is any good (there are
"partitioned views" which are basically "sharding" but I never used those),
but scale-up story is as good as they come - it can efficiently utilize
64-cores of the HP superdome.

I don't know much about Oracle and IBM DB2, but I know that the latter has a
free version without too many restrictions on it, so give them a spin as well.
If ease of use is importnat to you, common wisdom has it that Microsoft's is
the easiest to use among the top commercial vendors. NB: I did not verify the
common wisdom, YMMV :-)

~~~
moe
You're suggesting to use a commercial database but don't provide any reasons
beyond what free databases can do.

PostgreSQL is pretty much on par in terms of features and performance with
SQLServer and Oracle. So if you're looking for a free solution to roll your
own then look no further.

The only reason I can see for going with a commercial DB is when you not only
want commercial support (you can buy that for Postgres, too) but a
shrinkwrapped "don't bother me"- solution with SLAs etc. Be prepared to pay
through your nose for that, though.

In terms of usability there is no big difference between any of the commercial
or non-commercial offerings. They all have their special peculiarities (which
is why you always hire a DBA who is familar with _your_ RDBMS, not just with
"any" RDBMS) and the shiny GUIs that some of them provide are not normally of
interest in day-to-day work.

------
charlesju
Managed hosting works for us (Engine Yard), it is significantly cheaper than
hiring your own DBA.

~~~
xenoterracide
having managed hosting doesn't mean you don't need to a dba. In my experience
all the host does (worked for a managed hosting company) is set up the servers
and keep them running, their help really can't help you with tweaking queries
or tuning the db. I can't speak for on engine yard specifically. But I
wouldn't bank on your host knowing anything beyond the basics of databases.

~~~
charlesju
Engine Yard has several excellent DBAs on staff. They tell you what queries
are slow and tune the db to optimize for your type of traffic. That's why they
are worth the 30% premium on top of EC2.

~~~
xenoterracide
engine yard may be an exceptional exception to the rule. I was generalizing,
and speaking of managed hosting in general. Like I said, I can't speak of
engine yard specifically, I haven't worked for or with them.

