
The Database Administrator is dead - xivSolutions
http://thenextweb.com/kennygorman/2013/12/12/dba-dead/#!pPtRD
======
falcolas
I guess I had better tell my colleagues that our jobs are all dead! Wait, it's
an absolute statement for a headline, so it's actually "absolute crap".

> These days, the technology decision maker is the dude with Sublime Text open
> and a cloud control panel up in Chrome.

And when he is successful and gets clients, and a few thousand rows in his
database, he realizes that he needs someone to keep that database alive. He
needs someone to figure out how to make the cartesian product queries he's
written into efficient queries.

At first, he hires a consultant for a few one-off gigs. However, then he's
paying someone $200[1] an hour, typically with 8-16 hour engagements. After
getting sick of that cost, and still lacking any kind of long term caring
about his product, he comes to our team, and hires us to be his DBA, albeit
remotely.

Business as a DBA is booming. Nobody thinks they need a DBA, but the reality
is that you really can't afford to not have a DBA. We have customers coming on
board with no backups, no high availability plans, no disaster recovery plans,
queries that are performing cartesian products (and thus taking minutes
against very small datasets), and no monitoring. (And yes, a good portion of
users come to us while using the "solution" proposed by the OP (like AWS RDS),
for many the same problems.)

We set them up with comprehensive backups, automated failover solutions, and
24x7 monitoring. Suddenly, their DB is no longer the primary source of
downtime. They're no longer loosing customer engagement because their frontend
takes seconds to render. They're no longer in the position of loosing their
entire company because some junior developer accidentally dropped their users
table in production.

In short, DBAs are a required part of your business, if you're using a
database. You just haven't been burned bad enough by a poor database setup to
realize it.

[1] Actual hourly rates for a planned engagement. Emergency rates are closer
to $450 an hour. Why so much? You can't get a DBA from a college, from a
technical school, or from any other form of formal education. Most DBAs these
days are grown internally from developers or system administrators who decide
to (or are forced to) specialize while on the job. There are single-digit
thousands of us world wide, and we're in high demand.

~~~
csmuk
I disagree. _dedicated_ DBAs are on their way out.

Databases are becoming pretty good at managing themselves and the marginal
performance gains from tuning usually are easily offset by throwing bigger kit
at the problem or throwing more cash at the plan you are on.

~~~
falcolas
Often the act of throwing "bigger kit" at the problem requires specialized
tuning of the DB to be able to take advantage of it.

More memory? Increase the buffer pool size.

Faster HDD? Tweak the settings that determine how many disk operations are
attempted every second.

Bigger CPU? Figure out the point of diminishing returns on the number of CPU
cores for your DB, and start sharding onto multiple DBs to make sure you can
use all of the cores.

SAN? But I thought you wanted performance. ;)

Plus, what gives you the best DB kit for the buck? I could probably tell you
that (I am a DBA, and get paid to answer those questions), but do you know? Do
you know where to find out?

~~~
twic
Databases definitely require knowledgeable tuning. But so do most other
complicated moving parts in a modern infrastructure - web servers, app
servers, kernels, cache servers, etc. Sysadmins manage those quite
successfully. In my experience, they also manage databases quite successfully.
The idea that a database is a special beast that requires special keepers is a
holdover from a dark age.

~~~
falcolas
> The idea that a database is a special beast that requires special keepers is
> a holdover from a dark age.

As are most databases (which are 10-20 years old).

Compared to a webserver, a DB is significantly more complicated, and
significantly more important to your average business. Nginx takes a nosedive
or performs poorly, and nobody really cares. Your DB takes seconds to respond
to basic queries, and your entire business suffers.

I'm guessing you've yet to be bitten by DB performance issues that couldn't be
resolve by adding indexes or basic query profiling. I'm glad to hear it,
because it's not fun to have to set aside your day job and dive beneath that.

------
makomk
I think someone's spent too long in tech-startup-land and has forgotten that
there's a whole bunch of businesses outside of that, some of which don't have
access to reliable internet connectivity and need to keep going even if the
Internet's down.

------
dsr_
If you are working on confidential data, it is very likely that your security
policy does not allow you to store it "in the cloud", or to hand it off to a
third party without the kinds of contractual obligations that will perplex and
confuse anyone trying to sell a best-effort service.

Thus your databases will be in-house, thus you will hire one or more DBAs and
depend upon their expertise.

~~~
bluedino
>> If you are working on confidential data, it is very likely that your
security policy does not allow you to store it "in the cloud"

The regulations will change with the times, hopefully.

We stored little more than names/emails and non-identifiable/non-sensitive
data at my last job. The 'security auditor' for one client wouldn't let them
sign with us because our servers were hosted at Rackspace, so the servers were
not under our control and a Rackspace employee could access our data, since
they managed our servers.

~~~
dsr_
It's not a matter of regulation. It's a matter of liability. As long as the
company running the database-as-a-service won't accept full liability against
breach of privacy for the information stored there by clients, clients can't
store confidential information there.

And who wants to do that?

------
bluedino
The cloud is killing all the low-level admins. Mail administrators? Backup
operators?

Moving to BYOD will continue to reduce the numbers of administrators as well.
Eventually you'll have two groups of IT staff, a very small group of high-
level engineers who build and implement everything, and then a very large
group of low-skill helpdesk type people who reset your accounts and fill in
your login information on your device.

Amazon's Mayday support service shows this already. Soon, a form of this will
be on every product you can think of. Office, Windows, every tablet and
computer in your office.

~~~
secstate
I couldn't agree more. The most apt analogy to explain this to non-tech folks
I can come up with is how the automobile went from a hobby toy for the
mechanically inclined to a tool people use everyday.

There will always be a place in the world for engineers who understand how to
get the most of out an internal combustion engine and ways to improve it, just
like for database engineers. But we don't all need our car in the shop twice a
month having parts tuned. Nor do the vast majority of databases hitting
performance limits that would require full-time DBAs.

Consulting is a great business decision when you reach hundreds of GB in data.
In-house when your tossing around Tera or Peta-btyes. By the way, I can whole-
heartedly recommend [http://www.pgexperts.com](http://www.pgexperts.com) if
your having Postgres issues. After speaking with a few of their core engineers
at DjangoCon 2013 I would consult with them anytime I had PG performance
issues.

------
joosters
TL;DR: Cloud-based DB co-founder says that most of his customers don't employ
a DB admin.

Um, it would only be surprising if they _did_ , that would surely indicate
that his company wasn't doing a good job.

~~~
kennygorman
True.

But most of these folks didn't have DBA's or even OPS engineers before they
became customers.

Mostly they are teams of developers, they have developed the application and
have (I think correctly) worried about the customer experience, code base, IP,
performance, etc.

They do want the database driver API (in this case MongoDB) to just work
beyond the interface. Backups, scaling, filesystems, etc they want to be part
of the service, and just handled.

This is true for new business and it's true for businesses that are developing
_new_ apps or _new_ projects or even projects that can't scale out on other
infrastructure.

Full disclosure: I wrote the article in question ;-).

------
izietto
The DBA is essential for big companies, not only Facebook or Apple. The author
of the post works for a database-as-a-service provider, it's just marketing.

~~~
spydum
Exactly.. Why bother optimizing your database, just spin up more servers
(which we will happily continue to squeeze your opex budget dry)!

It seems cloud/SaaS is the new answer to throwing hardware at the problem.
Something about the inefficiency grinds at me. I do get at certain scales, it
makes sense.. But it's not always obvious where that line is.

------
morgo
I am a database administrator.

I somewhat agree with the conclusion which essentially says having your on
premise DBA is on the decline.

Hiring DBAs has always been tricky for employers anyway. It's a position of
responsibility (to protect the business' data) that is hard to know you've
hired the right person for, difficult to replace, and difficult to allow to
take any vacation/leave since there will seldom be more than one.

What I don't agree with in the conclusion is that by moving off premise it's
all moving to datastore-as-a-service. Large amounts will also move to remote
DBA as a service (consulting firms).

------
dschiptsov
If they define DBA as a guy who sit in a room 9-to-5 for a $100k+ salary,
then, yes, there is a recession, you know. But usually (or rather unusually)
DBA is an engineer, who could do much more than looking at EXPLAIN output and
tune some variables once in a week.

I have been Informix DBA for years and I could tell that we were the strongest
guys in a team, because in order to do our job we had to understand (abstract
out of running system) the data-flows, access pattern (and especially locking
issues), actual server's topology (disk controllers, channels, hard-drivers)
data partitioning (where this or that table-space lives, what's else on this
volume, this channel, this controller) what are the access patterns for each
table, how indexes are utilized, etc.

We also have patched, compiled and installed all the required software (have
you ever tried to compile Informix support into PHP4? you definitely should.))
and to teach coders how to use it, and then deal with access patterns of silly
scripts, etc.

The claims that some crap like MongoDB (of all things!) service could replace
skilled, productive, (but, yes, quite expensive) professionals is, of course,
utter nonsense (what else we could expect from MongoDB?).

DBAs and Sysadmins (real ones, not these clowns who use nothing but chef or
puppet and doesn't know how ./configure && make works) are becoming extinct
purely from economical reasons, and all these cloud services, ironically,
require even more knowledge to deal with, because all that virtualization crap
messes everything up even more (google for redis on EC2 for a change).

Sadly, idiots are taking over the world slowly but steadily,)

~~~
secstate
__DBAs and Sysadmins (real ones, not these clowns who use nothing but chief or
puppet and doesn't know how ./configure && make works) are becoming extinct
purely from economical reasons__

As long as we're insulting people ... it's going to be 2014 in three weeks.
__Who compiles anything for PHP4? __Boasting about an ability to handle other
people 's messy scripts sounds like an anti-pattern to me, and your
recommendation to your org should be to develop a plan to refactor anything
dependent on a library that has been deprecated for 6 years and actively out
of development for 5.

I also feel like the vast majority of folks claiming merit badges as DBAs in
2013 are the product of failed technology roadmaps and technical debt with no
plans to pay it off. I'm sure there are still places where true, full-time
DBAs are worth their paycheck, but in my experience these are far and few
between with the current status of OSS database options and hardware
performance.

~~~
dschiptsov
> As long as we're insulting people

People may assume whatever they wish.)

> Who compiles anything for PHP4?

It was long ago, but you probably wouldn't believe how many people are scared
to touch anything, leave alone to perform even a necessary security update -
"what if it stop working?!"

Unfortunately, frustration, which sometimes influenced wording of some of my
comments, is based on a quite long time in the industry, and I never asked for
it.

------
benjaminwootton
The DBA may be dead, but he has a lot more life in him than "MongoDB database-
as-a-service" is ever likely to have....

------
Xdes
DBAs have specialized knowledge related to the maintenance and optimization of
a particular SQL engine. Developers see the database as a black box where data
gets stored, but there is a lot going on under-the-hood to keep the data
reliable, consistent, secure, redundant, and available.

The last thing you want is a full transaction log during peak hours.

------
islon
> These days, the technology decision maker is the dude with Sublime Text open
> and a cloud control panel up in Chrome.

My sublime text is open and I have a cloud control panel open on chrome, am I
the technology decision maker? Nobody told me that here on the company.

Wait... maybe nobody told me because... I'm the decision maker D=

------
sourc3
Persistence ignorance introduced by mainstream ORMs about 5-7 years ago have
turned the new generation of programmers blind to the fact database IS part of
your application. Records you save don't get sucked into a magic black box
that just responds to your queries as your app needs grow.

DBAs are not going to go anywhere. Sure, you can scale the DB in the cloud
quite a bit in and perform well but it's not free :) In the cloud, you would
be literally paying for your bad design decisions in terms of hard dollars
rather than performance issues.

Most start-ups or in house apps are fine while the salary(dba) => cost of
cloud. However, that magic condition starts returning false pretty quickly if
you are doing any non-trivial data management.

------
bryan11
Who organizes how the data is stored? In some companies, the database is
largely ignored until performance and scaling is an issue. At that point a DBA
is brought in and the next months or even years are spent fixing the design so
the entire product can perform properly and grow.

------
carlmcqueen
The world is definitely changing for the DBA to which he made a very good
points. I feel he graced over a big facet in a single line of the article
which had to do with Apple/Facebook and large companies.

The banking world doesn't give their data to others and organization of ATM
data, market data, customer account data, etc is only getting more complex and
requiring excellent organization and management for terra data analysis to
protect cusomters for fraud and worse.

------
cognivore
So, a thought experiment.

Pick any data storage system you like - MongoDb, Redis, Riak, whatever.

Now, you get to place a bet. In the future (let's say 25 years) what will
still exist: your choice of NoSQL, or the standard database with schema, SQL,
access, and so on? If you choose wrong, you die.

Now you get to see the future and see if you die. Which are you betting on?

~~~
jrarredondo
Sometimes you just bet on red AND on black to stay safe while keeping the
wheel rolling and the drinks coming.

------
static_typed
The DBA is not dead! They just look and smell that way!

But in all seriousness, if your app uses a database, you are incompetent not
to employ an expert to help with the database, whether advising on the query
plan of those non-performant queries, or what is the best setup for the
current stage of the business and app, they are very useful.

~~~
mattmanser
Most people don't need dedicated DBAs and worse the DBA often never has
sufficient domain knowledge of the problem to make an intelligent suggestion
anyway.

DBs performance is complicated, yes, but the vast majority of it is extremely
simple. It's just none of this simple stuff has to be learnt until it's too
late and the cost of fixing it has dramatically increased.

There's a certain level where you need a DBA and that bar has been getting
higher for years.

What we really need is to demystify DB performance, which for the most part is
fairly simple.

What you need to do is teach your developers how to read those query plans.
Show them how to find the expensive queries. To give them tools to easily see
what queries their ORM is spitting out. To show them how to use a SQL
profiler.

Tell your developers how query plan caching actually works. How a clustered
index works and what you should and shouldn't put it on. Explain how indexes
work. Explain how DB pages actually work and then it's obvious why certain
indexes are a bad idea. Explain how relational keys are very important for the
DB engine and leaving them off is not an 'oops', it's a serious mistake with
long term consequences.

And that's at most a few days work. So why do you need that DBA?

Aside from that you need someone who knows how to maintain a DB, but again
that's not particularly complicated and once it's done you can forget about it
apart from the occasional sanity check that it's all working properly.

~~~
CrossWired
So basically train them to be DBAs?

~~~
tbrownaw
No, train them to _use_ the database properly. A DBA is the person who keeps
the databse running, much like a sysadmin is the person who keeps the server
running.

~~~
baudehlo
That's a naive view of both DBAs and sysadmins these days. They both do much
more than just keep things running. Sysadmin has mostly turned into opsdev.
Same with DBA.

~~~
tbrownaw
_Sysadmin has mostly turned into opsdev._

Not in places large enough to value loose coupling (e.g., if keeping your
servers running comes out of a different budget than improving your
applications).

~~~
baudehlo
Opsdev != dev

