
How 'DevOps' Is Killing the Developer (2014) - throwaway151514
https://jeffknupp.com/blog/2014/04/15/how-devops-is-killing-the-developer/
======
pmontra
> Good developers can be passable DBAs if need be

Years ago a DBA I summoned to a project told me that my DB design was not bad,
given that I'm a developer. Then he made it twice as fast by redesigning
tables, putting in triggers, etc. (it wasn't a matter of ORM.)

Sadly DBAs are not even on the radar of small to medium projects. If they are
ever hired it's too late to take full advantage of their work. The naive DB
design of developers is too naive. Somebody could say that a DBA would be a
premature optimization, IMHO the reality is that the tooling we are using
(especially ORMs and their migrations) are designed to create vanilla DBs and
vanilla queries (which are fine in all low volume applications.)

An example from a distant past: I remember when foreign keys were taboo among
Rails developers because Rails didn't have a way to express them in its DSL. I
added them with manual SQL queries in migrations. I was told they were useless
but I kept them in the DB (lately I discovered that many developers don't know
SQL and didn't attend to any DB course at university.) Fast forward to a few
days ago: a developer of a Django project asked me if we had indices in the
database, because he's been told that they could speed up queries. I explained
him that Django adds indices in the obvious cases (primary keys and foreign
keys), plus I added some on the fields we often use in our queries.

~~~
scarface74
You shouldn’t need a DBA for a small or medium project. Any developer who
hasn’t taken the time to learn basic RDMS theory like normalization, indexing,
foreign keys, etc. doesn’t take thier craft seriously. Knowing the basics of
how databases work is a required skill in almost any implementation.

The issue is “framework developers”. I am not opposed to learning and using
frameworks but you should always know at least one level below the framework
you are using.

If you are a web developer who uses Angular/React and Bootstrap everyday you
should know JavaScript, CSS, and HTML well.

If you use an ORM, you should know enough about how databases work to have a
mental model of what the database is doing.

~~~
ballenf
> you should always know at least one level below the framework you are using

I've never heard that good advice put so succinctly before. Has this advice
been around for ages and I just missed it, or is it a newly relevant concern?

BTW, what would _two_ levels be to, say, a React developer, out of curiosity?
Would it be the browser engines? Knowing how JIT compilers work, etc.? Or
would it instead be a deeper understanding of the language spec, so that you
know more internal functioning? Not that there has to be a clear-cut answer.

~~~
scarface74
I heard it on a podcast and I am going to butcher the explanation but they
explained it as knowing the “abstract machine”.

When I started learning AppleSoft Basic back in the 80s, I knew how to
optimize my BASIC programs because I knew how the interpreter worked and so
knew assembly.

When I started learning C#, I knew both C, Win APIs and how the CLR worked.

But for a web developer who communicates with backend servers, I think it’s
important to understand HTTP.

~~~
pc86
I've heard a similar idea on Hanselminutes. The way he states it is that
everything is layers of abstractions, and in order to be _great_ at whatever
your chosen layer is, you should be pretty good at the layer below it and of
passable quality (think junior dev level) at the layer below that.

~~~
scarface74
Actually it might have been Hanselminutes. Do you know which episode?

~~~
pc86
I've heard it a handful of times on various episodes but I know he's brought
it up when speaking to folks who work primarily one layer below him.

------
mikece
To be fair, it's the combination of DevOps done wrong and Full Stack done
wrong that will cause interminable chaos. Like "Agile," it's the given manager
and team's understanding and method of implementing DevOps and Full Stack that
are the issue.

The term "Full Stack Developer" is especially problematic because taken at
it's literal meaning it suggests that a developer should be fully competent at
ALL aspects of developing an application, which necessarily include deep
understanding of system administration, database administration, tertiary
dependency (caching apps like Redis, for example) administration, network
infrastructure configuration and maintenance -- all of which are impossible
for one person and we haven't even gotten to code yet! I think a different
term needs to be coined which implies variability until defined at a given
shop. I propose "Wide Area of Responsibility (AOR) Developer" which, for Acme
Corp, might mean "Is competent in writing server-side REST endpoints AND
managing development/production database schemas AND load balancing traffic to
production web servers" whereas at XYZ LLC it might mean "Can develop UX
designs in Sketch to customer specs and then code them with HTML, CSS, and a
JavaScript framework along with writing the REST endpoints into which back-end
developers will connect business logic from myriad systems based on the
contract the 'Full Stack Devs' specified for the needs of the UI."

Just a thought.

~~~
vkjv
> "A human being should be able to change a diaper, plan an invasion, butcher
> a hog, conn a ship, design a building, write a sonnet, balance accounts,
> build a wall, set a bone, comfort the dying, take orders, give orders,
> cooperate, act alone, solve equations, analyze a new problem, pitch manure,
> program a computer, cook a tasty meal, fight efficiently, die gallantly.
> Specialization is for insects."

-Robert A. Heinlein

~~~
learc83
Remember that in-universe those words were spoken by an arrogant super human
who was hundreds of years old.

------
jupp0r
The author didn't get DevOps (so don't many companies). It's not about putting
developers on-call, it's about taking operational concerns into account early
in the design of a piece of software and acknowledging that it's going to do
much better in the wild when the people building it are also accountable to
making it easy to operate.

Blaming DevOps (or full-stack, for that matter) for a toxic company culture is
just too easy.

~~~
habeebtc
Exactly so. I'm involved in a DevOps transformation at my company, and have
been exposed to the topic over the last few years, including attending the
Delivery of Things conference in 2016.

What I've learned is that:

1\. DevOps is simple to explain

2\. For some reason, everyone has their own idea on what it means, and most of
them are wrong

Key points from my perspective:

1\. DevOps is an evolution of Agile and so all agile practices are a subset of
DevOps. This point obviates the need for any points past number 2 below.

2\. It's a process which is supposed to get operational knowledge involved in
the development process much earlier to build it right the first time

Its over aim is to combat the fact that you can be proficient in only so many
things. As you become more specialized in one area, your time becoming
proficient in other areas suffers. So a pure dev is unlikely to know much
about QA or Hosting, but that domain knowledge would be valuable for them to
have in their daily duties. So you try and implement a process by which they
have regular access to where that domain knowledge lives, and can make use of
it.

It's really just a way of trying to solve the "Jack of all trades, master of
none" problem. Put another way, it shouldn't be shuffling more duties onto the
dev to save money on headcounts, but a way of increasing the value (not
volume) of a Dev's already high-value work.

~~~
delaaxe
Tim Ferris has something interesting to say on "Jack of all trades, master of
MANY"

[https://tim.blog/2007/09/14/the-top-5-reasons-to-be-a-
jack-o...](https://tim.blog/2007/09/14/the-top-5-reasons-to-be-a-jack-of-all-
trades/)

~~~
rhacker
Absolutely agree, And: [https://www.facebook.com/notes/kent-beck/paint-drip-
people/1...](https://www.facebook.com/notes/kent-beck/paint-drip-
people/1226700000696195/)

The Paint Drip concept has been around for a while now, so I'm surprised
people still reference "Jack of all trades, master of None" especially in the
software dev industry.

------
rhacker
" If there's a particularly nasty issue that seems to require deep database
knowledge, you don't have the liberty of saying "that's not my specialty," and
handing it off to a DBA team to investigate. Due to constrained resources,
you're forced to take on the role of DBA and fix the issue yourself."

Wow, I'm sooo glad we've come as far as we have. I recall a time where a DBA
was literally in charge of creating schemas for a product. Just about
everything this person is describing in the article rubs me completely the
wrong way. I am all about everyone who can help to try helping. All hands on
deck. This article is pretty backwards.

------
jwitko
This article was written in 2014 but I can personally attest from my
experience it is still just as relevant and true. As the owner / operator of a
DevOps As a Service company I see all too often people trying to replace
dedicated infrastructure and cloud architecture experts with "Someone who has
more of a 70% developer 30% devops background". It's unfair to the person
being hired with unrealistic expectations and typically damaging to the
company in terms of lost time and productivity.

~~~
HillaryBriss
i know of a company that has defined a position where a single person designs
and sets up some technical equipment for two totally different departments led
by two different, competing bosses.

this job-holder is routinely overworked and faces extreme demands on their
time, requiring on-site presence both early in the morning and late at night.
the job-holder must mediate interdepartmental conflict (two bosses). and
there's no path to promotion.

the company keeps losing people in this position. it keeps casting a wider
net, bringing in people from further and further away.

this is a solid, successful company with a reputation as one of the very best
places to work in the county. but it has never bothered to redefine the
position into two or more separate jobs. it's just churn and burn.

i'm not sure what the moral is. managers exploit the labor pool until they
can't any more. then they look for another labor pool. if that doesn't work
they whine and complain that they "can't find workers" to fill their jobs.

------
lmilcin
The issue is that there are successful people who can all those at the same
time. Unfortunately, it requires a lot of study and varied experience. A
newcomer can become passable Java developer within realistic timescales, but
will require years of experience to be able to contribute at an organization
that expects him to perform all those roles.

I have 20 years of experience in IT, 7 as admin (ops), 13 in development. I
still spend over half my time on purposeful learning and sometimes find mysef
out of depth, catching breath. I don't envy noobs who will have little chance
catching up.

Most of the time I hear devops it in reality is a person doing hopefully good
job in one area and passable in another and relying heavily on others for
everything else.

------
mamcx
> Good developers can be passable DBAs if need be

This assume a developer is not a DBA. Or as if you can't be a good one.

I start with FoxPro. So databases is always first for me... to the point that
I "mockup" the app with the database schema first.

So, I'm saying that for some people the database -> app is a straight, natural
path. DBAs in the past was more about control the access to a central BD in a
MAINFRAME. Modern RDBMS are very easy to administer and optimize.

In fact, I think that learn Java, .NET or C++ is far more harder than learn
how use a RDBMS in full...

------
swalsh
I think this is essential, and it will progress a bit further. As
infrastructure has become automated, and QC has been rethought in response.
The developer has been able to acquire those roles for improved development
time. However, as coding becomes easier, and more people learn to code. I
think another role, the product manager, will acquire the developer role (and
all the positions previously acquired underneath that). So ultimately the
roles of analysts and product managers will be extremely tech-savvy people who
understand their respective businesses extremely well.

Today, a business analyst has a small idea. She presents it, then gets
approval. A product manager works with her on the details to integrate it into
the product or operation, and a project manager inserts the idea into the
development backlog and prioritizes it. In a few months a developer gets
around to the task, spends a day building it, and deploys right away, and then
the analyst can measure if it has value. In the future, the process could be,
a business analyst has a small idea. She presents it, with a plan on how to
integrate it, and get's approval. Then she spends a day or three building and
deploying it and then measures if it has value.

We've been optimizing Development -> deployment, but in reality what the
business cares about is Idea -> Deployment.

~~~
s-shellfish
> However, as coding becomes easier, and more people learn to code.

The problem is coding WILL eventually become hard again. If things are
developed too rapidly, some aspect of development is going to accumulate
error, that's going to be a bottleneck eventually when it comes time to
automate whatever architectural layer is being 'left behind' as
permanent/vital/intrinsic to system functioning.

It's easy to lose the forest for the trees when so many 'radically novel'
tools keep popping up as 'solutions'. There are bugs in all of those things
and they will accumulate eventually to the point that the 'ease' of
development comes to a screeching halt. The bugs may only wind up surfacing in
'ideas', like the idea that one can forever hop skip jump from idea to
deployment. Somewhere, something is always being sacrificed as an expense of
getting something straight to market. (Soapbox: The sanity of developers gets
relegated as an innocuous and trivial side effect. Just throw more money at
it, right?)

------
xellisx
99% of the 'DevOps' stuff that come across my job search is operations that
deal with Azure/Google/AWS management, along with the the rainbow CI/CD tools
and has nothing to do with doing actual software development. One head hunter
that called today was looking for a "DevOps with Site Reliability
Engineering", which entailed dealing with data center electrical stuff.

------
lhopki01
I particularly love the idea that only developers are talented enough to
preform all the roles but no one else is. There is a hierarchy of skill in
organisation with developers at the top of it.

------
arwhatever
I think in a few years, if the complexity of cloud infrastructure management
is reduced, and its level of abstraction raised, it might make sense to me
developers do this work.

But if I'm trying to build some software app in a complex domain, and the
infrastructure tooling requires me to configure network routing rules, CIDR
blocks, etc., that seems like a huge failure to me, almost like having the
developers grow their own crops for sustenance or something, rather than
having people specialize in that and leveraging division of labor.

~~~
scarface74
How so? I just had to setup security groups and subnets to make an
implementation I was starting work. Why would I want to wait on the
“infrastructure guys” to do it when I could set it up myself in the dev
environment test it, create a cloud formation template and if I needed to, run
it through the other environments.

~~~
arwhatever
Your brain has limited capacity. You didn't learn about the security group and
subnet configuration in zero time, and most particularly not cloud formation
templates.

Meanwhile, both the infrastructure _and_ application development frameworks
are both getting revved and changed out from under you frequently.

~~~
scarface74
One reason I really haven’t attempted to keep up with front end development
outside of JavaScript/CSS/HTML is because things _change_ so much. On the
backend, things don’t change nearly as often, but you get additional features.
Your previous knowledge is not made obsolete.

~~~
arwhatever
Part of my perceptions were likely shaped by my previous employer, which was
completely schizophrenic, technology-wise. We were never allowed to complete
anything, and I think our performance was measured on how quickly we could
become 30% proficient with some new technology stack before our work-in-
progress would be tossed out and we were redirected to work on something
totally unfamiliar. Rinse, repeat.

It really made me yearn for just a year or two of competence and productivity
on some set task in between having the rug pulled out from under us.

------
HillaryBriss
_If a developer makes 100K a year, you can pay four developers 100K per year
to do 50% development and 50% release management on a single, two-person task.
Or, simply hire a release manager at, say, 75K and two developers who develop
full-time._

is this a realistic cost savings analysis? my impression has been that
companies don't hire four developers to do a two-person task. they simply hire
one "full-stack developer" to do everything. problem solved, money saved, i'm
a great CEO.

~~~
scarface74
They have tried that with me before. I was fortunately able to walk into the
CTOs office and get a budget to hire more people.

------
AzzieElbab
The problem with DevOps and other vaguely defined concepts is that the
interpretation is entirely up to your management. On one hand, heavy
investment into automation and having devs involved in production ops is a
good thing. It makes things go faster and puts devs skin in the game. On the
other hand, when your manager thinks that DevOps is about having senior scala
devs fix printers, you might be in trouble

------
lmm
Task-switching is expensive, but human-to-human communication is more so. As
one of those "bright" developers, I'd far rather just do the deployment myself
than hand it over to someone less skilled so I can focus on doing more coding,
because that's only going to mean I then get interrupted when there's a
problem with the deployment.

~~~
dsr_
When the company is five people, doing the deployment yourself is a necessity.

When you have a dozen developers working on the same project, doing the
deployment yourself is a danger to the company.

~~~
lmm
> When you have a dozen developers working on the same project, doing the
> deployment yourself is a danger to the company.

How and why? There may come a point where it's worth segregating deployment
from development, but I'm certain it's a long way north of a dozen.

~~~
workinfunk
Really? You put a dozen people on the same _project_? I find it hard to
believe that your project is so unique that A) requires 12 devs and B) is also
deployable by all 12 of them (without relying on automation) in a way that
doesn't throw off any of the other 12.

~~~
lmm
I don't know what you mean by "relying on automation" or "throwing off the
other 12".

Deployment is mostly-automated - put a note in our chat channel topic that
you're deploying (or queue if someone else is already deploying), push a
button to do a release and deploy as staging (and a small smoke test), do any
manual testing (or investigation of logs etc. if something goes wrong), push
another button to flip that deploy to be prod.

Any of the 12 devs can do this (and does, we each deploy our own
features/stories), any of the 12 devs can make changes to the deployment
scripting just like with any other code area (some will be more or less
familiar with them, but that's true of any area of the codebase). Nothing
unique about it, quite the opposite.

~~~
workinfunk
If your deployment is automated, your developers aren't doing the deployment,
the machine is, and the argument is moot.

~~~
lmm
Developers automate things, that's their whole job. The point is that we don't
have people in dedicated ops/sysadmin roles, and are better for it.

~~~
workinfunk
I believe we are violently agreeing at this point.

------
linsomniac
I used to avoid the DevOps title and called myself just straight Operations.
But recently, and I mean within the last week or two, I realized I'm squarely
in DevOps territory.

The system I Op runs really smoothly. Problems are infrequent. So instead of
spending my time fighting fires, I'm instead working on our next generation
infrastructure, running experiments with Kubernetes and Kafka and Influxdb and
Elasticsearch.

I'm not developing on our main product most of the time. But I am developing
our infrastructure.

As far as the totem pole with developers at the top, able to do all the other
jobs? That has not been my experience. In 30 years, working with probably 100+
companies large and small (for 18 years I did contract SysAdmin), I'd say that
the developer who even thinks about ops is extremely rare, less than 10%.

I agree that DevOps is more important for smaller companies, and that
specialization is useful. I haven't seen evidence that supports the totem pole
portion of that post.

~~~
jiveturkey
You aren't DevOps, you've misunderstood it. You are an SRE, give or take.

~~~
linsomniac
That's what I always thought, but other people I've been talking to recently
have been calling it DevOps, because they have it on the development side of
the house and deliver it to ops.

------
scarface74
The more I think about it, how can a modern developer not know devops? How do
you communicate with the devops folks? When I’m fighting software, I create my
build scripts (currently a yml file for AWS CodeBuild) as part of the repo. I
don’t use Docker but wouldn’t the developer create the build streps for the
Docker image?

When I am working on an implementation that can be deployed with
CloudFornation,I create the template test it and hand it over to the Dev ops
guys. Even if I’m not doing the actually deployments to any environment
besides Dev. I still need to know how to.

The same is true for database deployments and upgrades. How do you upgrade a
database through CI without knowing SQL?

------
394549
> The increasing scope of responsibility of the "developer" (whether or not
> that term is even appropriate anymore is debatable) has given rise to a
> chimera-like job candidate: the "full-stack" developer. Such a developer is
> capable of doing the job of developer, QA team member, operations analyst,
> sysadmin, and DBA.

Is this really what a full-stack developer is understood to be? I'd always
assumed it was just someone who was comfortable doing both frontend and
backend development on the same project.

------
arwhatever
You lose a lot of efficiency when you spread any particular software
development work beyond one person's head.

But you lose a lot of expertise when you spread your developers too widely,
technology-wise.

~~~
scarface74
I _can_ do all of those roles at a decent level of proficiency and I’m paid
accordingly. But, as the article mentioned, why would they want to spend the
money on my salary when they could hire someone a lot cheaper? I like having
the choice of being able to do it myself so my development is not being
blocked by someone else but not having to do s lot of the mundane work.

------
hashkb
Groan. Instead of a rising tide lifting all boats... some devs insist on being
divers. Nobody is turning down amazing specialists, but if you aren't a 90th
percentile specialist you should probably know a little of everything.

~~~
droidist2
Knowing a little of everything is good, knowing a lot of everything is
unreasonable. Sure there are those amazing people but expecting everyone to be
them is crazy.

------
arwhatever
Was the industry so pleased with the timelines and reliability of software
projects a few years back that it seemed sensible to heap another career
field's worth of complexity onto the developers?

------
Itaxpica
I had to stop reading at the totem pole part when it became clear the author
has no idea what the hell he’s talking about. This is basically weaponized
Dunning-Kruger.

~~~
xaedes
But then the author only used the wrong name for that what he is actually
talking about. The thing he talks about is a real problem that occurs...

~~~
Itaxpica
I’m not talking about the name he used, I’m talking about the content.
Insisting that a dev can be a DBA, or a QA person, or any other support role
is a great way to have an extremely shitty DBA, or QA person, or support role.
It’s the kind of thing that only a deeply self-centered developer with no clue
of the world beyond the tip of his nose would ever say.

------
madmulita
Where are those developers that can do my job? Ours certainly can't even do
theirs. That is why we are moving to the cloud: "see? we didn't need to learn
how to optimize dynamically adding resources almost solve our problems".

------
sagichmal
This article is a defense of entrenched mediocrity.

~~~
dang
Maybe so, but please don't post shallow dismissals to HN (especially of
someone else's work, as the site guidelines say:
[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)).

If you have a substantive point, give readers enough substance so we know why
you think that and can maybe learn something.

------
rs86
Zero informative content. Hey!

------
workinfunk
I love how he characterizes what developers do as "just writing code" \-- the
whole article is basically a whinge: don't make me do ops, don't make me write
tests, don't make me think about the problem, don't make me re-use existing
solutions, don't make me pick the right tool for the job, don't make me use
sustainable processes -- _I just wanna "write code"_. Code is basically the
poop that results after you digest the problem. The best code is as little
code as possible, but this guy just wants to make code poop for a living.
Sorry guy. Time to grow up.

~~~
mmt
> Code is basically the poop that results after you digest the problem. The
> best code is as little code as possible

This is, perhaps, one of the best metaphors I've read for explaining this
concept.

As someone who does one of the "lower" jobs, I rankled at his "hierarchy" of
roles notion. Never mind that I probably _could_ write application code if so
inclined (and do routinely write "infrastructure" code and sometimes have had
to _fix_ those developers' code in production).

I occasionally like to point out to "engineering" or "technology" [1] managers
that not all problems can be (or are best) solved with software/code. Some of
use, when we digest a problem, can poop out zero or even negative code.

[1] In quotes because they're nearly always actually only _software_
engineering or technology managers.

------
workinfunk
Author doesn't understand the DevOps skillset. The average developer can hack
things together, but they don't have an appreciation for operational
principles. DevOps is actually higher on the totem pole than dev, because
DevOps requires you to be skilled in both operations and dev. A traditional
ops person couldn't cover for a dev, but neither could a regular, even "full
stack" developer stand in for a sysadmin.

------
yumaikas
Can we get a (2014) on this?

~~~
dang
Late, but it's there now. If you want a more timely response it's best to
email us (hn@ycombinator.com) because we don't come close to seeing everything
that gets posted here.

------
justaj
Mods, please add (2014) to the title.

~~~
dang
Done now.

