
How 'DevOps' is Killing the Developer (2014) - zippy786
https://jeffknupp.com/blog/2014/04/15/how-devops-is-killing-the-developer/
======
scurvy
I've never met a developer that could setup a MPLS circuit or OSPF over an
IPSEC tunnel. The author says that developers can do these things, but I
highly doubt it. Let alone be proficient enough to do it in JunOS and IOS.
Network operations should not be lumped in with devops.

~~~
zamalek
> The author says that developers can do these things, but I highly doubt it.

Being able to do something from memory is something we only care about in high
school. Take me, for example: did my router support L2TP termination? No. Did
DD-WRT support it out of the box? No. What a developer can do is cobble
together information in order to find a working solution. So would I, as a
non-Dev-Ops be able to set up MPLS over IPSEC, having never done it before?
Yes: I have Google and I have a brain between my ears.

~~~
devnonymous
As a sysadmin in a prior life, I see this as the equivalent of a non-developer
(for instance, a sysadmin) asking can I create a site scraper using google,
curl/sed/awk/..bash, copy/paste and a little bit of of brain between my ears ?
Yes. As a developer you know there is a lot my subtlety to programming than
that.

I think what @scurvy was driving at is there is a lot more subtlety to network
ops than developers would be willing to acknowledge.

~~~
scurvy
Yes, indeed that was my point. While the two examples I gave were easy to
Google and get a "functional" setup going, there are enough things there to
blow up in your face if you just go with the StackOverflow/ServerFault
answers. You're going to run IPSEC, which DH group are you going to use? What
cipher? How often are you going to rekey? Should rekeying be based on bytes,
time, or both? Your Juniper box is connecting to a Linux tunnel server. What
should your inner and outer identities be? I haven't even come to OSPF yet...

While just an example, it shows that our professions all have subtlety and
nuance to deal with. You gain the ability to handle this through experience,
not Google. Throwing a "full stack" developer against every problem is
definitely not the solution.

------
orthoganol
The trend OP doesn't like is primarily because 'full stack' responsibilities,
DBAs, release coordinators, etc. is probably the easiest it's ever been, so
you might as well have developers throw in some extra hours to do it. I've
worked at a fortune 500 that used a Ruby gem called "lol_dba"... you can spin
up environments and deploys trivially with PAASs, or at the least use various
automation tools, scripts, and cookbooks for provisioning machines, running
deploys, CI, etc.

OP seems to think "full stack" developers no longer devote time to writing
code... "This is why we see so many developers that can't pass FizzBuzz"...
but really it just doesn't take us long to do the non developer stuff because
it's just easy these days, maybe he's not aware of the tools people use.

------
keyanp
Interesting read, but I wish it wasn't so inflammatory.

> "...release coordinators and the like are at the bottom of the totem pole.
> Why is it arranged like this? Because each role can do the job of all roles
> below it if necessary."

I find this to be belittling those who choose to work on a different part of
the stack than the author. It seems to suggest there is an innate incompetence
that keeps release engineers from functioning as "developers". That couldn't
be further from the truth.

Release Engineering @ Mozilla:
[https://www.youtube.com/watch?v=7j0NDGJVROI](https://www.youtube.com/watch?v=7j0NDGJVROI)

~~~
falsedan
Release Engineer here too.

FUNNY STORY: we make the developers act as release coordinators, because it
distracts us from the work of improving the release process by replacing it
with automated software systems.

The top of his hierarchy should be PMs/executive team; betrayed by his own
developer ego…

~~~
serge2k
It's a hierarchy of skills. PM and exec skills are almost entirely orthogonal
to tech skills. Putting them in this hierarchy makes zero sense as a result.

~~~
lowmagnet
Because understanding the business's and customers' needs is somehow "beneath"
developers in skill or status?

Sounds like some people need to try the "less technical" parts of the stack
like requirements gathering to appreciate their roles.

~~~
zaccus
"Orthogonal" means independent from, not inferior to.

------
endymi0n
I always like the band analogy: The guitarists / developers always think they
can play the bass guitar, because it's basically the same, just with less
strings.

Yet, they don't get that it's all about groove, rhythm and dependability and
are just way too unrelaxed and attention seeking for doing it right.

I'll take a grumpy BOFH that wires together my AWS regions in his sleep and
doesn't make costly mistakes - because he measures twice and cuts once - over
some kid trying DevOps any day :)

~~~
harryf
I remember once haranguing a group of Java developers to use the log4j's
RollingFileAppender. No matter what arguments I used and despite many nods and
promises, on live systems they just kept on using the default FileAppender
like there's no tomorrow, despite it being a relatively trivial config change
to switch to the RollingFileAppender.

Only the sysadmins "got it" because they were the ones that got to feel the
pain of cleaning up a NAS filled up by log files.

~~~
mhaymo
To my mind the whole point of DevOps is to make the developers "get it" by
making them feel the pain when they get things wrong. Developers make
decisions about the system every day, so it pays for them to understand how it
is maintained and operated.

~~~
digi_owl
My impression is that devops instead has turned into a means for getting
sysadmins out of the way so developers can use the latest shiny in
production...

------
jlawer
I am an "Infrastructure Engineer" by job title, but really I am a jack of all
trades. Many people working in small business need to wear many hats.

You can be deep on a single area (specialist) or be spread out (generalist).
Larger businesses want specialists, small business want generalists.

Personally I am a generalist. I deal with security, servers, routers,
switches, applications, laptops, Operating Systems, Backup, Databases, etc. I
write code in python, java and starting to get better with go. Its a handful
and I am always juggling things, but I also think that broad knowledge helps
things. I understand TCP/IP because I wrote a TCP/IP stack. I feel I have a
better understanding of the complex systems as I have a non trivial
understanding of each part.

Its a very different working situation, but it has its advantages and I dread
ever going back to a large corporate because of it. It isn't as efficient
compared to having a team of specialists, where everyone can spend 100% on
their speciality, but on the other hand, you can often do things with
surprisingly small teams.

~~~
einrealist
I share your opinion. In agile teams, it is important to do things that work.
If a team owns a stack sucessfully, there is no need to change that. But if
the environment changes (e.g. growth), the team may decide to split /
specialise in order to ensure that the business goal is not in danger. There
is no silver bullet.

------
slowmovintarget
Yarg!

 _Teams_ do DevOps. It isn't supposed to be an excuse to make developers do
everything. It's supposed to combine operations and development _on the same
team._

This is just like the perversion of agility in development by "Agile."

This, too, shall pass.

~~~
phamilton
I recently was part of a team that properly practiced DevOps and as soon as it
started working we had a reorg and it was broken apart.

If the premise of the article is "Developer time is precious, don't make them
do grunt work" the I would counter with "Developer momentum is precious, don't
let developers get blocked by operations/QA/release."

On our team, we had Ops and QA. But I needed something done and they were
already working on something then I could have a quick 15 minute chat after
standup about the Puppet change I needed to make and then go do it myself. An
hour later I'm done, the change is deployed and I'm moving on to the next
task. Now, without an embedded Ops resource on the team, I have to file a
ticket and pray someone picks it up within a few days.

~~~
lowmagnet
The way to not get "blocked by QA" is to have automated tests and QA people
who can write them as well as dream up scenarios to break your toys. They are
also able to do code reviews in your stack as well.

------
mattkrea
One coworker posted this a while back and I just dismissed it as someone
afraid of something that they can't handle.

While I understand the point.. I'm also a dev and I love "DevOps". Aren't we
all doing this because we love problem solving? How is this not just another
problem to tackle? I love having to figure how to automate mundane tasks
(again.. this is just code.. not much different than we write elsewhere) to
make sure I don't have to manage any systems manually.

/rant

~~~
nstart
It's actually a complex debate. We as developers naturally gravitate to a role
like this because it's in our nature to believe that we can solve all the tech
related problems. And all other problems, we believe can be solved with tech
and therefore we believe we can solve everything. Ok. That's an exaggeration
but there's quite a large shred of reality in that statement. Being
fullstack/devops/<insert fancy term> satisfies that craving to jump on to new
problems that we developers have.

And that CAN be a problem. It isn't necessarily always a problem. But think
about it. How many of us prefer jumping up to try working on a brand new
project. Shiny stuff everywhere. And then everything gets abandoned at the
grind and polish stage. The enthusiasm is gone. We crave the next new problem
that we can use as an excuse for jumping away from the grind we are in. I
think what the author is trying to get across is that this new "devops"
culture is creating a large number of developers who don't have deep knowledge
of anything. Ask a developer (myself included) to evaluate a set of JS
frameworks for a particular project which has somewhat specific requirements.
Behind most answers (this is based on observation, myself included again) the
primary argument will be the developer's preference; not an objective
breakdown of what is most suited for the task. We might not see too much of
this on HN (I think. I see a lot of intelligent debates here which I learn
from all the time) but on a larger scale developers are becoming half baked.

And that's basically the other side of what you pointed out. Problem solving,
learning new things is a lot of fun (discovering and configuring vagrant and
docker for the first time was incredibly fun). But it possible that the full
stack role is churning out people with broad but just somewhat in depth
knowledge. Tough call. Question is, is somewhat deep deep enough for most
everything?

~~~
mattkrea
I can agree with that.

Though on the other end of things.. I have a friend that is very into how some
languages are optimized for various tasks and he'll mention to me that X
language does something with their loops for speed, etc.

I don't have that same low level knowledge of the languages (not as low-level
at least; I have some) but at the end of the day X language is slower than
those that I prefer for the type of work we do (APIs mostly) so I don't care
that X language does all these things internally.. and I don't need to know
because it doesn't matter to us (nor our employer) in the end.

So my--hate to use a buzz phrase here but it's appropriate--'get shit done'
style tends to work for me and our company because evaluating tools comes down
to "does a framework or tool accomplish what we need without too much
overhead".

Is this the type of _deep_ knowledge you were talking about?

~~~
nstart
Yep. The question was, is the "I can analyze and tell you if I can use this
tool to get work done" knowledge deep enough for most work? Or is it essential
that if you are a front end dev to at least have an in depth knowledge to
analyze, explain, and predict where a JS framework will shine over another and
where it's pitfalls will lie. Knowing those pitfalls is also important because
someday, it will bite. I remember not knowing about the pitfalls of python and
default parameters being permanently set if given or passed a mutable
argument. That bit me hard. I really don't know where the line lies honestly.

------
bradhe
If you're a developer working on a service your primary objective is service
delivery. It's really, really difficult to do that without understanding
(intimately) all the components in play from top to bottom.

~~~
onion2k
_It 's really, really difficult to do that without understanding (intimately)
all the components in play from top to bottom._

I disagree. If a component is essentially a black box, where I know the inputs
and I can test to check I get the expected output in known conditions, then
that's enough. It has to be, otherwise there wouldn't be any way to use a
middleware, or a third-party API, or anything too complex for one person to
understand.

There are lots of software businesses where _no one_ fully understands the
internals of the product. That's fine. So long as there are people who
expertly understand the part they're responsible for, and the APIs are well
documented, everything will work great.

The problem with devops is that it gives people the impression that everyone
can be responsible for everything, with everyone on the team having a hand in
the design, the code, the testing, the deployment, etc. For a sufficiently
complex product that is just not possible. At some point it gets too big, and
people have to start accepting that they can't do it all, _and that they 'll
have to trust their colleagues to get things right._

~~~
MichaelGG
This works great when it works great. Black boxes everywhere. But then you run
into some issue, like, I dunno, your monitoring service is telling clients the
site is going down, even though you can't find any evidence of this.

If you know how to go deeper then you run a packet capture and start seeing
that, whatdya know, your top-3 IaaS provider's network is shit and is often
dropping packets, so those timeouts are actually occurring, though not on
anything you control. Now you're able to try to setup a repro, then open a
ticket with enough details to get something done. Versus opening a ticket
along the lines of "Hi Top-3 IaaS provider. Some monitoring service says our
site doesn't always respond, but I'm sure it's not me. Please fix your
network, thanks."

Not knowing how things work at a deeper level leads to experienced teams
working on high-profile sites to come up with stuff like: "We have 2 domains
in use, but need to go SSL only. We only have a certificate for one domain. So
please implement a DNS-level redirect from domain A to domain B to avoid
browser SSL errors."

Or an experienced dev insisting that a major vendor is shipping invalid
binaries because the runtime says so. And insists they aren't metadata-only
libs, but isn't comfortable opening up the binaries to verify and thus spends
a day trying to figure things out another way.

Being able to get to a baseline of what's working is a fundamental step in
troubleshooting. Just like you would always ask a home user to make sure stuff
is plugged in, verifying lower-levels of the stack are correctly operating
should be a common step - it's unfortunate how common it is that something
else _is_ broken under the level you're working at.

~~~
onion2k
You're confusing 'needs a deep knowledge of how components work' with 'needs a
deep of knowledge debugging'. A suitably senior developer should definitely be
able to deduce where a problem is most likely to be happening, and figure out
a way to test it.

In the case you're alluding to, it'd be much more useful to know general
information about why any monitoring service might think any IaaS provider is
not responding, and have the skills to test that. Knowing details about the
components you're using (eg AWS and Pingdom) is _much_ less useful.

------
hliyan

       There are two recent trends I really hate: 
       DevOps and the notion of the "full-stack" developer
    

I have to question the wisdom of bunching these two together. One is about
freeing the developer as much as possible from peripheral duties so he can
concentrate on his primary responsibility - design and development. The other,
at least to me, is more about avoiding systems composed of modules that don't
fit together properly, rather than getting more work done for less money.

~~~
krzyk
Not sure about the DevOps, you are talking like if operations guy would start
doing some dev stuff.

But our company goes the other way, they make Devs do Ops stuff and QA stuff
(and bussiness analytic stuff also). I really hate this.

~~~
lowmagnet
Why hate that? The work is part of what is necessary to deliver software. It's
like saying you want to be a general contractor on you own house but don't
want to have architected plans, framers, or a final inspection.

~~~
krzyk
I don't want to be jack of all trades, I really don't have QA mindset and I
don't like software testing as a black box, I prefer unit testing. All of
developers on my team hate when they have to do QA stuff. It is like being
hired to do plumbing, but being forced to do gardening :) It is all on the
same property but completly different tasks and skillsets.

------
kriro
The article seems all over the place. For example I think the conclusion

"""This is why we see so many developers that can't pass FizzBuzz: they never
really had to write any code."""

Doesn't really follow from the previous parts. DevOps is not the reason for
this nor is the full stack developer trend.

The author also seems to start from an implied base premise that developers
wouldn't do devops/fullstack if they had a choice. I disagree with that. Many
developers are very interested in understanding the implications of their code
better (more DBA skills help a ton for example) and quite a few _gasp_
actually enjoy devops or understanding the entire product from start to finish
at least on some level (full stack).

The benefits of DevOps also seem to be completely ignored. It's precisely the
fact that common development best practices are integrated with sysadmin, dba
etc. knowhow which makes it useful. It's not just "let the dev be a sysadmin
in a pinch". It's systematically improve things by automating them (following
development best practices instead of the hacked together bash script
approach).

DevOps and fullstack dev are conflated throughout the entire article in a
strange way which makes it kind of hard to argue.

------
tomohawk
Devops may be a lame buzzword, but most interesting problems cross traditional
boundaries. As a dev becomes more senior and wants to become more valuable,
they'll take it upon themselves to learn more about the other domains they
interact with. The next step is they'll start trying to take on these cross
domain problems that become apparent at this point. What I've seen is that
when devs stick to straight coding, the products they work on tend to be less
successful and their careers tend to be less than they should be.

------
myVocatio
To spoof J L parker, I was "once a developer". Have played many roles since
(PM,strategy consultant, VP, p&l owner,...).

DevOp makes sense to me for building complex solutions..but, for different
reasons than discussed here.

Consider this: a small team (usually 3-7 members) of "jack of all trades" is
proven across industries where the team has to deliver a complex solutions
under a very aggressive timeline ,requiring multiple skills. Two examples from
my personal experiences: Strategy consulting - eg @ McKinsey (work includes
interviewing executives, presentation, excel, data entry, hypotheses
development and verification....) investment banking eg @ Goldman Sachs (work
includes industry analysis, financial analysis, scenarios, presentation,
excel, data entry, ...)

"Jack of all trade" model in terms of DevOps works because:

1) goal for the team w/ "full-stack" skills is to have a reliably working
software solution 2) communication and management overhead increase
significantly as the team size grows 3) complex solutions need different
"amount" of various skills at different point in solution cycle 4) adding/
removing team members comes w/ significant cost to alignment

------
sargun
Personally, I love DevOps. I've played on both sides of the fence. More ops
than dev, I'll admit.

The way I view DevOps is that you have end-to-end guarantees about software
delivery. A lot of this draws from personal experience, so YMMV.

Story:

I was working in an organization where engineering had built a custom solution
with their own database to a problem. It was using their own DB system, with
the JVM, etc.. I was oncall for said service, because their DB kept getting
into a poor state. My only real recourse to a page was (a) escalate to the
devs (b) restart the service. I ended up furiously rewriting the service in
Erlang over a weekend, and ended up getting told that my replacement was too
complicated, and the previous one worked okay, because we could just restart
it.

Take-aways: Without feeling the pressure, and side-effects from being woken
up, and having to respond to on-call incidents for a service, it makes people
put more value on the development time than operational time, although a
service is very likely to be in operational time much longer than development
time.

I have two other stories, but it'll take some time to write, and it's getting
late, so I'll type it up only if someone is interested. Just so I remember -
the tale of LD_PRELOAD and HDFS, as well as the networks and root access.

1\. I think that ops staff should provide shared services to services
organizations (hosting, deployment, etc..) when N services need the shared
service (when N>2, depending on how many resources are available in your org).

2\. I also think that ops staff should be able to "consult" on projects, as
members of those teams during development. This is going to mean that those
engineers are probably going to need to do some more development! Or,
alternatively, the project team needs to be able to backpedal on some
decisions if those ops consultants come in later on the project.

3\. I think dev folks need to have end-to-end responsibility, including
oncall, AARs, and on-call remediation. They need to take feedback from ops
folks and act upon it, just as they would if product told them to do
something.

