
Confessions of a Full Stack DevOp - brianshumate
http://www.ansible.com/blog/confessions-of-a-full-stack-devop
======
simonw
Pet peeve: this URL is incorrect. When you're linking to an article on a blog,
it's important to use the permalink for the specific entry (in this case it's
[http://www.ansible.com/blog/confessions-of-a-full-stack-
devo...](http://www.ansible.com/blog/confessions-of-a-full-stack-devop) which
admittedly wasn't easy to find - you had to click the heading on the article
which lacks any clear visual way to tell that it's a link).

If you link to the blog homepage e.g.
[http://www.ansible.com/blog](http://www.ansible.com/blog) then in a few weeks
time this post on Hacker News will no longer link to the correct content.

~~~
dang
Thanks. We changed it.

We'll soon have a new way for users to suggest better urls, like you did, that
is more reliable than a moderator happening to see a comment. I'll announce it
as a Tell HN.

------
pradocchia
_The last time I looked for a senior sysadmin -- less than a year ago -- I
didn 't get anyone who was comfortable programming in Perl/Python/Ruby until I
started using the term DevOps._ [1] via TFA

Ah, now I understand DevOps. Well, the sysadmin who can personally automate
his job through programming--this is very productive person.

[1]
[https://news.ycombinator.com/item?id=7593681](https://news.ycombinator.com/item?id=7593681)

~~~
p8952
In my personal opinion this is what DevOps is, in fact "OpsDev" might even be
a better word for it in most cases.

It's about carrying out what are traditionally "Operations" tasks
(Provisioning, Configuration, Orchestration, Monitoring, ect) using tools and
workflows with are traditionally used by Developers (Version Control,
Automated Testing, ect)

In large companies this work is often carried out by operations people who
have expanded their skill set to include things previously limited to
developer roles. In startups this work is often carried out by developers, as
in the original rant, who are tasked with maintaining infrastructure as well
as developing.

~~~
beat
In the traditional development/operations divide, developers often have no
access to and little insight into the production environment where their code
will be run. As such, they cobble together whatever makes it work, and then
"throw it over the wall" to operations, where it has to run in a _very
different_ environment. This makes for hellish manual deployments, done on
multi-day phone calls with ops staff who don't understand how the code works
talking to developers who don't understand the rules and procedures of ops.

If you see places where production code has to run in WebSphere or Weblogic
because it's "enterprise", but want developers working in Tomcat to save
money, _run away_.

This leads to situations where operations starts providing "developer
resources" in terms of servers, dba support, maven repositories, etc, that
turn into a tar pit where development gets stuck in incomprehensible
bureaucracy, and "hack" around the system in order to actually get anything
done. And then security or someone else in ops finds some little skunkworks
dev system and has a fit because developers don't understand what they're
playing with - or worse, they find out about the skunkworks because they get
pulled in to fix what developers screwed up, with their own managers howling
about who's gonna pay for it...

So to me, the beauty of DevOps is getting dev and ops working together in
concert from the start to create a consistent development/production
environment, where the same automation can be used to maintain both. This is a
win for everyone.

~~~
secstate
And this is why DevOps has gained traction. Automated deployments, continuous
integration, and software that functions identically whether in development
env or production.

You can wax poetic about how it used to be, or complain about people being
overworked all you want, but that's an issue of expectations. Personally I
never want to go back to two week "integration" processes and multi-day manual
deployments.

DevOps to me is an exercise in not making things more complciated than they
have to be on the config/deploy side.

------
beat
This is a much better, more sensible article than yesterday. My good takeaway
here is that DevOps means whatever you think it means - it's a lot of
different things to a lot of different people. That's a much more positive
spin on my frankly negative opinion that it's devolving into a meaningless
buzzword to attach to resumes, recruiting, and PowerPoint presentations for
management.

I know I came in to DevOps (the word, not the job - I've been doing the job
since the mid-1990s) with a strong opinion of what it _ought_ to mean - cross-
org cooperation between development and operations. This article has me
rethinking that a bit.

~~~
cwp
Yeah, the word is only meaningful when the speaker and listener have enough
shared context to pin down the meaning. In that situation, there's probably a
better word anyway.

It's not quite the red flag that "ninja" and "rockstar" have become, but it
definitely makes me wary when I hear it.

------
volume
Here's my unsolicited opinion:

Dehaan is pretty much spot on. Knupp's post seemed to first latch onto the
idea that devops is about devs doing ops works. To add, I think Knupp should
do a rewrite after listening to some devops cafe and food fight podcasts, if
he really believes in what he wrote.

It's somewhat an unfair comparison though. If you're the author/creator of
ansible you surely have spent a lot of time thinking and understanding what
devops means. To be frank, I cannot tell if Knupps post is just link bait to
sell his book.

The concept (and value) of the devops mindset & culture is less debatable in
2014 than in the past 5 years. Those who latch onto the totem pole model will
simply get left behind while the rest of the industry moves on... or at least
that's what they tell me!

------
_superposition_
Knupps idea of the totem pole rubbed me the wrong way too. I've met great
developers who didn't have a clue about how to scale or monitor their software
outside of their framework. How to properly deploy across DEV, UAT and PROD
without making code modifications. How to keep these environments clean and
flexible to enable these deployments. Nor should they have to. Call me a Dev
because I read and write code. Call me Ops because I do deployments and fix
production issues. At the end of the day I solve problems and just plain make
stuff work. With or without so called DevOps tools.

~~~
billmalarky
You've met _great_ developers who couldn't deploy code?

~~~
_superposition_
Sure. I've met great ones who could too. I don't think deployment is part of
the development job description. Or is it?

~~~
cmelbye
To me, it seems like it makes sense that a great developer should absolutely
know how his code is running, how his changes interact with the production
environment in which the code runs, etc.

~~~
marcosdumay
You'd take a hit if you visit any big company that's not centered around IT.

It's quite common for the developers to be forbidden from knowing the details
of the production environment, and usually they have no easy way to get that
info. But don't mind, a DBA will think about the interaction between the
several programs, and the ops people will think about performance.

------
geerlingguy
> _When done correctly, automation tools are giving [developers] time back —
> and helping out of this problem of needing to wear many hats._

I've found this to be the case both in my full-time job (enterprise/healthcare
industry) and in my larger side projects. Using some simple tools to make sure
both server environments and deployments are uniform/repeatable/known in a
good state has given me much more time to focus on actually _developing_.

Additionally, there are times when certain features (especially surrounding
performance and scalability) _requires_ a deeper knowledge of part of your
stack. Being a good developer requires you to be a specialist (in your primary
language/tools) and generalist (in almost all other areas) at the same time.

------
nraynaud
I would also add that you just don't automatize stuff. Because the more you
add layers, the more you'll have to dig into at 3AM with the customer
screaming at you on the phone. It's using the complete stack and using the
least technology required to do the task at hand. While having access to the
whole picture.

I have heard too many times that something is complicated then we can
automatize it. But automatization has to be debugged, so, if the task is not
complex per se, try to find a simpler tool, try to limit your intervention in
the config files, try to limit the number of tools etc. But you can make those
tradeoff across the whole stack. And suddenly you're balancing between adding
a library to the code or a program on the machine, or a function package in
the DB. You have a wider view, and you have the pressure to do the right thing
because your phone is connected to new relic.

~~~
beat
The "Let Them Eat Cake" antipattern, that is. Maybe no one has written it and
I just made it up...

~~~
nraynaud
I don't get it. Google turned that up:
[http://robmyers.org/2009/09/21/free_software_debate_antipatt...](http://robmyers.org/2009/09/21/free_software_debate_antipatterns/)
and I don't get it either.

~~~
beat
Basically, the idea is that if something is hard to automate, then automate it
and then it won't be hard to automate. Peasants have no bread? Let them eat
cake!

I've encountered this a lot in the enterprise. Baby-step automations aren't
allowed because they don't contribute to a Grand Unified Theory of Automation
that will Solve All Our Problems. Worse, you wind up babysitting antiquated
automation that should be replaced, and you're not allowed to replace it
because it's already there and already works.

------
peterbotond
This is pretty much a good article on DevOps. In my words i see DevOps as a
bridge spanning the kernel, systems, networks, qa, packaging, and deployment.
for a highly optimal system all these need to be planned for with care and
attention. DevOps is a tough and long learning to be good at.

------
RA_Fisher
Great post. While I agree with this, "Automation shouldn't be about spending
all your time with a tool ..." I wonder if employers are pressured to push you
into more and more Ops work by this paradox?
[http://en.wikipedia.org/wiki/Jevons_paradox](http://en.wikipedia.org/wiki/Jevons_paradox)

This kind of situation, specifically (relative to other work):
[http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsPar...](http://upload.wikimedia.org/wikipedia/commons/b/b8/JevonsParadoxA.png)

------
rch
Back when I thought I was going to be a Mechanical Engineer, I had a summer
course that included surveying along with machine design and robotics. I felt
the latter were more relevant to me, but had I gone on to work in oil fields
in SE Asia I probably would have appreciated my time tromping around campus
taking measurements.

I see all of this as just 'engineering' work, but it's easier to delegate
certain things than others. When I ask someone new to configure Postgres for
secure remote administration, script or otherwise automate backups, and rough
out a schema for some tool we're thinking about, they'll likely accomplish at
least a subset those tasks. The remainder is an opportunity for discussion.
Someone else might just ask me to build something, and we'll talk about
usability when he's done with the physics side of things.

But if I leave, what does that physicist look for in a new hire? The skillset
of an 'engineer' is frustratingly broad and vague from a hiring point of view.
Getting so-called DevOps out of the way might give another new hire the chance
to develop as a programmer, and vice versa. Hopefully one or both of those two
will eventually develop into an engineer.

------
KaiserPro
I am a horseshoe crab. I am a sysadmin.

I can code, I do code, but I'm a much better sysadmin, mainly because I've
been doing it as a dayjob. You want networking? I'll do that. you want multi
gigabyte a second storage arrays? I'll do that. You want monitoring? I'll do
that.

You want me to recompile python 3.3, naaa you can do that. I can, but you'll
do better at it

~~~
lewaldman
Wow mate, don't get me wrong (I'm a SysAdmin/DevOps guy) but...

From where I came and from my background... RPM/DEB packaging is a SysAdmin
job.

And... In a properly built infrastructure, what is recompile python 3.3 if not
just repackage it?

~~~
KaiserPro
because unless you work in a startup you have to support legacy, which
involves recompiling python(or anything else for that matter) because someone
previous did something stupid....

------
xorgar831
DevOps is really the traditional Unix admin role, however the operator role
has now be renamed sysadmin in most companies.

------
Thaxll
"Full Stack DevOp", worst three words of 2013 in the same title.

~~~
rch
It's just a euphemism for 'engineer'.

~~~
chris_mahan
But you can't be an "engineer" without a Engineering degree from an accredited
university, so DevOps.

~~~
rch
Unfortunately, the profession is leading the credentialing at this point. I'd
expect that to change in the next decade or so, ideally resulting in a useful
PE designation (in the US anyway).

------
ghalusa
I can't read this article. The font and spacing is horrible. Makes my vision
shimmer and shake. Perhaps I'm just getting old.

------
michaelochurch
DevOps and "full stack" and Agile and the flamewars around them are just
bikeshedding. None of this gnashing of teeth actually matters.

You know what matters? Writing code and making it work. Solving problems.
Helping people. If the right way to do that is for me to make a system more
reliable, then I'll do my best even if I'm not interested in being "an ops
guy". If the best thing for me to do is read machine learning papers, then
I'll do that.

The real problem: technology companies have terrible leadership. You know how
small towns in the Midwest used to send their genetically unfortunate to San
Francisco, where there'd be more social services and a warmer climate? Well,
that's what the business world does with its rejects, too. Buses 'em to San
Francisco. Except, in tech, they end up making $400k as VPs of HR at startups
and get to boss nerds around.

"Agile" won't fix that. Nor will "DevOps". These terms mean too many different
things to different people, and ultimately fail to address the real, core
issues with technology.

