
A developer goes to a DevOps conference - fanf2
https://www.darkcoding.net/software/a-developer-goes-to-a-devops-conference/
======
darkr
This is sadly the state of the current mainstream DevOps "movement". It is no
longer anything that could be described as a movement, only as an industry, a
set of tools, and a thing a few years back that CTOs in laggard companies
announce to the board that they need to be doing.

It was supposed to be about efficiency and repeatability, and approaching ops
with a SWE mindset; but most importantly taking ownership of your shit, end-
to-end; which is something that the best and most effective SysAdmins and
Software Engineers always did.

We don't hire "DevOps Engineers", rather "Platform Engineers", in which we
have a hard requirement that you are approaching competence as a software
engineer in at least one language, and in at least one paradigm (e.g you
should be able to tell me about type systems, data structures, polymorphism,
higher order functions, composition vs inheritance, referential transparency,
TDD etc).

We also expect that all of our backend software engineers deploy and maintain
their own infra (as code), using the guardrails/services/systems provided by
our platform team. Deploying a new database cluster for a user-facing service
is a pull request from a backend engineer, not a Jira ticket for a "DevOps
Engineer"

There are companies out there "doing it right", but they are in the minority.

~~~
_bxg1
This is something I've been wondering about. I recently interviewed for a
full-stack dev position, and I felt I did quite well on all of the
_development_ questions. It wasn't a devops role - they had a whole separate
devops position - but I got passed over for another candidate and when I got
the news it was suggested that I should "get some devops experience" and maybe
try again in the future. I thought that was weird. I know generally what
docker/containers do and what purpose things like Kubernetes serve, I'd just
never used them. I figured I'd be able to pick up whatever I needed for doing
minimal devops tasks in the course of the job. Is it common to expect more
than that from "developers"? This was neither a small nor a foolish company.

~~~
darkr
I can’t speak for how common it is, but in my opinion, it is not unreasonable
to ask that developers have at least a reasonable understanding of how the
systems that they build actually work, throughout at least the majority of the
layers of abstraction that they run upon.

It is though quite unreasonable to expect experience with specific tools,
unless you’ve asked for them on the job spec.

The title “full stack engineer” is a whole other can of worms, not a million
miles away from this DevOps thread.

~~~
cmiles74
I agree with brundolf, these are tools and most people can get a handle on how
they work in an afternoon or two. We aren't talking about expecting new hires
set this infrastructure up from scratch, just use the tools that are already
in place.

------
madrox
Over the last 10 years, I've become convinced the idealized notion of DevOps
is not a thing. I have a couple reasons:

1\. There is a huge sysadmin workforce. Many of them never learned to really
code. Are these people supposed to go away? Are they supposed to learn to
code? Coding is a skill that takes time to master, and if these people were
sufficiently motivated to code, they probably would've become coders before
now.

2\. In general, software engineers aren't thinking about or care about their
CI/CD pipeline beyond it being easy to work with. They don't care about their
infrastructure except insofar as what they need to know to make their stuff
fast. There is a nice separation of concerns there that's nearly impossible to
avoid. As a result, it's very easy to build higher order tools to handle most
workflows and infrastructure. It's the whole reason AWS can exist and why most
devs don't care about the details of ops.

I realize I'm speaking in very broad terms. However, broad terms are what
define an industry. I've seen this play out enough times that I knew what this
article was going to be about before I even clicked on it. I'm pretty sure
most other people did, too.

~~~
mr_tristan
I’m convinced it’s driven by business communication and objectives more than
skill set.

I’ve _never_ worked with a “product manager” or “product owner” that could
effectively tell me what a SLO was, or simply tell me what the business impact
of bugs were. All I usually get is what we’ll lose if we don’t implement
feature X or Y.

This impacts “DevOps” because it almost always puts focus and priority on
anything feature-related. In order to do anything not feature-oriented, you
literally have to be fighting massive quality problems on a daily basis.
Thus... very few organizations understand how to have “devs and ops spend
quality time together”. Almost all time not building features is wasted in
their eyes.

In the end, this is why there’s a “dev and ops split”. The devs drive
features, which PMs understand, and the ops makes things run, and the PO/PM
types ignore them but sometimes like to make requests.

I think a pre-cursor to any real DevOps culture are SLOs that are understood
and reported to executives. If that doesn’t exist, DevOps usually ends up
being a BS term used to “remain relevant”. Without some kind of business
quality objective, real DevOps time is pure cost, and counter to business
development.

Just like “agile” BS, any real change usually starts at the top fo the
company.

~~~
Guthur
I'm not convinced that it's only POs that struggle with quantifying the cost
of a bug in production.

I can see the same when programming language advocates extol the benefits of
more static analysis for software that does not even remotely need the level
of reliability they are aiming for.

The guy who hacks it together in PHP will just get to market first and get
such a critical mass of market share that 5-10% more reliability probably
won't matter.

Of course a lot of this depends on market sector, self driving cars are a
million miles from serving banner ads.

~~~
mr_tristan
True, many POs/PMs drive a lot of prioritization of work, since (maybe
unfortunately), they're usually the ones left to "understanding the business
impact".

It kind of adds more credence to people not being so siloed in responsibility.
We all just bring different skills to the party. Thus, we're all responsible
for quality, for business impact and prioritization, etc.

Unfortunately, I've often had situations where, upon a reorganization, I'm
left with managers who love to "guard others from distractions" and end up
cutting team members off from broader discussions. Thus, leading to the
siloing.

This is why I think "DevOps" ends up meaning nothing. It's a _lot_ of details
that requires lots of communication and knowledge sharing that can be
overwhelming to, ahem, certain PHB types. Sometimes you can implement a tool,
other times you can just not build a particular feature, but the reasons are
usually very complex and never easily get broken into a simple one-liner task
definition with a story point cost.

------
h2odragon
> The most common job title seemed to be SRE (Site Reliability Engineer),
> although there was a long tail, and they don’t care much about job titles.

Oh I like that one. "I kept the pile of shit running and put the fires out
before they were noticed" is too long to put on a business card, anyway. "Site
Reliability Engineer" sounds almost tame.

~~~
StevePerkins
The name and job title churn in this space over the past five years is
breathtaking, almost JavaScript-ian.

I've been at my current company for 6 years now. When I first started, those
guys were "Operations", and the job title was "Admin". We used Google App
Engine, and they mostly clicked buttons in the GCP web console.

One year in, they re-branded as "DevOps". They were all "DevOps Engineers".
They wrote a lot of hacky Python scripts, using the "gcloud" utility, to
manage our new Docker-based services on Google VM's.

A year or two later, they decided to become "Platform Services". They changed
their titles to "Platform Engineers". They saw us developers using Jenkins for
our CI/CD pipeline, and decided to re-implement all of their Python scripts as
Jenkins jobs.

Earlier this year, they became "The SRE Group". They are "Site Reliability
Engineers" now. We eliminated their Python scripts, by migrating our
microservices to Kubernetes and using managed databases. So now they're back
to clicking buttons in the Google Cloud Platform web console again.

~~~
sevagh
You speak of them with such disdain. Is that a common attitude in your
company?

~~~
TurboHaskal
I can understand such disdain. I am a software engineer currently working in a
DevOps team (yes, team, you’ve heard it right). I’d say that 90% of DevOps
engineers I’ve met don’t even know what a linked list is, and they themselves
talk with disdain about developers.

So much for DevOps philosophy.

~~~
Aeolun
I wouldn’t mind so much if they at least knew what terraform was.

~~~
goatinaboat
_if they at least knew what terraform was_

Any real, experienced DevOps engineer will reply “a steaming pile of shit”.

~~~
watermelon0
Sure, it's got its problems, but please show me a better alternative.

~~~
goatinaboat
On AWS, Boto. On Azure, DSC (Powershell) and ARM templates.

You will need to maintain separate TF codebases for each one anyway, but with
the native solutions you get easy access to all features of the platform and
don't have to spend most of your time jumping through stupid hoops to try and
pretend that one tool can do everything.

No-one would write code worrying if it was valid in both Java and C# but that
is the same level as what TF claims to do. It's complete shit, ditch it and
you will be 10x happier, I guarantee it.

~~~
Aeolun
Sounds to me like using a single tool instead of a bunch of separate ones with
different functionality is a win to me, but we are each entitled to our
opinions.

Boto doesn’t seem comparable to terraform at all, unless you enjoy building
state management yourself.

Terraform has been better than anything else I’ve tried so far.

------
porpoisemonkey
I recently transferred as a developer to a development operations role at a
medium-sized company. This article exactly describes my experience - the main
focus of the ops team seems to be on build, deployment and monitoring
technologies focused on a migration towards containerization running on AWS.

Code and tooling is built on a "what works" basis and no particular attention
seems to be paid to the overall design of the software or testing (likely due
to a real or perceived lack of time). On-call rotations (and how to eliminate
work for them) is the hot button discussion topic.

The #1 question I get asked by the developers is "Did you _really_ want to
transfer from development to ops?" which me think that a lot of developers
look down on operations roles and see it as a demotion. I find that quite odd
given that 1) ops keeps the product up and the money coming in with relatively
little headcount and 2) most of the people working on our ops teams have a
formal education in software engineering.

~~~
empath75
> The #1 question I get asked by the developers is "Did you really want to
> transfer from development to ops?"

I’m not sure what job listings they’re looking at, but as an ops person
(kubernetes) I’m interviewing for jobs with close to half a million a year in
total comp.

~~~
kubelust
Second time I'm hearing (what's for me) a mile-high comp number for kube jobs
and I'm now really tempted.

Working as a data scientist - software engineer in a midsize company, I
constantly battle amateur ops folk and "backend" fullstack jockeys from
introducing kubernetes into a saas product I mostly created by myself (makes
money but there are maybe ten users per hour tops, why would I need kubernetes
for that?).

My org has seen multi-day downtimes for the entire eng team workflow because
the eks cluster went down and they couldn't figure out how. We have four
people dedicated in the infra team for this! I'm not really an ops person but
I see where the failings of these folks and stacks are, and feel like I might
be able to learn to be half-decent if I put the time. What advice would you
give ?

~~~
imtringued
Kubernetes is a full time job. If you want to capture 90% of the benefits of
containerization without wasting too much time on a complex solution then
simply restrict yourself to only use docker with bash scripts and maybe a load
balancer if you really want to have HA.

I've found nomad to have a lower complexity than kubernetes but if you cannot
directly integrate service discovery into your application then you will need
to use a service mesh which is an all or nothing thing but using something
like traefik's support for consul means you will have to use regular service
discovery alongside the service mesh. It's not a huge burden but there should
be a better way.

------
jancsika
> DevOps means the veteran admins had to check in their personal scripts

Oh my, this is an epiphany.

I don't even know what DevOps means. But if the implication was that the
"veteran admin" was making arbitrary state changes and is now forced by DevOps
to document it in the commit history, I am firmly in favor of whatever DevOps
is.

~~~
Nursie
What it was supposed to be -

Software development, deployment and ops come under the same role. The DevOps
engineer controls the horizontal and the vertical, the code, the environment,
the build, the deployment, and as such is a multi-talented unicorn.

What it seems to have become - Sysadmins writing scripts around terraform, and
formalising their work to the extent that it is at least usually reproducible.

~~~
orthoxerox
> a multi-talented unicorn.

That's not necessary. Natural curiosity and willingness to treat dev/ops
(whichever is not your team) as _us_ , not _them_ , is sufficient.

~~~
Nursie
I'm not sure I've seen that be any more prevalent now than it was when we had
developers and sysadmins.

To me it's become like 'agile' and interesting set of viewpoints and
philosophies totally ruined by the industry that's grown around it.

~~~
orthoxerox
Well, at least the DevOps industry brings actual value to the process (well,
not when you are sold OpenStack, OpenShift, Vault and whatever else to run a
single container image).

------
ljm
> Everyone hates YAML. Everyone writes a lot of YAML.

I don’t know a single engineer (ops or not) who enjoys writing YAML, yet it is
utterly unavoidable.

I’ve lost count of the amount of bugs and broken deploys that have happened
because of YAML, or because of a type error caused by it.

~~~
weberc2
Yeah, what I want is JSON with comments and nice multi-line string support. I
don't like how much syntactic magic YAML does (I don't need or want the
country code for Norway to be parsed as a boolean False value). I still don't
know what the exclamation point does (e.g., !Ref).

And clearly YAML is the wrong tool for infra-as-code since CloudFormation has
to build a macro system, conditionality, referencing, and a couple different
systems for defining and calling functions (templates being one and their
implicit functions being another). We also see tools like Troposphere and CDK
which are effectively different ways to generate CloudFormation YAML via
programming languages (or more precisely programming languages that were
designed for humans).

And it's not just limitations inherent to CloudFormation--Helm has long had
templates for generating YAML, but those also weren't sufficiently
powerful/expressive/ergonomic so Helm3 is supporting Lua as well. And as I
understand it, Terraform is constantly adding more and more powerful features
into HCL.

So what's the solution? It's pretty simple--we should keep the YAML around,
but it should be the intermediate representation (IR), not the human
interface. The human interface should be something like a functional
language[^1] (or an imperative language that is written in a functional style)
that evaluates to that YAML IR layer. The IR is then passed to something like
Kubernetes or Terraform or CloudFormation which understand it, but it's not
the human interface.

As for the high-level language, something like [Starlark][0] would work well.
It's purpose-built for being an evaluated configuration language. However, I
would argue that a static type system (at least an optional static type
system) is important--it's easy enough to imagine someone extending Starlark
with type annotations and building a static type checker (which is much easier
for Starlark since it's a subset of Python which is intended to be amenable to
static analysis).

This, I think, is the proper direction for infrastructure-as-code tooling.

[^1]: Functional in that it is declarative instead of imperative--not
necessarily that the syntax should be as hard to read as OCaml or Haskell.
Also, while YAML is also declarative, it doesn't have a notion of evaluation
or variables.

[0]:
[https://docs.bazel.build/versions/master/skylark/language.ht...](https://docs.bazel.build/versions/master/skylark/language.html)

~~~
sedachv
I do not understand the need for all of these different new language
implementations and data formats. GuixSD vs NixOS already showed that Scheme
is a superior solution as a configuration language, scripting language,
template language, and intermediate representation. A single language that has
30+ years of successful production use, tons of books and documentation. Why
re-invent the wheel in four different, incompatible ways?

~~~
weberc2
Is NixOS a scheme? Anyway, we moved away from Nix because of all its problems
(the language being only a medium-sized one). Also, I detest CMake, but by
your own standard (longevity, popularity), it is better than Nix or Guix.
Frankly those tools haven’t shown themselves to be “superior” in any
meaningful way. Yes, they have been around for a while, but having been around
for a long time and not enjoying any significant adoption is not very
compelling.

~~~
sedachv
> Is NixOS a scheme?

That question does not make any sense. I am talking about Scheme the
programming language: [https://schemers.org/](https://schemers.org/)

NixOS is a GNU/Linux distribution built on top of the Nix package manager. The
Nix package manager has its own custom configuration language. GuixSD is a
GNU/Linux distribution built on top of the Guix package manager. GuixSD uses
Guile Scheme as the package configuration language, system configuration
language, scripting language, and implementation language for many of the
system services (such as init). GuixSD does more things better with an
existing standard programming language than NixOS does with its own custom
programming language.

> Also, I detest CMake, but by your own standard (longevity, popularity), it
> is better than Nix or Guix.

What does CMake have to do with anything?

> Yes, they have been around for a while

What are you talking about? The first stable version of NixOS, 13.10, was
released in 2013. GuixSD 1.0 was only released this May.

Your post is hard to make sense of.

------
dpeterson
I am a solutions architect at Red Hat but I came from being an application
architect. I recently gave a talk on cqrs and event sourcing at a midwest user
group. I am routinely surprised how few "DevOps" engineers care to grasp
application architecture yet want to start dabbling in things like Kubernetes.
Automation using Ansible, yes, no problems there. Creating microservice
architectures using containers and kubernetes tooling is yet a ways out for
many. New tooling is absolutely necessary to implement the things we have all
been talking about for at least 10 years. Developers can no longer just care
about writing code. They have to think about how to manage breaking up their
monolith, not just for performance; that's not typically the driving force
behind adopting microservice architectures. It's about improving lead time and
reducing the batch size of releases. You can't have microservice architectures
without both sides(dev and ops) understanding what each actually do. I have
first hand knowledge how difficult it still is to break those silos down.
Developers, if you still think writing code is the hard part; well, I've got
news for you. Ok, writing good code is still hard but there's a lot more to
think about to get the most out of that code when it goes to production. As a
starting point to answer "why do any of this?" I recommend the book
"Accelerate: The Science of Lean Software and DevOps: Building and Scaling
High Performing Technology Organizations".

------
pi-victor
been in an ops role for 2.5 years, 2 years as software dev and currently i'm
in a devops role for 2.5 years now. this article hits the nail on the head.
this was nothing more than a rebranding of the word ops. it has very little to
do with actual software development. not to say that i don't enjoy it, for the
better part of the time i was a software dev, i really missed setting up
machines and day to day network/system administrator work. i know my way
around 4-5 programming languages so i'd say if the need arises i can use the
right tool for the right job. however, kubernetes, aws, terraform, packer,
ansible, bash, etc are everyday things. i'd say in the last years i've
developed a love hate relationship with pretty much every tool out there
(looking at you terraform and k8s). the pitfall with this position is that
there's a lot of buzzwords flying around and 9/10 and if you have some idiotic
management you end up implementing some overkill solution to fix your
problems. the other pitfall, from an infrastructure perspective, is that the
ecosystem around kubernetes is so big that there are now an insane number of
moving parts that you need to maintain and monitor.

------
jldugger
> Being a gathering of system administrators I’d expected more IRC, there was
> no mention of it.

We just don't tell you about it -- Slack is more visible to management, and
thus infosec has made slackbots a fucking integration nightmare.

------
dev_dull
> _As a result they also talk about mental health more than programmers do._

It really is. All of the lifelong DevOps/sre people I know have spent time
grappling with the consequences of this. On all experiences also seem so much
more dramatically in larger companies where teams can properly load balance.

------
rinchik
> DevOps is Ops with new tools... I did not meet or hear from any developers.

This is awful! It's not DevOps, it's an old infra gathering! Very
disappointing.

I expected devops conference to have devs ONLY. Devs that that are curious and
passionate about whole product life cycle, hardware, scalability, deployments
and networks, devs who also pulled and transformed/transitioned their ops
friends into devs.

Is it a general trend with DevOps or just devopsdays Portland conference
thing? Is it cargo-culting?

~~~
Conan_Kudo
> I expected devops conference to have devs ONLY.

In the past five years I've been doing this, I can emphatically say that the
number of software engineers/developer types that I've met at DevOps events
are so few that I can more or less count them with both of my hands.

In many ways, I consider DevOps an evolution of what systems administrators
do. These days, there's so much demand on infrastructure teams to do more with
less. Automation systems have massively improved our productivity that the
business types are happy to do so much more that it leads to a vicious cycle
of compression and stress.

Don't get me wrong, I love automation and such. But it's terrifying that we're
asked to put together systems that we can't comprehend.

I'd consider myself an oddball in the "DevOps scene". I was a bona-fide
polyglot software engineer before I pivoted. I did embedded systems in C and
C++, full stack Java work, Nodejs stuff, etc. I later did QA automation work
in Python, and pivoted from that into DevOps. It wasn't that far of a jump.

But most people I know in the "DevOps scene" are nothing like that. They don't
have years of experience in OSS allowing them to see the real gold in the pile
of fool's gold. They don't have professional experience doing software
development. Many of them only barely understand software architectures. Life
cycles? Beyond systems maintenance, there's not much for them to learn about.

I would argue that proper DevOps teams wouldn't have things like on-call
rotations that give periodic or constant on-job demands, as that's usually an
SRE-type responsibility. But DevOps is just a fancy way to reduce your Ops
teams for most places.

~~~
busterarm
I also moved from development to ops and am now at a company with a large
engineering organization.

Most developers don't care about ops. They expect everything to "just work"
and they don't expect to have to help that process along with the way their
code is written. It doesn't matter how junior or senior they are either.

I'd say more than 50% do things that are extremely irresponsible wrt resource
utilization. Especially if it's I/O.

------
hackerm0nkey
I’ve started my career very early on on the age of 16 as a system admin. Heart
and soul was on pure engineering and less on what people label these things on
CVs.

A few years later I moved up the stack into a full time programmer and been
doing that ever since 20 years down the line.

I have built systems from the ground up literally, loading the van with the
equipment, installing the racks, the servers, the EMC’s ...

Having seen it from all perspectives, sys admin, DBA, Ops and all it’s
variants and a programmer. I’d like to think of my self as an engineer who
solve problems or build things to solve problems.

The tools that you use does not define you. I have seen and still seeing loads
of people Devs who don’t understand or willing to understand how ops work Or
how the crap they produce runs and get maintained.

And ops who write hacky code in the name of infra as code.

I wish we can just be Engineers and stop labelling and define people by the
tools they use. Things will be better IMO

------
Pxtl
"everyone hates yaml" hah. Shocking how we're stuck on these problems.

The old line about "the problem that it solves is not hard and it does not
solve them well" about XML is kind of hilarious when we're still facing the
same thing about its current popular replacements - JSON and YAML.

~~~
koolba
I'll take _limited_ YAML over JSON any day of the week for things that might
be edited by hand. The cleaner diffs when appending to a list alone are worth
it.

~~~
djhaskin987
True, but when the YAML file gets long and complicated, such as when a
kubernetes YAML spec, it becomes difficult to figure out what indentation
different items need to be at and you end up needing to use tools other than
just your laptop:
[https://twitter.com/Caged/status/1039937162769096704?s=19](https://twitter.com/Caged/status/1039937162769096704?s=19)

~~~
Pxtl
Hah, I can see my own old response to that tweet.

Anyways, as horrifying as it is, that's actually an argument in favor of XML.
After all, would being lost in {} be that much better than indents? You'd
still be scrambling to figure out how deep you are in a deeply nested object.

The obnoxious repetition of xml's closing elements actually helps here since
you can see _what_ object you're closing.

...

But I hate XML.

At this point I'm fantasizing about using heavily-sandboxed Lua and going
fully Turing-complete so when your files are getting horribly complicated you
can start doing some minimal reuse.

------
tofflos
For me the meaning of DevOps has shifted from "developers and operations
working together" to "operations picking up developer tooling". As more and
more administrators become familiar with the new workflow I expect them to
transition from mostly relying on manual testing and scheduled drills to
writing automated tests for infrastructure fail-over, backups, firewalls, etc.

I'm not a big fan of YAML the format but I'll gladly put up with it for the
valuable content that's encoded in those files - the entire system described
in a transparent accessible manner at an abstraction level which enables the
developer, the network guy, the server guy, the storage guy, the architect and
the security guy to have a meaningful conversation.

------
Animats
From the article:

 _DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means
developers and operations merged into one team. I was wrong._

 _I did not meet or hear from any developers. It was a gathering of system
administrators who use version control, and write a lot of YAML. Programming
languages were not a topic..._

 _DevOps means the veteran admins had to check in their personal scripts, and
everyone is expected to automate more things. They relate to the software
their business runs (“app”, “asset”) the way a merchant-navy captain relates
to what’s in the containers on his ship._

Ah. This must be DevOps for big shops. Not "why are we paying operators when
we could have the developers do it?"

------
retr0grad3
I'm an organizer for one of the (many) DevOpsDays Conferences held around the
world. Speaking only for myself, I struggle with making sure that DevOps is
about Dev and Ops working together, not just Ops / SysAdmins using a
different, more marketable job title. When Operations Engineers become DevOps
Engineers, it subconsciously says to Developers that they don't belong.

I'd encourage all of you that feel excluded, or feel like these are just
SysAdmin conferences in sheeps clothing, to go to your local DevOps Meetup and
start there. Demand that DevOps continues to stand for a community and
cultural way of making software development suck less and not just "the next
Agile". /rant over

~~~
sudosteph
> When Operations Engineers become DevOps Engineers, it subconsciously says to
> Developers that they don't belong..

I think anyone who actually feels that way about their peers getting a title-
change is being incredibly over-sensitive.

The fact is that for many years, some "System Administrator" types did already
think like "developers" and use similar tooling. Those folks just didn't get
recognized or paid for that, because the role was defined by what you did
(keep things operational), instead of how of you did it. So when "DevOps"
became a thing, it was actually not a huge leap for many Operations types. So
it's reasonable that many of them found positions with the "DevOps" title
appealing due to the increased pay and prestige, and the fact that they
already had the skills to do it.

Conversely - many developers don't actually have much experience running
production systems, and it can feel demeaning to hear those people say that
Operations-types do DevOps "wrong" because operations people don't want to use
the latest bleeding-edge tool or the most efficient programming language. A
preference for stability and maturity is something learned from experience,
and stability happens to be very valuable when it comes to production
situations.

All that said, I still love hearing from dev-types who share their experiences
around DevOps. Both sides should continue learning, hearing each other out,
and valuing each other's contributions without assuming ill-intent or
inferiority due to their educations background or previous titles.

~~~
Conan_Kudo
> The fact is that for many years, some "System Administrator" types did
> already think like "developers" and use similar tooling.

The funny thing is that it was the opposite for me. I became a DevOps Engineer
because a lot of my work was being self-supported with my own infrastructure.
I found that I had a knack for finding and putting together good systems to
support my work. In many ways, I guess I was considered the quintessential
DevOps person. In my QA role, I wound up being a developer who needed to put
together infrastructure to support the needs of the QA group and to a lesser
extent software engineers.

When this was recognized, I was moved into the sysadmin group as a founding
member of a new team. That team grew and split out into its own group that
specifically interfaced with both software engineers and infrastructure
engineers. My team is not perfect, but it's been the best I could ask for.
We've done amazing work transforming our engineering processes over the years,
and software engineers are now more often part of the infrastructure bringup
process than not. We've still got a ways to go, but we're getting there.

------
cbushko
I find this article to be a little frustrating. It is kind of like a layperson
going to a mechanic convention and saying "all I heard about was engines,
brakes and tires. Everyone is interested in tires".

If you are not in the domain, then you will not understand why the 'devops'
engineers are all looking at these things. Kudos to the developer for going to
the conference and learning something new.

Devops are exactly the same as Software Engineers. There are those people that
just slap code together and do not care about design, tests or the longevity
of the product. Devops can be the same where they just push the button in AWS,
create some bash scripts and get things running.

Then you have those Devops that approach building product and infrastructure
in a repeatable, scalable, traceable fashion. They care about software
engineering practices and building something to last.

I work as something in between a platform engineer and ops. I run 4 kubernetes
clusters in GCP, had to build up all the underlying resources (dbs, service
accounts, storage, clusters) and get the product running on it. I use
terraform for this, including kubernetes deployments, and it works quite well.

Just like software engineers, devops/ops is under time constraints and have to
hack things together. Whatever it takes to keep the business running...

------
aswinmprabhu
> The most common job title seemed to be SRE (Site Reliability Engineer)

This article (plus the comments) kind of scares me because I am a CS student
who just received an opportunity to work as an SRE Intern at a pretty big
company over the next summer break. I took it because I wanted to find out
whether SRE is a role for me but I am a bit worried that if I accept a more
permanent SRE role after the internship, will it hurt my chances of moving to
dev role later in my career.

~~~
throwaway122379
My job is 50% dev 50% devops/sre i would say.

One thing I noticed about whole SRE/devops scene lately is that there seems to
be no grasp of the KISS principle, there are people building elaborate glass
houses on top of k8s (tho k8s+pipelines is very useful its easy to get carried
away)

another observation is that the people who are not good programmers but are
good at appearing to get stuff done (but who create tons of tech debt) seem to
fall naturally into SRE roles, as there is no testing culture in pipeline
driven development

SRE currently is like the wild west, it be many years before it starts to
resemble engineering

~~~
abhiyerra
This. I run opsZero and the amount of CI/CD implementations I see in different
infrastructures is crazy. Most code is not tested or overtly complex and every
DevOps person seems to reinvent the wheel a hundred different ways.

Living in the world of Terraform, Kubernetes, and helm has been a godsend
because it codifies deployments into a few buckets that removes a lot of this
custom code. To be honest I’m thinking the days of DevOps may be numbered.

------
geggam
Its hilarious to read Developers and Operations guys bash each other with
DevOps.

DevOps is about getting those two groups in harmony where they compliment each
other.

Nothing else.

------
pnako
You have a monolith. The monolith is deployed by a grumpy old admin using a
shell script. He monitors it with another shell script. This is the old, bad
way of doing things. Monoliths are bad because they are the kind of thing only
neolithic cavemen would care about. So you break up the monolith into micro-
services. Of course, you use (other) services to provision and monitor your
new micro-services. You now have dozens of tools and scripts and configuration
files. This is, naturally, the modern way of doings things. If tomorrow
morning, your 500 customers suddenly become 3 million customers, you can just
flip a switch and scale on demand in the cloud! Soon you'll be so successful
that you can even hire an entire team to look after your new non-monolithic
system.

~~~
kbr2000
It's soon time to invent nano- and pico-services, so everything gets to be
scalable by default. That way we can finally stop bothering about first
examining the problem domain at hand, and pull out all the stops at once --
looking busy like bees in the process.

Excuse me while I construct this house from micro-bricks and molecular mortar
first.

------
rcarmo
I’ve always “automated stuff away” and worried about making sure whatever I
developed had a sane, reproducible way of a) being tested and b) being
deployable at scale, so I’ve watched DevOps/SRE unfold as I hopped between
both sides of the fence (I’m a Solution Architect at Microsoft now, and spent
a few decades in telcos doing the above).

I’m going to be (intentionally) critical here, but please understand that
these are examples taken from a biased sample (I work in Portugal with very
traditional enterprise customers and early stage startups, either of which
usually lack-or don’t care about-senior talent - juniors of all stripes tend
to be over-enthusiastic about jargon and bleeding edge, and the blast radius
is huge on both kinds of companies).

The way I see it (in both enterprise and the local startups I deal with) is
that DevOps has created as much confusion and undue ceremony as Agile in many
places, all of which share a common trait—developers there do not understand
(or want to deal) with software architecture and just want to hit their sprint
targets without reasoning about the architectural impact of implementation
details.

Enterprise devs will usually not have any real control about architecture,
internal endpoints or even dev environments (all the Ops is taken away from
them) and startup devs are usually rushing things, setting up faster infra to
fix code problems and building up technical debt _in architecture terms_ (the
Ops stuff is usually haphazard and too bleeding edge).

There are some positive exceptions, though. I’m currently helping a customer
do Kubernetes from a _governance_ perspective (i.e., figuring out all the
steps from dev to prod including environments, namespaces, network policy
groups, etc.), and even though this is being done alongside already existing
deployments, _taking the time to plan and work things out inside your org_
makes a lot of difference.

We’re not calling it DevOps, SRE, or Agile. It’s not a buzzword-laden, Valley-
anointed process. It’s just (senior) engineers (Devs and Ops) talking and
systematically going through what will work and won’t...

------
peterwwillis
It occurs to me that DevOps is Plato's Cave. So many people think they know
what DevOps is, yet all they really know is what they've always been familiar
with, and they can only interpret it based on that. It's kind of depressing.

~~~
scroy
Reminds me more of the blind men and the elephant:
[https://en.wikipedia.org/wiki/Blind_men_and_an_elephant](https://en.wikipedia.org/wiki/Blind_men_and_an_elephant)

------
readme
A question to ponder: is devops a niche created by developers who can't
sysadmin, or sysadmins who can't code?

“A human being should be able to change a diaper, plan an invasion, butcher a
hog, conn a ship, design a building, write a sonnet, balance accounts, build a
wall, set a bone, comfort the dying, take orders, give orders, cooperate, act
alone, solve equations, analyze a new problem, pitch manure, program a
computer, cook a tasty meal, fight efficiently, die gallantly. Specialization
is for insects.”

------
nfrankel
> DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means
> developers and operations merged into one team. I was wrong.

Nope, he was right. DevOps is all and only about removing the wall between Dev
and Ops. That might go as far as merging the team, but not necessarily.

However, the conference is _dead wrong_ is they convey the feeling that
"DevOps is Ops with new tools". That's akin to putting lipstick on a pig, and
does DevOps as an organization tool a lot of wrong (just like Agile, etc.).

------
vaidik
DevOps is about collaboration between devs and ops towards building software
and operating it efficiently to meet business needs. Traditionally building
and operating were looked very differently but experience has taught us that
features and stability cannot be achieved without understanding of how
features were built and what needs to be done to run those features in
production efficiently. In an ideal world, the same person should be able to
build and operate systems in production however we don't live in an ideal
world - traditionally there was a big divide between devs and ops, and in
complex environments you can't learn to do everything yourself. Hence...
DevOps.

There are multiple ways this can be achieved. This article is a good read to
understand different models of implementation of DevOps culture -
[https://web.devopstopologies.com/](https://web.devopstopologies.com/)

"DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means
developers and operations merged into one team. I was wrong." \- This is kind
of right but not completely. DevOps is about bringing ops closer to dev teams
by either merging ops or retraining devs to be able to be better at ops (there
are more ways as indicated in the above article about DevOps topologies). The
idea of doing ops with new tools is that these new tools help us with doing
ops like we are building software. Many software development best practices,
abstractions, architectures can be now applied to ops as well. This shift in
doing ops with these tools which are a lot like other software development
tools enable ops to come closer to devs and vice-verse. Here is a great talk
by Mitchell Hashimoto (the creator of Terraform, Packer, Vagrant, Vault, etc.)
that got me into DevOps -
[https://www.youtube.com/watch?v=UTQQggVx4sI](https://www.youtube.com/watch?v=UTQQggVx4sI).
This talk explains the use of tools with the backdrop of DevOps.

Other than these two things, your observation was spot on. I was not at the
event but that's the story at most DevOps events.

I think software architecture will soon become an area of focus in DevOps
circles as the right architecture also essential for achieving CI/CD, agility
and DevOps.

Never the less, welcome to the world of DevOps. :)

------
BiggieCheese
The DevOps movement petered out because it solved most of it's technical
problems with tools that have become industry standards.

As a sysadmin, you used to have to write scripts to solve EVERYTHING.
Especially since tooling for ops people in the 2000s was a load of hot
garbage. A lot of tools back then had GUIs, which weren't automation friendly.

Nowadays most ops problems have some easily automatable tool, which I think
gives less incentive for sysadmins to reach for a programming language.

~~~
kbr2000
It seems more likely to me that those "devops tools" only solve a small slice
of the big sysadmin cheese, and not in a flexible way (one useful skill of
sysadmins is being able to react on the spot to what's coming to them at that
moment, using basic tools and logical reasoning, much like a firefighter).

Being able to differentiate between what's worth to automate and what is not
is another one of those useful skills. Knowing a scripting language to glue
those basic tools together is too. Being proactive (assisted by monitoring
software and knowledge of the underlying systems so you can tell what
consequences some event have) is another...

None of that is new nor surprising to sysadmins. It might sound fresh to
others, being trapped to reinvent a small slice of it (badly). Also, "devops"
is just another funky management fad by now.

------
ragerino
As a lazy developer I enjoy DevOps principles. First time I don't have
infrastructure issues. My code builds and deploys automagically and sys-admins
can focus on important stuff.

I'd also like to describe it as just another tool in my toolbox. Created by
admins for developers. I don't mind YAML at all. It's better than JSON or XML
and it's a first class citizen in Go, Python, JS and other languages.

------
beardedman
Can anybody give me a reason why containers are popular for production
deployments (NOT referring to K8 use-cases)? As a development workflow they're
amazing - but cheap cloud based VPSs + Ansible/Puppet work really nice. Why
would you add another layer of abstraction in there when all it does is hinder
performance?

~~~
slyall
Because you are deploying a single artifact with minimal dependencies that
behaves in a known fashion.

Whereas if you are deploying code blobs they start depending on the operating
system to provide stuff. You have to have the right version of Ruby on each
machine. This becomes a problem when the right version varies between products
and you would also like to update the OS to something new.

With containers the application code is decoupled from the operating system.

~~~
beardedman
Yea I get that. To me this scenario gets mitigated with due diligence checking
when spinning up a virtual machine - but I recognise the benefit in having a
single artifact.

------
eitland
I've been on both sides of that fence for 13 years now (currently in a really
nice dev job.).

To me it seems like things are getting better. Often devs and ops are the same
kind of people only on different teams, or even embedded in the teams. Nobody
is hiding anything and people are working together if stuff breaks.

------
kylek
>> Many were curious about service-meshes (basically a smart network proxy?)
such as Istio or Linkerd, but almost none were using one.

Going to blame Ubuntu's psychological priming via ads in their MOTD for this
one

~~~
VectorLock
What ads are in Ubuntu's MOTD and what do they have to do with Istio/Linkerd
service meshes?

~~~
williamallthing
Ubuntu's MOTD displays dynamic ads and at least one of these ads was about
Istio. E.g. as best I can tell, every Ubuntu installation that hadn't disabled
dynamic MOTDs displayed this Istio ad for most of August:
[https://bazaar.launchpad.net/~ubuntu-motd/ubuntu-
motd/trunk/...](https://bazaar.launchpad.net/~ubuntu-motd/ubuntu-
motd/trunk/revision/68?start_revid=70) .

(Of course as a Linkerd person I think it's ironic to advertise Istio on
MicroK8s because Istio is anything but micro. But Microk8s has great Linkerd
support these days, so maybe we'll get a Linkerd ad one day, and harmony in
the universe will be restored.)

------
rachelbythebay
Spot on. And totally depressing. Apps as black boxes. Ugh.

~~~
scruple
This has been my lived experience with DevOps since I was first subjected to
it ~2014. In a word, depressing, yes.

I've been fighting to maintain full control over a few of my teams
applications, because I refuse to let the juniors and intermediates, who
weren't lucky enough to gain full-stack experience, live in a world where they
don't understand how these applications and services, etc..., _actually_
operate in production environments. But I'm losing ground all of the time.
Management, emphatically, _does not care_.

------
goatinaboat
I can imagine it the other way round: "hey have you tried that new javascript
framework? It only came out last week, it's pre-alpha but I already have it
running in production. What? No, not a problem, I just chuck it over to ops
and they keep it running".

~~~
codingdave
You are exaggerating to make a point, but... what else would you prefer? That
your sysadmins approve your frameworks? How detailed would you get? Do you
want a formal approval cycle before installing a new npm package?

I'd prefer an ops team that trusts the dev team is competent, and trusts that
if the tests pass, the devs probably made acceptable decisions.

~~~
atmosx
> I'd prefer an ops team that trusts the dev team is competent, and trusts
> that if the tests pass, the devs probably made acceptable decisions.

You don't need anyone to _trust you_ as long as you're responsible for
deploying, monitoring and responding to failures regarding the app, meaning:
being on-call for the app(s).

~~~
codingdave
Have you worked in a place that has a dedicated DevOps team, but makes the
devs be on-call for the apps?

~~~
spc476
I work (and have worked for nine years) at a company with a dedicated DevOps
team. As a developer, I have _never_ been on call, but I do have to be there
for deployment of my code to production (which, due to the nature of the code,
happens around 2:00 am my time) to ensure everything is working as expected
(and only once was a roll back required).

