
DevOps didn’t exist when I started as a developer - tcsf
https://circleci.com/blog/devops-did-not-exist/
======
caymanjim
DevOps absolutely existed when the author started as a developer. It was just
called systems administration back then. There's been a focus on developer-
specific systems administration over the past decade, and a particular
developer-focused role has been carved out and labeled "DevOps", but make no
mistake: it is systems administration. Just a niche within it.

When I mention "systems administration" to younger coworkers, they look at me
like I'm a crusty old relic out of touch with modern engineering. People have
no sense of history and little understanding of where the devops role came
from and what it really means. I had a CTO tell me he hadn't heard the term
"system administration" in ten years.

What were our devops people doing? Running web servers, managing networks,
configuring DNS, managing backups, configuring cloud services. Absolutely none
of it in support of the dev team I was on. The developers were responsible for
managing their own CI/CD pipeline and deployments. The people called the
"DevOps Team" were responsible for managing production. In this case, they
were systems administrators and were not in any way a devops team, but the
terminology is so skewed now that they were labeled as such, and I was labeled
an out of touch old greybeard.

This isn't to demean devops in any way; it's a valuable role, an evolutionary
step in software team organization, but by no means is it new and by no means
is systems administration a dead role.

~~~
strken
When I hear systems administration I assume it's pets not cattle, and that
you're administering individual named servers with bash scripts rather than
administering a mesos cluster with chef or puppet or whatever.

I'm not sure where I picked up this impression from.

~~~
alchemism
It comes from bright, talented SWEs who do not, bless their hearts, understand
that IT Operations is an entire profession unto its own, and not merely the
grunt-work they are forced to do while working as interns or juniors.

------
johngalt
DevOps is the intersection of three things.

1\. Developers learning that running code is not the same thing as reliable
code. Certain things must be put in at design time to allow for operations.

2\. Operations people supporting development by formalizing/streamlining the
deployment process to have changes occur faster, safer, and more often.

3\. Aligning goals and attitudes in such a way that prevents conflict between
teams. Development doesn't get to shift the costs of outage prone
unmaintainable code onto an ops team. Or that ops doesn't roadblock/veto all
changes.

In the bad old days, it was common for developers to act with little care for
operational outcomes. The responsibility was far removed, and bugfixes didn't
win many accolades. Managers want wish fulfillment and pushed developers to
toss new features into the mix as fast as possible, resulting in tremendous
pressure against ops teams to deploy despite increasingly crufty and debt
laden code bases causing unpredictable problems. Ergo, ops teams who blocked
all changes for trivial issues because they were judged only based on outages,
not feature delivery.

~~~
ozim
Yes, devops is not one person, but team is devops. It comes from standards,
that developers should not have access to production data, so you need ops.
But on the other hand you don't want to have dev and ops team people who throw
shit over the fence.

So basically it comes to having devs, test, ops, sec, persons in one team with
common goal of delivering featuers.

Because most friction was on boarders like dev | test | ops | security, all
those people being on role specific teams would do what is important for their
team. So test people would blame devs, sec people would blame ops, test or
devs. When you are in one team that has common goal of delivering feature X
and you are measured by delivering feature X, everyone is willing to take
pragmatic approach instead of "code has to be perfect", "tests have to be
perfect", "security has to be perfect".

I read somewhere "magic happens if you seat testers and developers in the same
room", I belive it was "Project Phoenix" book. But I despise from the start
having those devs vs test people or test people beeing happy about finding
bugs. They should be sad because what matters is delivering feature, so they
should be happy when they don't find anything. That said, thay should do all
they can to find bugs, because we have common goal: "working software".

------
irrational
This article was frustrating to me. When it started out in the mid-90s (which
is when I started my career also), I thought our experiences would be more
aligned. However, I've never worked with an operations team.

In all the places I've worked I've been expected to do everything from setting
up servers (originally physical servers, later cloud servers), hardening them,
installing software, optimizing the software, installing and optimizing the
database, creating database schemas and related objects, writing untold number
of sql queries, writing server side code, writing front end code, etc.

I was hoping that the article would explain what devops really means today and
how I can jump on the devops wagon to hopefully make my job of doing all of
the above easier.

~~~
dogprez
The article is lost on me too. ~15 years professional experience as a software
developer and I never had to deal with an "Operations" team. My understanding
is that it's the people that would be doing Amazon's job if you were trying to
not use AWS or some other cloud service?

"Devops" means not fighting with those people??? haha

~~~
alchemism
I take it you’ve never worked in any of the Fortune 500, then.

~~~
irrational
I work for a Fortune 150, but I’ve never worked with a operations, devops,
etc. team.

~~~
alchemism
Who monitors production? Who provisions routers, wireless access points, VPN
servers?

Just because you are “throwing it over a wall” doesn’t mean the team on the
other side doesn’t exist.

------
SamuelAdams
Personally I think DevOps is like religion. It means whatever someone wants it
to mean. Some companies think DevOps means having an automated pipeline for
building, testing, and shipping code. Other companies think it means micro-
services. Others think it means making developers do DBA / SysAdmin work.

All of these things are fine for companies to do. How you run your org is on
you. But I wish companies would go deeper than "DevOps" when putting up job
posting requirements.

That's like saying they do "security". There's a lot of different ways that
can be interpreted. I (as an applicant) only know a handful of those
interpretations really well, so it's important that companies clarify these
terms in their job descriptions. This way they have the right candidates
applying.

~~~
headmelted
> Personally I think DevOps is like religion

As long as no-one is knocking on my door asking if I have five minutes to
discuss Agile development methodologies.

I could reasonably defend _most_ of what people attach to the "DevOps"
buzzword as sane practices most of us were already doing before the hoopla.

Agile (with a capital A) is the absolute worst thing that has ever happened to
the software industry. Kill off that cult and you can DevOps my work with
containerized OWASP Gitflows until the end of days for all I care.

Snark aside, Dave Thomas makes a compelling argument for trying to choose
terms that are as hard as possible for others to co-opt for selling snake oil
(anecdotally, I encounter people on a weekly basis who are self-described
Agile or DevOps experts, but have no technical background whatsoever).

~~~
icedchai
I once worked at a place full of Agile cultists, including a bunch of
"certified" "coaches." They spent about 1/4th of the time in planning
meetings, retros, grooming sessions, sending emails about updating percentage
complete on jira tickets, etc.

They made some of the dumbest technical decisions I have ever seen, were
perpetually rewriting things, changing core APIs, breaking other parts of the
system. The stuff barely worked, and this was after 5 years and dozens and
dozens of developers.

The place was so dysfunctional, you could not even create your own branch in
source control. You had to request it from the IT group that maintained the
"enterprise" SCM server. The IT groups were doing their own form of Agile, so
these requests could take weeks.

Truly awful.

~~~
Sharlin
Seems to me that place would have been dysfunctional with or without their
dysfunctional form of ”agile”.

~~~
icedchai
You are correct. The Agile-esque nature of the place was just added icing on
the cake, so to speak.

------
peterwwillis
Nailed it.

I also got a chance to work in such an organization in the 00's, where dev and
ops closely aligned with each other's work to produce truly reliable,
repeatable, fast, quality products, often. It was fantastic. Nobody would say
"oh but you can't do that, that's not Infrastructure as Code!" They would say,
"what can we all do to make this better?"

~~~
jascii
In the 90's and 00's I was responsible for maybe 20-30 servers, now I am
responsible for somewhere around the 2000 mark.. While "oh but you can't do
that, that's not Infrastructure as Code!" sounds lame, and your co-worker
should learn to be a better communicator, things need to be reproducible when
working with more complex systems and that calls for more rigorous review
practices. Please consider this: It's our pagers that go off at 3am when your
code fails..

~~~
walshemj
In my experience ops call the developers when something breaks overnight in
any case.

~~~
jascii
In our organisation, developers have need nor clearance to access live
customer data. This means they stay out of production and we have to deal with
breakage.

Usually I can just revert to an earlier version and file a bug report,
sometimes I have to cherry pick.

Having developers "fix" things in production under time-stress at 3am without
proper code review sounds like extremely poor practice.

~~~
walshemj
So you have some one every shift who is familiar with every part of a complex
system and the requisite programming languages.

And reverting part way through a monthy telco billing run might not be the
best idea.

And in my case they also looked after online services and other systems.

~~~
peterwwillis
Preventing calling the devs requires dev to front-load ops. Ops needs triage
documents, remediation strategies, architectural and workflow diagrams,
dependency charts, distributed tracing, intelligent logging, custom
application metrics, and tests they can run during incidents to isolate
causes.

The devs can do all of this on their own, teach ops how to use it, and then
they'd only be called when it was a code issue. But as a dev, you probably
don't know all of the above, so ops has to go to dev and be like, "hey y'all,
if you don't want to be called, this is what we need."

And this is what DevOps is intended to fix: get everyone in a room, talk about
problems, find solutions among everyone. If your org isn't doing this, you can
start the change.

------
dvtrn
I found this line interesting, everything else considered:

 _DevOps is NOT…easily achieved nor implemented_

Debatable, at least IMO. DevOps isn't easily achieved nor implemented if
you're trying to implement ALL THE THINGS to say you did and check-off a
series of "We did DevOps thing x" boxes as so many companies appear to want-at
least from reading various DevOps-y job descriptions lately.

It is easier implemented if, like Gene Kim tells us in "The DevOps
Handbook"-we start our DevOps transformations with a small, sympathetic team
and iterate outward.

------
lanstin
In 1997, when my first code I wrote went live, they gave me a pager and said
"Welcome to Ops." Each time the code in production had an abnormal end, I got
an email. I was expected to fix it. Managers and directors discussed these
metrics.

------
aodin
The change in job description from just "operations" to "developer operations"
over the past decade is best captured by the growing number of systems that
must be managed per person, often larger by a factor of 10 times or more (e.g.
going from 10 systems per person to 100+).

I appreciate the self-reflective views on this "new" job role, but they often
seem to miss the rather basic market forces at work. More systems, more
automation, new skills and demands.

------
abjKT26nO8
Could web developers please stop using floating top bars? It breaks the UX of
the "Page Down" button. It's annoying beyond imagination. Thank you.

~~~
Roboprog
I resisted. Management insisted.

Good thing it was a non public app instead of a blog

------
zwkrt
Sidebar: does anyone else get the feeling that infrastructure as code is just
cloud vendor lock-in by another name? Especially since the output of the code
ends up being wild unstructured JSON/YAML files with no spec or discernible
schema.

My favorite is how in the Microsoft toolchain you can build a CI pipeline in a
visual UI on the right that automatically updates the YAML file on the left.
“We know you don’t want YAML, but we also know your boss says you need infra-
as-code...”

~~~
JBerlinsky
You might want to look into Terraform [1], which works across multiple cloud
vendors, with pluggable open-source "providers" facilitating the communication
with the backend cloud platform.

[1]: [https://www.terraform.io/](https://www.terraform.io/)

~~~
LIV2
Terraform is great but if you want to change to a different cloud provider you
pretty much have to rewrite everything

~~~
eropple
Terraform is _okay_ , if you don't value things like "loops that aren't an
awful hack", but in 2019 Pulumi is significantly better and their cloud-
agnostic implementation actually kinda works okay.

~~~
snlnspc
Pulumi would be perfect if the community edition was simply self hosted
without support vs free with a single user only. At a stingy small org, I have
no hope of ever using it instead of simply installing Terraform when the
starter edition excludes secrets management and the API.

~~~
eropple
It's a bit rocky and I've been filing bugs to help, but S3 state support is
strictly no worse than Terraform's and supports secrets just fine.

~~~
snlnspc
That's good to hear! Admittedly the main issue for us is convincing others
that 1 user at $50/month is justifiable.

~~~
eropple
S3 support's free. It definitely works well enough.

------
omani
cant believe so many people still get it wrong.

devops means Developer/Operations.

means, the sysadmin/sysops/ops guy now has to know things about development to
center his daily-doing around code (ideally the code dictates what is going on
[Infrastructure as Code]).

a very simple example to this: back then the ops guy or sysadmin did
everything by hand. today, he uses code to get things done. bash vs. a HTTP
JSON API. manually by hand vs. ansible [markup language yaml], etc.

so many people just dont know or understand what devops means. but it is so
easy.

it is the fusion of development and operations.

thus, while you need two people back then, now you only need one person to do
the same job.

a devops is a sysadmin who ideally knows how to code.

a devops is a developer who evolved into a sysadmin or is doing both because
he learned it.

devops is development/operations.

it has nothing to do with agile, waterfall or a "mindset".

------
auslander
CircleCI trusts 8 analytics companies with your source code and API tokens

[https://kevin.burke.dev/kevin/circleci-is-hopelessly-
insecur...](https://kevin.burke.dev/kevin/circleci-is-hopelessly-insecure/)

------
rurban
It certainly did exist when I started my career a decade earlier than this
guy. It was called the integration team, which was responsible for
coordination of the release. In the 80ies.

It was also responsible for the build servers, the SCM, and checking and
prioritizing all the different parts and fixes of the various dev teams, and
checking back with the PM and testers. Only hackers did cowboy coding,
companies preferred processes and best practices. That's it was called
"Software Engineering" and not hacking. I heard Margaret Hamilton coined that
term in the 60ies for the Apollo project, which she managed.

~~~
gaahrdner
Get 'em! Pretty sure it's just called "teamwork!"

------
sytse
I think the evolution is Waterfall to Agile to DevOps.

1\. Waterfall is a cycle time of weeks

2\. Agile is a cycle time of days

3\. DevOps is the same cycle time as agile but delivering the the user instead
of just showing a demo at the end of a sprint.

------
njharman
[The (or any) Internet didn't exist when I started as a developer]

DevOps is about how CTO organizes engineering.

Is there an Infra team that supports everyone, maybe a separate
Ops/security/etc teams that supports everyone. Then a bunch of App/Service dev
teams. This is not devops.

Or is there only* a bunch App/Service teams, each wholly self-supporting and
independent. Each doing their own infra, dev, ops. Maybe some teams' "Service"
is used by only other internal teams (point being you can break up scope as
fine grained as you want). The key is teams are self-supporting, taking care
of the entire life-cycle of an App/Service.

Most "devops" sort of isn't or fails cause it's not "deployed as an
organizational system, top-down (from CTO) across the whole organization.

All the things people typically bike shed over re: devops don't matter. Are
you independent and self-contained? Are you responsible for entire life-cycle
of Product/Service/App/Etc? Does the buck stop with you? Then you are devops.

~~~
vardump
> [The (or any) Internet didn't exist when I started as a developer]

Internet is widely considered to have started in 1969. Although I'd consider
something between 1982-1986 as the years when it truly became an international
network.

I've been in this business only for less than 25 years, you make me feel so
young. :-)

During that time, internet has just always existed. Ah, those loong boolean
logic web search (hxxp://altavista.digital.com/) queries during my early
years.

------
Nursie
> DevOps is NOT… a job title or role

In which case DevOps as it exists in the real world is roughly on a par with
'agile' practices in the real world. Nothing like they were originally
conceived, and nothing like the wildly optimistic descriptions of them that
are shared amongst practitioners.

~~~
MTZ-x86
Agile: We don't have a formalized process of developing software, just wing
it.

DevOps: We don't have admins, so you are reponsible to run the systems you
develop.

This is at least how I perceive it. But the second point is not the worst
thing that could have happened. I hate to write configs as much as anyone, but
it has given me a better perspective on how to design systems that actually...
you know... run.

------
thinkingkong
Generally Ive found tight feedback loops to be incredibly productive. Whether
its for product, release, design, development. Having the ability to make
modifications with feedback from customers, systems, business partners and
colleagues changes everything.

~~~
rmetzler
Yeah, the faster you’re able to iterate, the faster you can improve.

~~~
Ididntdothis
Only if you make meaningful iterations. I see a lot of demand for iteration
coming from above and suddenly you are in this cycle of fixed length “sprints”
where you worry more about releasing on time instead of meaningful releases.

------
suyash
One could argue that it still doesn't exist today at most companies. I think
whatever most people call DevOps is an evolution of Sys Admin role in the era
of Cloud Computing.

------
gdulli
Lots of things people think of as crucial today didn't exist when I started.
And the overall ratio of successful work to unsuccessful work was just as high
then as it is now.

~~~
jayd16
But the definition of success evolved as well. Richer experiences are
expected.

------
shipit
The goal of DevOps is to get code as developed to testing, staging,
production. The deeper objective should be transfer of context- what is being
deployed? do we know how? does it work as tested? and the who. As much DRY to
reduce surprises and increase precision all through dev to deploy cycle.

Some of the best open source projects have CI included- thats the right
approach to DevOps -- its not an alien, bolt-on after the fact practice, its
code that takes care of code.

------
slowhand09
IMHO DevOps has been around for many years. Is has just evolved as tooling
became better and eventually tooling to connect tooling became better. It
became META. When I started doing database work we developed and implemented.
We learned to produce reports on what we'd implemented by querying system
tables. We learned to use those reports to produce scripts that could recreate
the database on the same or a different platform.

------
damontal
I miss Devops Borat.

[https://twitter.com/devops_borat](https://twitter.com/devops_borat)

------
ConcernedCoder
DevOps didn't exist... fella, let me tell you something, when I started, color
screens were not the norm and you kept your homemade double-sided 5 1/4"
floppy disks ( thank-you paper hole-punch ) in a plastic flip-top container
with the contents written on the label in black sharpie.

------
LoSboccacc
it wasn't his own thing and there wasn't the scale yet to have it was a proper
position, but when you were moving connection strings in a config file and
linking it in etc you were doing devops. when you were building a shell script
to fetch pinned dependencies so that the environment was repeatable, you were
doing devops. weekend you were automating the package build/install and
putting these files in CVS, you were doing devops.

heck we had versioned openpkg scripts in 200something to quickly rotate
servers during upgrades, that wasn't called with fancy names and wasn't a
standalone role but the seed was there.

------
algaeontoast
Well, I wouldn't exactly call Dev Ops a productive career path. Most senior
engineers I know seem to think it's a surefire way to drive your career into a
dead end.

------
alexnewman
Devops has been standard practice at every financial firm i ever went to. Back
then it meant devs where a pager. Ops also wore a pager and would support us.

------
luord
> “a developer with operational tendencies”

Hmmm, I like the ring to that, might put it on my linkedin profile.

On the article itself, can't say I disagree.

------
awd
I like to comment DevOps from a security perspective, a trend I noticed in my
day job.

Windows is/was often bashed for being insecure. Lots of that stems from the
decades of development related to centralized management solutions. A default
windows workstation in a domain setting will open a bunch of ports, a bunch of
which can be used for command execution. The attack surface for this system
includes, but is not limited to:

\- Remote access with local admin users via tools such as SMBExec, wmiExec,
DCOM, Psexec, Powershell remoting

\- Remote access domain admin users access via the same

\- Local/domain admin access via RDP

\- Remote domain admin access via group policy

All these have had their own associated vulnerabilities over the years.
Examples are SMB relay attacks, which enabled an attacker to abuse flaws in
NetNTLM and obtain access to machines by relaying other people's credentials.
And then we're not even talking about the 'real' exploits, Eternal Blue,
Eternal Romance, Blue Keep, MS14-068, MS08-067, and on and on.

Pentesters, researchers and Microsoft have been hammering away the kinks for
years now. The 'fixes' and root causes for each individual issue are well
understood and each new domain functional level increases the security of a
default windows Domain by leaps and bounds.

When you look at the Unix/Linux side you'd see that no such attack surface
ever existed. You manage your systems over SSH, and this can still be bad, an
easily guessable root password shared between Dev, testing and production is
still a death sentence. But by default there were no tier0 systems in your
network, apart from those of sysadmins.

But now with DevOps things are changing on that side. With Ansible, Puppet,
Terraform, your various container management systems, the CI pipeline, jenkins
and numerous development teams able to push both to infra repositories and
your actual products this has changed:

You use an automated CI pipeline? Any system in the chain is a tier0 system.

Your developers are maintainer status or higher in your source repositories?
Then they are domain admin or equivalent. They can disable protected branches,
push a backdoor, and watch their attack propagate through the pipeline.

Did you make it inconvenient for your developers to access various build
systems? Then they are sharing credentials to these systems over your company
chat.

It seems, from what I've seen so far, that while the 'architecture' of modern
mass centralized IT management and development is more secure. You can't relay
an SSH key for example, like you can in NetNTLM. But the institutional
knowledge isn't there yet. New attack surface has opened up, and infosec
people have not yet completely caught up with the new 'eggs' in the basket,
even if they are aware.

~~~
crankylinuxuser
Exactly!

I've talked that this idea of "devops" is terrifying. It turns a whole bunch
of devs whose security mindsets are in "eh password1! is great" because of
convenience. And it turns them into domain admins. Gates are good... Although
that means in having "Dev" and "Sysad" as 2 different groups. And I as a sysad
can and will actively say "NO" with a reason, because something didn't pass a
sniff test.

And with the whole AWS push on everything, do you know what else it does? It
gives the devs an unlimited line of credit that company has to pay for.
Nowhere before did sysads, devs, or other non-manager and non-C-level have
unfettered access to the company wallet. And right now, if you have IAM access
in AWS, you do.

------
louwrentius
I thought that DevOps means: "doing operations with a development mindset."

That captures the gist of it imho.

------
gaahrdner
It didn't exist when I started either, but talk about a let-down of an article
conclusion!

------
oliverwiegers
DevOps is pretty much not a technology but a mindset

------
test111
It's a critical skill now

------
trollied
All that’s really happened is that developers have learned or taken on the
responsibility to build and deploy to production, and also be knowledgeable
and responsible for the development stack, hence “full stack developer”.

Unfortunately, it’s incredibly dangerous if everyone involved doesn’t know
what they’re doing.

Then kubernites and docker turn up. To make things easy?

No. They’re making the infrastructure more difficult to manage so that the
developers can deploy things easier.

Developers should not be deploying. Large businesses should not be doing very
regular deploys to live - it’s simply too risky without a shitload of testing.

Everyone praises startups with a few million customers that are open about
their fuckups/downtime, but they are almost always because of their shitty
modern deployments and lack of testing.

Crikey, look at Monzo. “We use a shit database and fucked up scaling it, but
sorry your transactions failed and you looked like a twat buying your coffee”.

Try and revolutionise industries, fine. But ....

~~~
YawningAngel
If you want a database that is arbitrarily horizontally scalable, there aren't
really good open source options.

As a former employee it isn't clear to me that Monzo couldn't have used boring
stuff for a while and switched to C* or some equivalent later when it was
genuinely needed, but it's worked well enough for them that I'm not inclined
to critique the decision post hoc.

~~~
trollied
Their last outage worries me greatly.

~~~
YawningAngel
I think there's a clear critique of Cassandra in there, in that it shouldn't
even be possible to configure nodes to serve data they don't have.

Monzo... It's very hard to say. I think based on the contents of the blog post
they acted reasonably and competently (yeah, the test rollout was a badly
designed test, but that's easy to say post hoc). The only fair basis for
critiquing Monzo would be if they had notably more averages than is 'typical',
and we don't have the data or conceptual framework required to do that. As a
customer, I can't think of risks that Monzo exposes you to that aren't also
present with other banks. I bank with them ¯\ _(ツ)_/¯

