
Why One Programmer Doesn’t Do DevOps Anymore - jjmojojjmojo
https://lionfacelemonface.wordpress.com/2015/03/08/devops-is-bullshit-why-one-programmer-doesnt-do-it-anymore/
======
lasermike026
Sys Admin/Engineer here. We had it right 15 years ago. You couldn't become an
admin unless you started as a programmer. You needed a deep understanding of
your OS and good knowledge of C. At that time the devs ("programmers") were
less knowledgeable than sys admins. You had to have to skills to analysis,
debug and fix their code. When sys admins stopped coding the result was weaker
admins. Sys admins ought to code within their domain, engineer, and maintain
their systems for performance and high availability. It is long over due that
sys admins return to their roots and get ahead of the pack.

~~~
mrjatx
I'm part of the recent sysadmin generation that had no programming skills and
grew up doing what I consider more "application administration." Lots of
clicking Next with no real understanding of what to do if that Next button
quits working. Stack tracing? Procmon? Miserable understanding of DNS and
networking, what's a /24? These were things senior admins knew and used once
the tickets went up the chain because customers were screaming.

I was always (well, not always, once I'd proven myself) a senior
admin/engineer, now I focus almost purely on AWS automation, and it bums me at
out how many admins are completely happy with just being Next button admins
who are absolutely AFRAID of touching a bash or zsh shell, or even a Cisco
router without the UI.

I suppose it shouldn't bother me, it's done great for my career.

~~~
uiri
Are you talking about sysadmins on Windows? I can't imagine a Linux sysadmin
being afraid of bash, zsh or a ios shell. My impression is that knowledge of
some combination of bash, perl and python scripting is a basic job
requirement.

~~~
DyslexicAtheist
the root of the problem (the idea that sysadmins can't program) might come
from the fact that every company whether furniture/clothing store or any other
sizable non-tech business needs sysadmins. But only if technology is your core
business you need good developers. This probably leads to generalization of
sysadmins not knowing their bits/bytes. Sure guys working for ISP's and in
datacenters are a different breed but they still get thrown into the same
category because of the job-title.

~~~
fsniper
I think the problem is, some companies are expecting a sysadmin to be the "IT
guy". The one who is only expected to call contractors to fix that recurring
network problem, and install enough Windows boxes for the new hires.

------
mugsie
DevOps as it has been implemented in tons of company _is_ bs.

Its like the "Agile" push that companies thought they could get by buying a
tool, and changing nothing else (release schedules, feature based release
dates etc).

My favorite way of looking at it is simple - replace the word DevOps with
empathy. That is the core of the idea, so lets try it - "Empathy Team",
"Empathy Person", "Empathy Engineer".

DevOps is about dev and ops teams communicating and having empathy for the
requirements of the other team.

~~~
mhurron
> DevOps is about dev and ops teams communicating and having empathy for the
> requirements of the other team.

Most posters here don't seem to get it. Most replies appear to be the idea
that DevOps means you fire your Ops people and the Developers do Ops previous
job and their own Development job as well.

~~~
icedchai
Or, in the small company case, you never hire any dedicated ops people to
begin with. You have developers fighting fires all day, getting interrupted
constantly, causing them to hate their jobs.

~~~
gibsonje
Small company. We don't have many problems. We have a couple pain points but
we fix them over time. When hosts die, it generally doesn't mean anything.
There's redundancy in all things. I don't find myself interrupted often.

Previous job had a "Systems" team and a "Database" team. Both were just teams
of obstructionists.

"Oh you need something today? Well, if you get a jira ticket made I could look
into getting the requirements written up sometime in the next 2 weeks, to get
the work assigned out next month."

Not saying that's the norm, but damn I hope I never experience that again.

~~~
gknoy

      Both were just teams of obstructionists.
      "Oh you need something today?
        ... get a jira ticket made 
        ... work assigned out next month."
    

It is possible that you might have a different opinion of the situation if
viewed from their perspective. It's likely not deliberate obstructionism, but
rather them trying to triage the requests so that they can be most effective
across all customers. (It's possible that your ops team did not adequately
communicate this to you.)

In my current job, I am the sole developer responsible for maintaining a
certain set of our codebase. I have multiple internal customers, each of whom
have separate (external) customer-facing needs. I have enough tickets created
for three people working full time, so I am constantly having to triage which
are the most important.

Part of the solution to this was to have a proposed plan for when to work on
things -- I can then say, "Thank you for this well thought out ticket. I will
be able to work on that in June." The most important thing is that my users'
opinion about the criticality of things does not always match the overall
picture, and I am always having to choose which tickets to postpone.

~~~
ploxiln
This is a good point, but it's a sign of dysfunction. I've seen the case when
50% of the issues are "P1 highest priority" (and most of them forgotten in a
week or two). People making requests need to get some idea about what
development effort (and maintenance!) costs. I don't know a good way to do
this, except to be in a very small company.

------
hijinks
As a "DevOps Engineer" (use to be called Sys Admin) I feel a lot of posters
here are using the term wrong.

A lot of us old sys admins are now calling ourselves devops because times are
changing. To me devops isn't about taking a developer who likes ops to save
some money. Its about taking ops out of their walled garden and into the
development process. As someone in ops, I'd much rather be part of the
development process as I'm the one supporting the code in production.

If its a choice of message queues.. I'd rather have a vote to the one I know
about supporting.

It isn't really about cost savings as some companies I've been in I'm one of
the highest paid engineers on the team. Its more about bringing ops into the
process.

~~~
leftnode
In addition to that, I always thought DevOps meant operations people using
development practices and tools to automate and simplify their lives? Does it
have to be more complex than that?

~~~
caw
If you take your definition as "ops people who code and automate", you get a
really efficient ops team who can orchestrate several hundred or thousand
servers.

You still then have a friction point between "dev" and "ops" relating to hand
off. Ops could go full PaaS, but more than likely you'll end up with a "cloud"
infrastructure where your dev team will have to learn ops, or the traditional
model where stuff gets thrown over the fence. Good luck deploying that service
because there was no capacity planning and it'll take 6-8 weeks to get
hardware from the vendor, regardless of whether ops can rack it and build it
in under a day.

It's far better to have an org overlap so that ops is involved in dev planning
to help address roadblocks, assist in capacity planning and management, as
well as help reduce friction in tasks.

~~~
late2part
But, the purists would reply, if you simply outsource all your hardware to AWS
or another place, you don't ned that ops or capacity planning. Until you do.
This weekend AWS ran out of i2.2xlarge instances in a particular AZ, for
example.

~~~
ecnahc515
Hate to tell ya, but at anything other than the 'prototyping stage', you'll
probably need a dedicated OPs guy, AWS or not.

------
wtbob
From one of the comments:

> It’s easier to train a developer to do sane operating procedures than to
> teach a sysadmin the developer mindset (testing, revision control, breaking
> down to logical units and similar).

Having done both jobs, I don't really think that's true, unless by 'sysadmin'
one means the type of low-skill position which resets passwords. A system
administrator is responsible for administering a _system_ of hosts and
software; his should be a _more_ complex job than a developers. He should be
responsible for test systems as well as for production systems; his scripts
should live in source control. I'll grant that functional decomposition _may_
not be as vitally important to his work as it is to a normal developers—but
then, his scripts will inevitably grow to be pretty complex, so maybe it will
be after all.

That doesn't mean that developers are stupid either. Sure, at
BigEnterpriseCorp a developer is just a seat-warmer who implements directives
like 'add a method to the Foo class which takes two arguments, adds them and
returns the result,' but a real developer is a student (whether formally
taught or informally self-taught) of computer science; he understands and can
choose & implement both algorithms & data structures in order to solve a
problem.

A good DevOps hire is going to have some skill in both areas, not as a weak
sysadmin and a weak dev, but as a strong sysadmin and a strong dev. A good
DevOps team is going to be _more_ skilled (and expensive) than the equivalent
dev and ops teams, but it will also be more productive and less error-prone.

If you're trying to build a devops team from Initech, Initron and Initrol low-
skill employees, you're probably going to fail. If you're building a devops
team to reduce costs, you're probably going to fail. But if you're building a
devops team to increase profitability and competitiveness, you may succeed.

~~~
acveilleux
In the Unix/Linux world, a lot of the Sys Admins I've met have been very
strong technically. Knew at least a scripting language and basic C
programming. The trend however has been for less and less coding skills and
more superficial. Sometimes not even any shell scripting skills.

In the Windows world, a lot of the Sys Admins do not have those skills. It's
changing, things like PowerShell and various others are helping, but a lot of
Windows Admin are just not that comfortable away from the GUI. I think the
comment you refer to is by someone with a Windows dev background. GUIs don't
version...

The mainframe world folks tended to be very silo'd (and IBM engineers were
never far to hold hands.)

------
SeanDav
So many misconceptions here as to what DevOps is. I think _mugsie_ comes
closest with his use of the term "empathy". This is very very close, but I
would add one more term "cooperation".

DevOps, when implemented correctly can have enormous, order of magnitude
increases in both productivity and quality, but it is not voodoo, despite this
almost unbelievable claim. It involves empathy and cooperation and it requires
this because in order to implement DevOps in a meaningful way, an organization
has to vastly increase communication between "silos". It has to formalize,
document and automate as many processes as possible.

DevOps is NOT easy to implement and involves a cultural shift and absolute
management buy-in. It is best implemented in small projects, where clear wins
can be demonstrated. Use this success to bootstrap the process to bigger
projects.

I do agree with the OP that if you have a DevOps team, you are probably not
doing DevOps. The whole point of DevOps is to get rid of silo mentality. No
more us and them, but all us.

Implemented correctly, everyone is DevOps.

------
passive
It's fascinating how much this engineers experience shares with mine. We both
appear to have similar approaches to working with technical systems, that has
led to skillsets that are more broad than deep. We've both had interview
problems because of this, and we've both ended up in DevOps.

I've also had more or less the exact same experience with DevOps, though with
(hopefully) a happier ending. I was on a team working for a client that
desperately needed "DevOps-type" things. So I built them, and their
infrastructure team ignored them.

Eventually we stopped working with that client, and I was moved to a different
organization, where each development team was doing it's own thing,
operations-wise, and had no interest in adopting a uniform approach, so my
(short) efforts there were also for naught.

But then I had the opportunity to come back to the first client, as their
DevOps Manager. Before I took the job, I talked to their current development
organization about what their pain points were, and what actions they would be
willing to take to alleviate them, and I built a strategy around that. A key
part of this strategy was gradually eliminating the need for a DevOps team, as
I firmly believe the best application of "DevOps" is as a joint project
between Development and Operations to optimize shared processes. Once the
friction is minimal, you don't need a team there anymore. So we have a
developer embedded in our meetings, and have more or less full control of any
infrastructure.

~~~
Fuxy
I think there is a fundamental disconnect between what DevOps is and the
classical business organization structure.

DevOps is not a department and it will not work if it is a department DevOps
is a way of doing things to achieve maximum efficiency and everyone on
development and operations must embrace it for it to work.

If either development or operations resist the change then either the
resisting members should be eliminated or the switch over to the DevOps way
aborted.

DevOps doesn't work without cooperation and effort from all team members.

~~~
digi_owl
So in essence, devops treat every projects as if it was a startup? That is, to
hell with departments, managers and lines of communications, one project, one
team, one office?

~~~
Fuxy
Not quite, Managers can be separate, accounts can be separate, advertizing can
be separate etc. but Development and Operations must be treated as one unit no
separation.

In other words anything that can have a line of communication and still work
can stay as it is but in order for DevOps to work development and operations
must essentially become one with both developers and sysadmins working
together on architecting a flow that works for both of them.

One office and one project is not really a requirement but it helps a lot in
the beginning however if you can keep the cooperation a culture alive even
working remotely could work.

The key is people who are always willing to learn and try new things and
putting effort into change and staying on the same page with their teammates.

~~~
michaelt
So who gets the production system and database root passwords?

------
KaiserPro
So this is the thing. There are three things that really go against a true
"devop" system:

1) cost.

Ops are cheaper than devops, sometimes twice as much. Yes yes, flexibility.
But thats a misnomer. If your dev teams are doing donkey sysadmin work,
because they don't have the experience, or are up all hours fixing things,
you're going to endup with a high staff turn over.

2) division of labour

As a company grows, they can employ more and more specialist staff. This gives
them a competative edge as they are able to make nuanced decisions based on
skill, or experience. A decent storage admin will be able to build a storage
system that will scale relaibly, cheaply and with less downtime than someone
who is not a specialist.

3) burn out.

A human is not designed to multitask 14 hours a day. Coding, testing,
supporting, infrastructure procuring, for anything other than the smallest
company is too much for a single role.

There are great things that come from a devop "culture"; greater personal
autonomy. However like all things, its not one size fits all. There is a great
deal of arrogance where people assume that a sysadmin never automate anything.
or that they can't program.

They can and do, you know why? because they want to carry on searching ebay,
scamming $someone boasting about network speed, or just plain getting
shitfaced.

~~~
RankingMember
I was with you until "because they want to carry on searching ebay, scamming
$someone boasting about network speed, or just plain getting shitfaced.", but
I'm guessing you're just being cheeky.

~~~
gknoy
I was thinking it was a BOFH ("Bastard Operator From Hell") reference. [0] The
BOFH is a fictional sysadmin who is worse than any you have ever had, in
entertaining ways. He basically does exactly the things the OP mentioned,
while dealing with clueless management, undeliable vendors, and a Sith
Apprentice^W^WYoung Coworker who learned well, and is constantly trying to
stab him in the back. He's like a mix between Catbert and Wally. ;)

0:
[http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/](http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/)

------
trcollinson
I am a big believer in DevOps to the point where I often consult to "implement
DevOps" at organizations. The frustrating part if that organizations seem to
want to create "DevOps Teams" to augment their other teams. An organization
cannot have a "Development Team", an "Operations Team", and then a "DevOps
Team". DevOps is a set of good principles which help to guide the process of
making, releasing, and maintaining good software products. These are by no
means the only set of principles and organization should live by, but
Developers and Operations should work to implement these principles because it
will make their process more efficient. It is certainly not a matter of simply
tossing in yet another department full of people and hoping that more cost
efficient work gets completed.

When I work with clients the inevitable question comes up: "What Principles
should we implement then?" and the answers are usually more questions really.
How long does it take developers to set up their working environments and get
to coding? How is code tested and prepared for deployment? Are stakeholders
feeling secure about deploying this new code to production users? Can we
deploy effectively every time without deployment related issues? When
something bad happens in production, what is the recovery procedure and how
long does it take?

The answers to these questions and many others often show that there are a few
issues. First, developers and operations aren't speaking to one another enough
and aren't really working together. Second, there is very little trust in the
current environment and that lack of trust is made up for by adding more human
intervention. Third, there is very little automation and this causes the lack
of trust but few people what to change their habits to fix it. And finally,
there is such a great divid between engineering and operations that the pains
felt by one group and not shared by the other.

DevOps has the principles needed to start to solve these problems. However, we
don't need a department to implement those principles. We have enough
departments.

~~~
sombremesa
Well said. I think this is what the author of the OP was also getting at -
it's better to grow the culture (while sharing responsibility) where it
belongs rather than purchasing it wholesale and shoving it down dev throats.

As an employee of a large software focused corporation that does many of these
things right, I had no idea how bad things can get for smaller, newer
companies that haven't figured all of this out.

~~~
alxndr
Even small companies can have momentum and unanalyzed patterns that are hard
to change.

------
ownedthx
Replace DevOps with QA and an equally valid article.

Dedicated QA teams should be avoided unless truly necessary. Otherwise, you
have the same problem of developers throwing over under-tested code to a group
of separate people that are smart but treated poorly.

A QA guy who loves to write automation gets the same promises... "oh you'll be
programming mostly; manually test maybe once a month". But when they get into
the door, it's the exact opposite.

Developers should all be passionate about QA (read as: the quality of their
product), as well as how their product is actually deployed (read as: executed
in the most important of environments--production!).

------
TranceMan
Last month I went to the UK PHP Conference where Jeremy Quinton spoke about
DevOps. The biggest take-away I got was:

'Do not set out to create a DevOps team or employ people for that role'

My interpretation was the role simply doesn't exist. If your company has a
team of developers and a team of people who keep the wheels turning - in an
ideal world you already have a DevOps team as long as they collaborate. If you
also employ smart people you will get 'developers that know how to do sysadmn'
and 'sysadmins that know how to develop' \- definitely this indeed should be
embraced but not expected.

------
hedwall
Obligatory lusis quote and link...

"Devops means giving a shit about your job enough to not pass the buck. Devops
means giving a shit about your job enough to want to learn all the parts and
not just your little world.

Developers need to understand infrastructure. Operations people need to
understand code. People need to fucking work with each other and not just
occupy space next to each other."

[http://blog.lusis.org/blog/2013/06/04/devops-the-title-
match...](http://blog.lusis.org/blog/2013/06/04/devops-the-title-match/)

------
clavalle
>You can’t have DevOps and still have separate operations and development
teams. Period.

I cannot disagree more. You can't have developers of your product knee deep in
DevOps and be effective (at a certain size, smaller teams with less demanding
operations needs can).

DevOps should have very clear demarcation of responsibility in an
organization. What the OP describes is an blending of responsibilities and
that is the root of his problems.

DevOps /is/ operations. It is just that simple. It is design and automation of
support systems. Nothing more, nothing less.

~~~
efuquen
If 'DevOps' is operations, that what is the point of the term? Ops teams have
been doing automation of support systems for a while, so what exactly does it
mean to be DevOps? I think that's part of what the OP is getting at, if your
fancy 'DevOps' team just ends up being separated and isolated from the dev
team you're back at square one, you've accomplished nothing and the same
divide exists with a new label.

As someone who works on the dev side, the ops side, and the mixture of the two
I feel the OPs pain acutely and have had mixed experiences. People on the dev
side need to understand and willing to learn about ops, it's not rocket
science and you don't have to be knee deep in it to get the basics down. When
ops becomes this black box and becomes a bottleneck is when you start to see
dysfunction.

~~~
clavalle
>If 'DevOps' is operations, that what is the point of the term?

It is taking more of a gestalt systems approach to the entire support
structure. This means more intelligent support programs and treating it like
designing and delivering a software project (or, more accurately a series of
subprojects) as opposed to a loose amalgam of scripts.

This also requires the product development team to be keenly aware of that
support architecture and how their decisions factor into it especially in
terms of testing, deployment, and module interactivity.

> Ops teams have been doing automation of support systems for a while, so what
> exactly does it mean to be DevOps?

DevOps has been around for a while -- way before there was a word for it. I
think DevOps has come into its own, recently, though because platforms have
become hugely more flexible while at the same time a consensus built on what
it means to manage those systems. The difference, I think, is whether things
are built to remain somewhat static or if change is designed for. Are
development ideas like encapsulation and decoupling etc designed into the
system? Smart sysadmins have been doing this stuff for a long time but now
organizations are taking the approach into account all the way up and down the
chain. And, yes, that means that product developers need to be fully aware of
operational needs but, in my opinion, I don't think that means they build the
operational tools themselves.

>your fancy 'DevOps' team just ends up being separated and isolated from the
dev team you're back at square one

Demarcation of responsibility doesn't mean demarcation of information. The
product development team will need to be intimately aware of what is happening
in operations and vice versa. In a DevOps implementing environment it is
unavoidable.

> the dev side need to understand and willing to learn about ops, it's not
> rocket science

These days it's getting pretty close. I don't think it is fair to product
developers to say "Listen, we like the idea of what Docker can do for us so
you are going to learn it inside and out." Somewhere you have to say "Hey, we
probably want to implement Docker. Ops, dig in. You know how our product is
currently structured and how operations currently works with that. Try to
figure out what needs to change and what doesn't. X and Y from the dev team
will be made available for deeper discussions and testing as needed. When the
overall gist of what has to change is known, we'll all get together, ops and
produce dev, to get everyone on the same page." It's a bleeding of
information, not a full on 'shake it all up' scenario.

Neither side can be a black box anymore if the current state of software
development and delivery is going to be taken advantage of. But neither can
everyone know and do everything or context switch as needed.

Keep in mind: I am not a DevOps theorist -- maybe I have it completely wrong
according to the people that came up with it but this is my practical
understanding of the movement.

------
mgrennan
Former embedded developer, former systems developer, former language
developer, former application developer, former senior systems administrator
and now DBA. (30 years) Simply put, the problem IS TITLES. They cause the "not
my job" problem. If you've been called, help fix the problem. If you don't
understand the problem, stay with it, maybe you'll learn something. Just
because your a "title" doesn't mean your s--t don't stink. Unless your are
designing your CPUs, writing the languages that your write your OS and the
applications that run your hardware, you are riding on the back of someone
else. Do what's ask of you and ask how you can make everyone's job better.
Then you'll be employed and paid well for a long time.

------
cubano
I think the most striking point made here was the one about culture and how
without everyone buying into the DevOp paradigm, its just another unused tool.

I say this because I've lived it...the last 3 (three!) companies I've
contracted at were pushing dev code to live servers! Even after I setup
svnserve and created the repos, no one was really interested in changing
workflow. I choose svn because, lets face it, it has a much lower learning
curve then git.

It wasn't until I absolutely refused to work on the live server that other the
devs grudgingly started using SVN...I'm sure they are all back to FTPing to
the live server now that I'm gone.

Without the proper culture from above, things just aren't going to change.

------
mrjatx
I'm devops and in my opinion the title infers this; my job is to keep the
communication between developers and the operations teams in sync and
copacetic. Usually, Ops doesn't know dev and dev doesn't know ops so if
there's nobody to translate and explain things in each others technical
language things can't be done quickly and reliably. The wrong environments
will be configured, things won't have backups, it will be like the wild west
and things will be stressful for both sides.

My job, just like it was when I was a sysadmin, is to keep things running
smoothly. To optimize both sides of the coin and to take things off their
plate that I myself can deal with that might be a blocker to them finishing
tasks and to us getting features out or server migrations done.

Some devs know ops and some ops know dev, those guys, more often than not I
just check in with and see that things are running smoothly. For the other
folk I try to create systems and workflows that get blockers out of their way.

I try to be a useful and reliable tool to both sides as much as I can be.

Hopefully the way I've been doing this for the last two years since I moved
into a more Devops related role has been helpful to everyone.

~~~
pbhjpbhj
Completely OT: "copacetic", new one for me - where have you come across this
word; the etymology is intriguing to me.

[http://en.wiktionary.org/wiki/copacetic](http://en.wiktionary.org/wiki/copacetic)

~~~
mrjatx
Boy that's interesting, I've never thought about it. I'm an American who grew
up in Germany but have an Italian and German family who grew up in the
projects of NY (the Love Canal, Niagara Falls). My grandmother (german) and
grandfather (italian) married and their families constantly fought and pretty
much hated one another.

I remember them fighting (mostly arguing) and the men going "Ey ey, lets keep
things copacetic." (They might have said capasetti)

I always thought it was a pretty and a smart sounding word. I guess not.

~~~
pbhjpbhj
> _pretty and a smart sounding word_ //

Sounds good/smart to me! The wiktionary page mentions possible Italian-NY
etymologies so you fit in there. Funnily enough I was going to try and be
clever and guess you were a New Yorker [based on that] but noticed elsewhere
you appear to be in TX.

Interesting, thanks for responding.

~~~
mrjatx
Army brat, I've been everywhere!

------
kokey
I was recently looking for a new job, and from the job specs and interviews I
have realised that 'DevOps' means very different things to many people. I am a
system administrator who likes to automate things and write code, and also
spent decades supporting developers and good Agile environments, etc. and
production ops environments. Some companies think DevOps is a developer who
knows how to deploy services on Ubuntu, and their interview tests the
implementation of algorithms covered in recent CS courses. I removed any
mention of DevOps from my CV, that made things less confusing for everyone.

------
300bps
It seems OP is using the word DevOps to describe what is traditionally
referred to as "special projects". Essentially, the non-technical CIO hears a
new buzzword and hires a team to implement it in an ivory tower without fully
partnering with existing team members. From OO to SOA to Agile to DevOps and
everything in between these initiatives almost always fail. That of course
doesn't mean that OO is bad, or that DevOps is bad. It is just that the
"special project" methodology is terrible.

------
jussij
To me DevOps is a cost cutting measure whereby an organization can save a few
hundred thousand dollars by having one individual not only do the development
but also handle the deployment and the testing.

In theory that will eliminate the need for an additional tester and a second
individual to do the change control.

But those short term gains expose the organization to the massive risks that
the one and only one individual will make a mistake, with no second individual
available to do a double check.

The reality is people make mistakes and by eliminating people from that double
checking process may well save money in the short term, but in the long run it
greatly adds to the of risk failure.

While there are short term gains, the risks to the organization can be
massive:

[http://www.bloomberg.com/bw/articles/2012-08-02/knight-
shows...](http://www.bloomberg.com/bw/articles/2012-08-02/knight-shows-how-to-
lose-440-million-in-30-minutes)

~~~
mugsie
That is actually an example of the old style of IT.

The person deploying the software had no idea how it worked on an internal
level.

The person who wrote the software had no idea how it was __actually __deployed
in production.

They didn't have a quick rollback in place (this is the exact reason you have
quick rollback ability)

The deploy was done by hand, with a user typing. That is probably the biggest
risk you can have in a production system. It means that fully repeatable
builds in test & pre production are impossible.

I am not saying "DevOps" would fix this, but a bit of empathy / comunications
between the dev team and ops team might have averted it.

[http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock...](http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption)

------
mark_l_watson
I have been referred to as a devops person since I have been developing
distributed systems since 1998 and liked knowing and working the full stack,
including hardware.

My definition of devops: small teams totally own their project, iterating on
design, implementation, deployment. I have no experience with one giant devops
team for an entire company.

------
legolin
It seems that the main problem here was not DevOps but rather the culture in
which he was asked to do his work. Creating a devops team whether "by fiat or
by force" doesn't sound like a recipe for a good experience.

My experience is different: DevOps saves me many headaches by letting me
easily deploy things for small clients. A larger company I work with uses Chef
to manage their 40+ servers; the IT department doesn't touch these servers.
Rather, 3 or 4 developers share in the work of writing cookbooks, adding
instrumentation, deploying code and maintaining the servers. It would be handy
to have a full-time guy as well (OP mentioned the amount of work needed to
keep up with just your basic security patches) but we -- as developers -- love
the access we have to our entire stack.

~~~
jjmojojjmojo
I don't want to take away from your point, but you touched on security - I
mentioned it because it was a constant pain point for both myself and for our
teams. Like operations, it's something you can be good at on a superficial
level, and do OK for most things, but really getting it and being good at it
takes experience and time.

I don't think it came through in my post, but having a security expert (or any
specialist) in the mix can be a huge help in making the product better - I
know this from personal experience - you just have to make sure they're
integrated, challenged, and respected in the group.

------
bonn1
Don't know if the concept of DevOps is bs but I know that it's unfortunate if
a developer never did system administration: there are many of this "git push
heroku" kind of devs who pee in their pants if they have to deal with bare
metal and the production servers. It's not that they wouldn't be able too
handle them, it's that they are afraid because they never did and besides,
good system administration knowledge leads always to better application and
system design.

~~~
snorkel
I blame Java much more than heroku. Corporate drone Java developers have a
much greater tendency to only write code (too much code!) and not be
interested in learning anything beyond the confines of their comfy auto-
completing IDE. It seems to me Java developers will howl "not my job!" the
loudest whenever anyone suggests that maybe they can mange their own
environments.

~~~
HelloNurse
In the enterprise Java world, the comfy auto-completing IDE is a smart
strategic choice when "managing the environment" means weeks of frustration at
installing, configuring and fitting together by trial and error an application
server like IBM Websphere and the application. Funny tools like Maven,
Subversion and certain Eclipse plugins add extra puzzles and unpredictability
for the true amateur, but the normal unforeseen complications are usually
enough to forfeit any hope of good automation.

------
chrismarlow9
I'm a dev who learned everything by experience. I'm in my 20's and have been
doing this for ~20 years, so I have a bit of experience with both the new and
the old world.

My take is that the code quality of whats allowed to be in production has
gotten worse, that's it. It's an effect of an abundance of RAM and
virtualization. The abundance of ram lets you just trash it and do things in a
horribly inefficient way. Sure the extra memory can help with time memory
trade off if you have something that doesnt need to scale a great deal, but
processing things inline will still be faster. Developers have forgotten how
to handle streams. Usually forcing a dev to learn a highly functional language
that's immutable can fix these issues (it must involve a large dataset, where
you know they will tax out the memory, and then code review to help).

I mention virtualization because a common pattern I've seen is to write poor
code, and just put more boxes behind the load balancer when things start
failing because the technical debt is so high it's an impossible mountain to
climb. This obviously isn't the right approach and will leave your devops guys
worn out chasing your everyday distributed systems problems that happen
because it wasn't built right for distribution and the dev has never read the
fallacies of distributed computing.

And while you're fixing this technical debt a team of competent engineers
builds a competing product and trumps you hard (second mover advantage).

Maybe people paid a bit more attention when things cost 10x as much as they do
now and every answer (both bad and good, and you can't tell the difference)
wasn't a stackoverflow search away.....

~~~
aaron_m04
So you're in your late 20s and started working when you were in elementary
school? :-)

------
snorkel
Unfortunately the term DevOps has been diluted in the job market. Now most
DevOps job postings are actually just plain ol' sys op roles, where
"development" means writing tools for the actual developers. If you want to
spend at least 50% of your time writing code then avoid any job postings that
mention "ops".

------
unfocused
I'm not surprised by his experience as it mirrors a bit of mine. This was
previously discussed on HN:
[https://news.ycombinator.com/item?id=2674841](https://news.ycombinator.com/item?id=2674841)

Although in his post, he says:

"Another lesson learned: you can’t force cultural change. It has to start from
the bottom up, and it needs breathing room to grow."

I would argue that cultural change has to also start from the top. You can't
change the culture if the leaders of the company don't work with you on it.

The tendency of people at the bottom is to eventually start behaving like
their managers as the managers project what is acceptable and not acceptable
behavior. Trying to start change from the bottom up is a hard, hard thing to
do, with a low probability of success if senior management isn't on board as
well.

~~~
jjmojojjmojo
> I would argue that cultural change has to also start from the top.

This is where the "breathing room" comes in. I think that fostering an
environment where people are free to organize themselves without edicts and
policy produces overall better processes (and ultimately better quality).

I think it's OK for management to lay a framework, provide a particular kind
of training, but it's best to provide goals that forward the company, step
back and let the people closest to the work decide the best way to achieve
them.

------
tymski
Whether you want to or not, it's your choice. However, the push towards DevOps
as opposed to Dev is based on efficiency; it's easier to hire a developer that
knows operations, instead of a developer and an operations employee. Even
without DevOps, a lot of positions are Full Stack, so you need to know front
and back-end development and technology. I don't think it's merely a trend,
it's more or less a push towards saving money and time. If you don't want to
be a full stack eng or a DevOps eng, then you're limiting yourself and you're
limiting the opportunities you'll have. It sucks for those who want to limit
themselves to one specific group of tasks instead of branching out, but it's
just how it is.

------
tomohawk
If you want to make a devops omelet at a place that has an over the wall
mentality, you have to break some eggs (and walls).

I was part of a successful transition to devops from a thoroughly silo'd
operation, where the boss basically gave the dev team the responsibility and
authority to make it happen. He was also able to snag some of the traditional
ops team and make them part of the new team. After a period of storming, the
new team came up with some innovative ways to get the job done. Once the team
realized that they were it in terms of getting it done, it got done. Quality
and job satisfaction improved, cost was reduced.

I've also seen various unsuccessful transitions. In every case, the
responsibility was divided into various silos, which sandbagged everything.

------
sbt
DevOps may be hyped, but it's not outright bullshit. DevOps is just the
realization that you would rather have a fewer people knowing everything, than
many people knowing disjoint areas. The same force underpins the full stack
crowd, or the notion that pizza size teams are more efficient. While this is
not always possible in practice, in my experience it is better if the project
is small enough that you can pull it off.

The memory bandwidth in one single person's brain is simply much higher than
the bandwidth between people. To maximize performance you want to run your
project on as few brains as possible.

------
chomp
How does one reconcile DevOps with the Sarbanes-Oxley act?
[http://en.wikipedia.org/wiki/Separation_of_duties](http://en.wikipedia.org/wiki/Separation_of_duties)

~~~
mugsie
With difficulty. For example - you can still have an Ops team, and they have
the required access. The developers access to production is done on a case by
case basis, with some sort of tracking of who logged in. and why. Also, doing
full pre - merge code review helps, as it is yet another set of eyes on the
code that will be pushed to production...

In short - its not a "one solution fits all", and it is something that needs
to be discussed with you SOX auditors, to find a working solution for your
company

~~~
chomp
Thanks for the response! We keep a distinct separation of duties, but we try
to keep both groups in constant contact to minimize animosity in both groups.
We pick and choose some DevOps ideals to make it fit, was just curious about
other people's experience.

------
mpdehaan2
Having been around since the inception of this space, when things were much
more tools focused, to now, when it's a lot more about "culture" (all good), I
feel 95% of DevOps talks are people discussing what DevOps is. As such, it's
now a metasyntactical label for a lot of things that can by good in various
scenarios. I agree it is time to retire the label and instead talk about those
specific things -- automation, culture, continuous improvement, etc, in nearly
all instances.

------
dsr_
A senior systems administrator can and does write programs for automation,
monitoring, configuration management, and all the little things that are not
customer-facing applications.

I used to be able to use "senior systems administrator" as a short-hand for
those skill sets, but about 2010 that stopped working.

Now I need to use "DevOps" as an imperfect substitute in the title; otherwise
only junior people apply, even though the rest of the ad goes into the same
depth about skills and abilities.

------
stcredzero
_So that’s what DevOps really is. You take a bunch of really skilled,
passionate, talented people who don’t have their heads shoved so far up their
own asses that they can take the time to learn new things._

What you want to look out for is the formation of "walls." People have an
instinct to do this. People will form cliques -- which effectively are
invisible barriers to understanding and honest collaboration.

In a 5 person startup, this is easy. You have one group. Most of the critical
information flows are inside this boundary, and those which aren't, the group
is already intensely aware of. (Investors, customers.)

However, when headcount has started to grow, and when you are forming a
separate _group_ to take on a task that requires intense collaboration, maybe
you're doing the wrong thing? Maybe what you want is a separate _role_ for
people who get embedded in a variety of different groups? What if DevOps
people belonged to DevOps, but also, and primarily, belonged with a particular
group of developers? Then, they could have firsthand experience of intense
collaboration, but also meet once a week with all the other DevOps role
holders and communicate to standardize and avoid duplicated effort?

A generalization of this: Make sure you're building _bridges_ and not
_compartments_.

------
reitanqild
Funny and, unfortunately, relevant, especially the last tweet:
[https://twitter.com/devops_borat](https://twitter.com/devops_borat)

------
peterwwillis
DevOps is not a job. It is marketing speak.

Sure, there's this general idea that development and operations teams should
work together, or of a programmer also having some experience adminning
systems or vice versa. But this is as much a job description as "non-domestic
car mechanic" is a job description. Generalized ideas that don't say anything
but an indication of the background expected of the person.

"DevOps" or not makes no difference; _all_ corporations face the same problems
with multiple segregated teams and people doing whatever they want in order to
get their jobs done, as well as a sort of tunnel vision that prevents one from
properly supporting your teammates or a product. To this end, there is only
one thing that will make a significant difference in the work done, no matter
what you call your team or position.

The best engineer in the world becomes a janitor under the wrong leadership or
teamwork.

------
devonkim
I think the right thing to do is to firstly get rid of the term because it's
become so meaningless and abused in so many ways. At least "cloud" has some
semblance of at least meaning "whatever that we used to do in the 90s for IT
is BAD" and implies great upheaval, but DevOps is a really, really weak
movement in comparison because most companies that are dysfunctional try to
contain whatever DevOps is into their existing (typically broken, so let's
propagate that error 100x, right?) practices.

While I agree that sysadmins should be developers first it is increasingly
less likely you'll find developers willing to become computer janitors and
deal with the ugly realities of operations in, for example, larger companies.
I wouldn't think that John Carmack ever liked having to figure out the
installer scripts and keep them working across all the supported platforms
through all the BS that Microsoft and different Linux distributions could
throw at you, yet that's what a more "ideal" DevOps organization described by
many model would have John Carmack do.

What should have happened though is that instead of hobbling the John Carmacks
and dragging down their productivity, you should have fired the developers
that refuse to cooperate with operations and those that keep actively putting
off writing "boring" code or never documenting things that do matter for
operations to figure out and learn through a failure that something broke.
That kind of behavior is the developer equivalent of not being a team player
at all. It's true that none of these things actually make you a better
programmer in itself, but these things are what contribute to making you more
PROFESSIONAL instead of a cowboy renegade programmer that doesn't care if
nobody else can use his code because "clearly, if you can't figure it out
quickly you don't deserve to use it" (not an exaggeration at all I've heard).

Having been in both development and operations in rather large (and small!)
companies now, I'd far, far, far rather stay as a hermit-crab developer than
deal with the constant dysfunctions of enterprise Faustian nightmares that are
required for you to deal with as an operations engineer. You can generally
ignore a lot of the noise in a super-siloed off R&D organization, for example,
but you cannot ignore your pager going off at 2 am because a random server
crashed causing someone to get very, very angry and demand immediate
resolution when it is your JOB to keep those things going. Hardly going to
write anything really meaningful with that ruining your work week pretty much
every week.

------
diminoten
Is it just me or are development practices (sem ver, git flow, test coverage,
in-house code guidelines, even vcs and CI) the kinds of things that go out the
window when a DevOps team forms, because they're "too slow"? For example, who
has time to ensure 100% code coverage for your script that's going to fix the
problem with your load balancer?

The exact same thing I'm noticing is happening with the "automation" teams
that are cropping up. It's like folks are discovering ways of getting around
process, forgetting completely why the process exists in the first place.

~~~
stunthamsterio
One thing I generally insist on, is that the DevOps Engineers are held to the
same standard as the Dev's when it comes to version control, testing and
documentation. After all - that goes hand in hand with the 'Dev' part of the
title.

------
parasubvert
Devops is a professional movement to codify knowledge about how web companies
operate at scale and speed. It gets misinterpreted sometimes (more lately as
it becomes a fad). Doesn't mean its bullshit. Same problems with Agile, Lean,
Six Sigma, or any other codification of how humans could or should work
together to deliver a product.

Many of the lessons learned here aren't exactly universal, for example some of
the better Devops cultures (e.g Google) do have separate dev & ops teams, they
just have good collaboration.

------
waprin
I think DevOps is a bit of a fad anyway, because a huge chunk of the problems
people are solving will soon be solved by off-the-shelf standard cloud
solutions. It makes no sense for everyone to be writing their own
deployment/configuration Chef and Docker setups when they can just use
increasingly powerful and flexible PaaS options like AppEngine/Heroku.

------
serve_yay
This piece is so confounding about what Devops is and isn't, so much that I'm
ready to just flip the bozo bit on anyone who uses the term credulously. This
is silly.

~~~
jjmojojjmojo
Now you know how I feel.

------
TylerH
tl;dr because waking up in the middle of the night to debug someone else's app
really, really sucks.

------
adamconroy
A smart, meticulous, better than other people person like myself can't get
past the hubris, sorry.

------
DyslexicAtheist
why devops isn't bullshit: [http://blog.valbonne-consulting.com/2015/03/10/is-
devops-bul...](http://blog.valbonne-consulting.com/2015/03/10/is-devops-
bullshit/)

------
hydrogen18
Once again, we have the stylish new white background with barely legible text
design pattern showing on HN.

~~~
afandian
Agreed, the font is very light weight. The way it renders on my screen
(Macbook Air) gives me a headache.

~~~
agarcia-deniz
I'm glad it's not just me. looks like he changed it to something else

------
dschiptsov
It seems like a mere attempt to adapt labor-division best practices from
assembly plant industry. More managers, bigger budgets, easy staff rotation -
that kind of crap. Also a fresh new meme to ride on. Nothing special.

------
korzun
I will sum this up:

> We had edicts to make things happen, under the assumption that if we built
> it, the developers would embrace it.

Followed by:

> What matters is good people working together without artificial boundaries.

Former will never produce latter. Always work with engineering before building
anything that impacts their day to day.

