Hacker News new | past | comments | ask | show | jobs | submit login
Why One Programmer Doesn’t Do DevOps Anymore (lionfacelemonface.wordpress.com)
269 points by jjmojojjmojo on March 9, 2015 | hide | past | favorite | 163 comments

Sys Admin/Engineer here. We had it right 15 years ago. You couldn't become an admin unless you started as a programmer. You needed a deep understanding of your OS and good knowledge of C. At that time the devs ("programmers") were less knowledgeable than sys admins. You had to have to skills to analysis, debug and fix their code. When sys admins stopped coding the result was weaker admins. Sys admins ought to code within their domain, engineer, and maintain their systems for performance and high availability. It is long over due that sys admins return to their roots and get ahead of the pack.

I'm part of the recent sysadmin generation that had no programming skills and grew up doing what I consider more "application administration." Lots of clicking Next with no real understanding of what to do if that Next button quits working. Stack tracing? Procmon? Miserable understanding of DNS and networking, what's a /24? These were things senior admins knew and used once the tickets went up the chain because customers were screaming.

I was always (well, not always, once I'd proven myself) a senior admin/engineer, now I focus almost purely on AWS automation, and it bums me at out how many admins are completely happy with just being Next button admins who are absolutely AFRAID of touching a bash or zsh shell, or even a Cisco router without the UI.

I suppose it shouldn't bother me, it's done great for my career.

Are you talking about sysadmins on Windows? I can't imagine a Linux sysadmin being afraid of bash, zsh or a ios shell. My impression is that knowledge of some combination of bash, perl and python scripting is a basic job requirement.

the root of the problem (the idea that sysadmins can't program) might come from the fact that every company whether furniture/clothing store or any other sizable non-tech business needs sysadmins. But only if technology is your core business you need good developers. This probably leads to generalization of sysadmins not knowing their bits/bytes. Sure guys working for ISP's and in datacenters are a different breed but they still get thrown into the same category because of the job-title.

I think the problem is, some companies are expecting a sysadmin to be the "IT guy". The one who is only expected to call contractors to fix that recurring network problem, and install enough Windows boxes for the new hires.

I've worked for several MSPs and yeah, I mostly mean Windows. But we worked on just about everything. Especially Cisco/Dell CLI equipment (routers, switches, etc).

If a task involves "clicking next", then that task should not require any specialized knowledge to do. I totally agree that all developers should be willing do write code to automate "DevOps" tasks whenever they see fit. I know a lot of programmers who do this. These programmers always have a bunch of scripts sitting in their ~/bin, and you hardly ever see them clicking. I am a "programmer", but I love "DevOps" stuff, and mostly recently I wrote a large automation framework that got CI team interested. Tons of energy and time is wasted everyday where teams don't understand (and don't want to understand) each other's code.

> If a task involves "clicking next", then that task should not require any specialized knowledge to do

I would disagree with this point. If you're installing anything on a production box you really should know what that installer is going to do to your environment. Have you walked through the MS SQL Server installer wizard? You absolutely do need specialised knowledge if you're running that on a client's server.

SQL Server is probably one of the easiest "big" software systems available in the world. Without having much specialized knowledge you can setup a powerful HA cluster. It's mostly clicking next and following some basic instructions. Out of the box, it'll perform pretty well (just remember to click the "Optimize for Ad Hoc Workloads" checkbox in the properties page).

SQL Server requires vastly less "specialized knowledge" than, say, Postgres. I would be surprised to find that most SQL Server installs come with someone with really in-depth knowledge.

The same applies to a lot of other things MS ships, like setting up AD. Sure, knowing what you're doing helps, but MS sorta built an empire on making a lot of that stuff pretty trivial to do. For another example, look at setting up IPSec. A cute little UI you click on, versus well, the process on Linux.

Sorry no, it's an easy clicky pointy path to hell when you discover one month later no-one understood what the file path options meant (and there can be many when installing Standard+) and then the system volume ran out of space.

Or what accounts to run SQL's various services under and wonder why stuff doesn't work or their SQL box gets hijacked ("oh I'll just run this under the SYSTEM account").

The install wizard for SQL Server 2008R2 Standard Edition has around 20 pages (counting tabs on a page) of user selectable options and settings. You certainly do need to know what you're doing and understand the implications of each every one of those settings.

Sure if you're just doing a quicky install of the Dev Edition on your workstation you can afford to be a bit lax and click on through, but on a mission critical production box you need to RTFM or let a SQL admin who knows what they're doing perform the install.

I've been called in too many times to unf*ck a SQL server because a customer has declined one hour of installation support because their devs thought they knew better.... "oh yeah it's just clicky pointy install, why do I need help with that". Trust me GUI installers do not equate to "easy install" if you have no clue what the installer is asking you.

I'm a SQL (MS and MySQL) admin, amongst many other roles, for a webhoster with 19 years MS SQL Server experience (started on SQL 6.5 in '97), been there seen it all.

I mean, if you consider knowing that databases require disk space to be "specialized knowledge" then, I mean, I guess so. Otherwise the installer is rather straightforward in telling you what the paths are for. And creating a new DB also shows you which files will be used. There may be 20 pages, but most of that is ... pretty straightforward, easy stuff.

I've run SQL Server in life-critical production systems (911 call routing) and found it to be very straightforward. Just make sure the alarms are turned on and enabled for replication.

I've run it in HA clusters, both with a shared DAS and with the new shared-nothing system. A walk in the park. The only difficult thing I've personally run into with SQL Server was figuring out the Broker Service stuff and dealing with replication outside of a domain. And that was probably mostly networking configuration issues - the config wizards are pretty amazing.

I'm not implying that there is no specialized knowledge, and many SQL Server installs are probably suboptimal. But it's far from necessary to know much to go very far with SQL Server.

Compare to Postgres, where doing anything correctly requires mucking around with the config file, and, last time I used it, logging into the shell with a special user. And HA, well, good luck.

Sounds like you both came to the conclusion that to do things it's best if you pay attention and care.

I may be the only one around though that doesn't find config files to be 'mucking' or a form of mysticism that all should kneel before the user who can wade through them. That also, is not 'specialized knowledge'.

I managed to install Postgres from source, and have a standard setup. It wasn't so bad. Run the usual configure make, make install scripts, and go through a tutorial page for setting up the initial user. Ok, I have a fair bit of Linux experience but I am not a sysadmin.

> Stack tracing? Procmon? Miserable understanding of DNS and networking, what's a /24?

I'm primarily a developer, but I'd love to understand networking/low-level OS concepts like these in greater detail. Every time I start with a tutorial, I just get overwhelmed with how much I don't know — is there a book you'd recommend for a developer getting into devops (advanced networking, low-level OS architecture, configuring a machine from scratch, etc.)?

For networking, I think this is a great place to start: http://www.amazon.com/TCP-Illustrated-Protocols-Addison-Wesl...

For OS, I would recommend looking at Bach (on the design of UNIX) and Tannenbaum's books.

Thanks for the recommendations! Got 4 of them for less than $40 :)

Where do you even get a job being a Click Admin? I mean, I could use a hobby that pays about now...

The problem is that such a sysadmin is expensive, and what companies call "sysadmin" today is an almost completely different job.

Microsoft revolutionized personal computers because they made things easy and cheap. Their stuff was meant to be used by the average joe, not a greybeard who'd survived the Unix Wars.

At the same time, IT is always seen as a "cost center" to be reduced and outsourced. Unix? Too expensive, replace them with Windows, and if there's a problem, re-image the system.

The sysadmins you describe still exist, but they're called SREs by Google or who knows what elsewhere. Today's "sysadmin" is an outsourced phonecenter guy in Bangalore.

I still believe, a good or the best sysadmins are the ones who can code. A sysadmin should know the system inside out, including the software running over it.

As a sysadmin for the last 15 years, I'm fine with this, as long as you're going to pay me as a senior software engineer (if I have to know the app inside and out) who is also going to be on call constantly and doing infrastructure all day long.

If you as a business or employer want to continue to squeeze additional roles and knowledge into people, be prepared to pay for it.

Why would you want to get paid less? In my experience developers in Indy get about 80k. SysAdmins for IS and SQLServer get about 120k. They also don't have to work that long since the servers themselves are fairly robust.

> SysAdmins for IIS and SQLServer get about 120k. [My note: I presume you do mean IIS]

Presuming this is USD, but even in the UK as GBP - "In my dreams" - and I'm a developer as well as a sysadmin for Windows (IIS/SQL) and Linux (Apache/MySQL), 25 years man and boy but have always kept on top of my game :)

Outside of London you'd be lucky to hit 45K GBP (~70K US$) in a permy role with that skillset, perhaps 60-65K GBP if contracting.

The UK pays quite a bit less than the US. $120k /year for a good sys admin would be a deal in the states.

Midwest pays about 80 to 95K for a decent sysadmin (Unix/Linux w/ 10+ years of experience), only the specialized shops (trading firms, any place with a lot of money flowing through them) will go 120K or more.

If you know of any place north of Chicago that pays 120K or more for a 20-year Unix/Linux engineer with C systems level programming talent, and won't burn you out in 2 years like the trading firms downtown, I'd be interested.

A deal in San Francisco or NY perhaps, but not most of the country.

Of course you should get what you are worth and the position needs.

My experience has been people rarely want to pay appropriately when they're demanding you do several roles.

That was how I started a lot longer that 15 years ago - a common task would be oh we have just brought this expensive piece of kit (200% of my yearly salary) its its in that room hook it write any drivers then go and see Bob to work out how to digitize droplets in 3D

When I worked for a large Telco one of the reasons they employed me was to have an in house sysad for our Primes - we had the largest cluster in the UK running MR back in the early 80's

I rember one of my IBM mates at the time being horrified that we allowed to write all our own JCL

I've been at this for over 20 years and have zero programming experience. So, your blanket statement isn't true - most of the best admins I've worked with have had no programming experience but deep systems understanding.

I doubt you can fix today's devs' code, unless it's a very very junior dev. It's like saying a doctor should be able to fix a chemist's work.

I don't know if it was intentional, but this comes off as an incredibly arrogant comment.

Agreed And my Dr is fixing my biochemistry you should see the amount of pills I take a day to get it back into balance.

Hah, I wish that were the case. Over the last decade, popular programming languages are just catching up to what existed in the 70s.

Some do. They're called MD/PhDs. While they're somewhat rare, they do indeed exist.

DevOps as it has been implemented in tons of company is bs.

Its like the "Agile" push that companies thought they could get by buying a tool, and changing nothing else (release schedules, feature based release dates etc).

My favorite way of looking at it is simple - replace the word DevOps with empathy. That is the core of the idea, so lets try it - "Empathy Team", "Empathy Person", "Empathy Engineer".

DevOps is about dev and ops teams communicating and having empathy for the requirements of the other team.

> DevOps is about dev and ops teams communicating and having empathy for the requirements of the other team.

Most posters here don't seem to get it. Most replies appear to be the idea that DevOps means you fire your Ops people and the Developers do Ops previous job and their own Development job as well.

That's exactly what my previous company tried to do with their development team.

It caused the dev team to implode (everybody but one person on the 8-man team left within a year). The systems' stability and overall architecture was utterly terrible, servers would fall over because they would log so many things that they'd run out of disk space ("what's logrotate?" "What's a slave database?"), no metrics, no definition of success for a service, etc.

I suppose you could pull this "DevOps" thing off if you really wanted to, but you have to hire folks who can program and actually like sysadmin work. Otherwise you end up hiring developers to do everything, who are terrible at sysadmin work, and who get burnt out because they never get to actually program.

You're still trying to make developers do everything.

DevOps is not get rid of the Ops team. DevOps is not get rid of the dev team. It is get Ops and Dev to talk and understand each other. Ops should not be completely clueless as to how the developers workflow works. That doesn't mean they are a developer. Development should not be completely clueless about what happens during deployment but that doesn't mean they're on call to handle OS issues.

DevOps is these two teams are working for the same company and same goal, so damn well talk to each other and act like it.

Hint: In a devops work flow, if you were large enough to need a separate ops team, you still have a separate ops team.

Or, in the small company case, you never hire any dedicated ops people to begin with. You have developers fighting fires all day, getting interrupted constantly, causing them to hate their jobs.

Small company. We don't have many problems. We have a couple pain points but we fix them over time. When hosts die, it generally doesn't mean anything. There's redundancy in all things. I don't find myself interrupted often.

Previous job had a "Systems" team and a "Database" team. Both were just teams of obstructionists.

"Oh you need something today? Well, if you get a jira ticket made I could look into getting the requirements written up sometime in the next 2 weeks, to get the work assigned out next month."

Not saying that's the norm, but damn I hope I never experience that again.

If you were the guy getting hit from 20 different people, each with their own emergency, has to be done now, $4 million deal counts on it BS task, I wager you'd be singing a different tune. Oh and come to find out, that $4 million dollar deal? It had fallen through the previous month only no one had communicated it to the person makeing the request.

My favorite was the marketing manager who thought an email, with a full size image attachment, and note saying 'this needs to go on the marketing site' was sufficient requirements.

Requirements and prioritization are not a bad thing. Excessive bureaucracy is.

  Both were just teams of obstructionists.
  "Oh you need something today?
    ... get a jira ticket made 
    ... work assigned out next month."
It is possible that you might have a different opinion of the situation if viewed from their perspective. It's likely not deliberate obstructionism, but rather them trying to triage the requests so that they can be most effective across all customers. (It's possible that your ops team did not adequately communicate this to you.)

In my current job, I am the sole developer responsible for maintaining a certain set of our codebase. I have multiple internal customers, each of whom have separate (external) customer-facing needs. I have enough tickets created for three people working full time, so I am constantly having to triage which are the most important.

Part of the solution to this was to have a proposed plan for when to work on things -- I can then say, "Thank you for this well thought out ticket. I will be able to work on that in June." The most important thing is that my users' opinion about the criticality of things does not always match the overall picture, and I am always having to choose which tickets to postpone.

This is a good point, but it's a sign of dysfunction. I've seen the case when 50% of the issues are "P1 highest priority" (and most of them forgotten in a week or two). People making requests need to get some idea about what development effort (and maintenance!) costs. I don't know a good way to do this, except to be in a very small company.

Tickets at least require things to be documented, which is the biggest problem with ad hoc devops.

Sounds good. Does your company hire remote? :)

As an outsider i feel that the whole devops thing has come from a few web people spinning up some slapdash service using AWS etc and call it a day. Then something breaks and it is firefighting from that day onwards.

I wonder if one can draw parallels to the BYOD thing.

In both senses it seems to boil down to someone bypassing the IT admins because their manager wants to see results ASAP, and IT is going "sorry, budget and/or regulations says we can't at this time".

I actually look at from another perspective:

It seems to have stemmed from Ops teams saying "What can we do let dev teams build what they want, but in a way we can support it"

e.g. - a system needs an entry in the /etc/hosts file (a bad example I know, but a nice simple one for the basic use case)

old method: Dev files a Jira ticket, and the Ops team make it part of their golden images for production.

new method: Dev makes a change in the Config Management DB. Gets reviewed by a member of Ops, who either approves, or makes a suggestion to change it. Change gets tested against staging, then merged, and the system gets updated.

Ops know what the system needs, Dev knows how long it will take, and has an understanding of how it is deployed in real life.

Actual new method: dev logs in to production server, changes something live as part of a emergency / hot fix. It never gets documented anywhere.

I wouldn't go that far. Its documented on the system when it gets hot imaged and the image is never rebuilt from the config mgmt tool again /s

so not code as infrastructure and therefore precisely not devops

yes, but that adhoc activity is what many (incorrectly) perceive as "devops", so still relevant.

My view is slightly different. I see the rise of DevOps as coming from realizations about how modern applications are structured and how that's incompatible with the way operations used to be done. Think standard n-tier architecture vs a micro-services architecture that auto-scales depending on load.

Applications used to run on a fairly static number of instances that were maintained by an operations team. When more capacity was needed, operations would provision new machines. But now, especially with the increasing shift toward the cloud, infrastructure is seen more as something that's created programmatically, or even in response to events (load or otherwise) in the network. It's a powerful conceptual shift to think of infrastructure as something that isn't physical, but is more ephemeral. Just like a process on a server may die, a server on a cloud hosting platform may die. But you can engineer for failure at the infrastructure level the same way that you'd engineer for failure at the system level.

With this change, you can no longer rely on a fleet of sysadmins to manage everything for you. Instead, you rely on scripting everything..."Infrastructure as Code." You create recipes to create each and every part of your infrastructure and tools to orchestrate everything. Gone are the days where you actually ssh to boxes in a terminal...there are far too many boxes to make this practical. Instead, ssh becomes the protocol that your orchestration uses to communicate with its fleet of machines. Things are much more dynamic and you start to view your infrastructure more holistically rather than as a collection of individual machines.

Once you've started to think this way, you realize that neither the traditional development mindset or the traditional operations mindset work in isolation. You need to meld the two together so that operations understands better how to automate everything and development understands more about the environment in which their code will run.

What you describe is coping with situations of high infrastructure complexity and unreliability (many servers, moving and failed servers, etc.) by adopting advanced scripting and more abstract and sophisticated administration tools. It is an interesting technology trend, but I fail to see any significant social and organizational aspects beyond demanding everyone to be more competent.

Yeah I can see an argument for having development engineers and production engineers so that the devs writing new features can focus on that without getting distracted by production issues. The production engineers do monitoring, config management, and bug fixes for prod issues.

The problem is in most orgs, the roles for people who fight fires in production tend to be demoted to a lowly "support" title, and paid/hired for accordingly, which is really not commensurate with the value they provide to the organization.

I had several reasons for leaving my last company, but this was one of them. We had no QA, no IT, no ops. We just had devs doing everything. It might not even have been that bad if we were fairly compensated, but we were all making 3/4 market value in terms of pure salary, no insurance, and no equity to make up for it.

We were understaffed, underfunded, and underpaid.

I was in that situation once, in a previous job. It sucked, so instead of fighting fires all the time, we fixed it. We sought out and exterminated bugs, we set up better monitoring, we wrote automated fixes for the things that could be automatically fixed, and so on. Pretty soon things quieted down for good.

> Most posters here don't seem to get it. Most replies appear to be the idea that DevOps means you fire your Ops people and the Developers do Ops previous job and their own Development job as well.

Exactly, you have to fire the developers too.

yeah, CTO should be doing devops...

> DevOps is about dev and ops teams communicating and having empathy for the requirements of the other team.

DevOps is about not having separate dev and ops teams (or QA for that matter). It's about having one team that coordinates on implementing user stories and comes up with the right solutions. Some members of that team have certain skill sets - i.e. some are more ops focused and some are dev focused - but it's one team. It's about creating personal accountability; there's no more blindly assigning tickets to an offshore ops team and hoping it gets done right. If you're doing DevOps right, you developers and your operations people are on the same team, sit in the same area and attend the same scrums.

DevOps means that instead of saying "Ok, my work is done, but the deployment scripts need to be updated so I'll create a ticket with the ops team and they'll get it done next week", you can say "Ok, my work is done, but the deployment scripts need to be updated and I don't know how to do that, so I'll assign it to Mike." The feedback can go the other way too - if an implementation of a feature is too cumbersome to deploy, the ops guys are in the scrum with the dev guys and can work on a better design together, up front.

One good solution I've seen to this is a hybrid approach. QA was their own team, but sent a representative to each other team. That meant QA engineers had two scrums each morning -- one for the team they were assigned to, and one for QA as a whole. It worked very well. QA had their own processes that other teams didn't need to worry about, but QA also understood the needs of all other teams.

Obligatory quote echoing your final sentence, from the seminal DevOps thought leader Jon Hendren: "DevOps is mostly just a way of reorganizing all your nerds and making them talk to each other more"

"replace the word DevOps with empathy"

This scheme also works with diversity, quality, best practices, and change management.

I seriously can't figure out what the hell you guys are talking about.

Yup - it does. I just seem to prefer the empathy quote for some reason :)

Empathy is the actual mechanism for how those other things (quality, etc) are achieved. Diversity helps empathy (represent the user and relate to them better).

Another way to extend the analogy is its a standard management technique when provided with something you don't want to do, set up an isolated team with no deliverables, no authority, and no direct reports. Ta Da we got us "quality" or "diversity" or "Dev Ops" and we didn't have to change anything.

That's close, but still a problem. If you put change responsibility formally on the shoulders of a team or individual you're throwing them under the bus. It needs to be approached as a cross-functional team initiative (which is reproduced to scale out). A DevOps team might work if they're in a leadership position, but how does it work on the ground?

As a "DevOps Engineer" (use to be called Sys Admin) I feel a lot of posters here are using the term wrong.

A lot of us old sys admins are now calling ourselves devops because times are changing. To me devops isn't about taking a developer who likes ops to save some money. Its about taking ops out of their walled garden and into the development process. As someone in ops, I'd much rather be part of the development process as I'm the one supporting the code in production.

If its a choice of message queues.. I'd rather have a vote to the one I know about supporting.

It isn't really about cost savings as some companies I've been in I'm one of the highest paid engineers on the team. Its more about bringing ops into the process.

In addition to that, I always thought DevOps meant operations people using development practices and tools to automate and simplify their lives? Does it have to be more complex than that?

If you take your definition as "ops people who code and automate", you get a really efficient ops team who can orchestrate several hundred or thousand servers.

You still then have a friction point between "dev" and "ops" relating to hand off. Ops could go full PaaS, but more than likely you'll end up with a "cloud" infrastructure where your dev team will have to learn ops, or the traditional model where stuff gets thrown over the fence. Good luck deploying that service because there was no capacity planning and it'll take 6-8 weeks to get hardware from the vendor, regardless of whether ops can rack it and build it in under a day.

It's far better to have an org overlap so that ops is involved in dev planning to help address roadblocks, assist in capacity planning and management, as well as help reduce friction in tasks.

But, the purists would reply, if you simply outsource all your hardware to AWS or another place, you don't ned that ops or capacity planning. Until you do. This weekend AWS ran out of i2.2xlarge instances in a particular AZ, for example.

Hate to tell ya, but at anything other than the 'prototyping stage', you'll probably need a dedicated OPs guy, AWS or not.

that's noops/movedops not devops

You are correct.

Before operations lived off tickets which just said what I did not how long it really took or if my estimates were off.

At my workplace, the DevOps title is basically our Build Engineer who babysits TFS, handles non-compiling code and code changes, & architects and maintains the CI process.

It may sound like that's Dev's job. But they're busy maintaining and detangling the codebase, and handling daily code deploys. Meanwhile Ops acts as a liaison with our hosting company, installs 3rd party software, performs some break/fix work in the environment, and does corporate IT work.

It used to seem crazy, and it probably looks crazy from the outside, but it's still working.

From one of the comments:

> It’s easier to train a developer to do sane operating procedures than to teach a sysadmin the developer mindset (testing, revision control, breaking down to logical units and similar).

Having done both jobs, I don't really think that's true, unless by 'sysadmin' one means the type of low-skill position which resets passwords. A system administrator is responsible for administering a system of hosts and software; his should be a more complex job than a developers. He should be responsible for test systems as well as for production systems; his scripts should live in source control. I'll grant that functional decomposition may not be as vitally important to his work as it is to a normal developers—but then, his scripts will inevitably grow to be pretty complex, so maybe it will be after all.

That doesn't mean that developers are stupid either. Sure, at BigEnterpriseCorp a developer is just a seat-warmer who implements directives like 'add a method to the Foo class which takes two arguments, adds them and returns the result,' but a real developer is a student (whether formally taught or informally self-taught) of computer science; he understands and can choose & implement both algorithms & data structures in order to solve a problem.

A good DevOps hire is going to have some skill in both areas, not as a weak sysadmin and a weak dev, but as a strong sysadmin and a strong dev. A good DevOps team is going to be more skilled (and expensive) than the equivalent dev and ops teams, but it will also be more productive and less error-prone.

If you're trying to build a devops team from Initech, Initron and Initrol low-skill employees, you're probably going to fail. If you're building a devops team to reduce costs, you're probably going to fail. But if you're building a devops team to increase profitability and competitiveness, you may succeed.

In the Unix/Linux world, a lot of the Sys Admins I've met have been very strong technically. Knew at least a scripting language and basic C programming. The trend however has been for less and less coding skills and more superficial. Sometimes not even any shell scripting skills.

In the Windows world, a lot of the Sys Admins do not have those skills. It's changing, things like PowerShell and various others are helping, but a lot of Windows Admin are just not that comfortable away from the GUI. I think the comment you refer to is by someone with a Windows dev background. GUIs don't version...

The mainframe world folks tended to be very silo'd (and IBM engineers were never far to hold hands.)

So many misconceptions here as to what DevOps is. I think mugsie comes closest with his use of the term "empathy". This is very very close, but I would add one more term "cooperation".

DevOps, when implemented correctly can have enormous, order of magnitude increases in both productivity and quality, but it is not voodoo, despite this almost unbelievable claim. It involves empathy and cooperation and it requires this because in order to implement DevOps in a meaningful way, an organization has to vastly increase communication between "silos". It has to formalize, document and automate as many processes as possible.

DevOps is NOT easy to implement and involves a cultural shift and absolute management buy-in. It is best implemented in small projects, where clear wins can be demonstrated. Use this success to bootstrap the process to bigger projects.

I do agree with the OP that if you have a DevOps team, you are probably not doing DevOps. The whole point of DevOps is to get rid of silo mentality. No more us and them, but all us.

Implemented correctly, everyone is DevOps.

It's fascinating how much this engineers experience shares with mine. We both appear to have similar approaches to working with technical systems, that has led to skillsets that are more broad than deep. We've both had interview problems because of this, and we've both ended up in DevOps.

I've also had more or less the exact same experience with DevOps, though with (hopefully) a happier ending. I was on a team working for a client that desperately needed "DevOps-type" things. So I built them, and their infrastructure team ignored them.

Eventually we stopped working with that client, and I was moved to a different organization, where each development team was doing it's own thing, operations-wise, and had no interest in adopting a uniform approach, so my (short) efforts there were also for naught.

But then I had the opportunity to come back to the first client, as their DevOps Manager. Before I took the job, I talked to their current development organization about what their pain points were, and what actions they would be willing to take to alleviate them, and I built a strategy around that. A key part of this strategy was gradually eliminating the need for a DevOps team, as I firmly believe the best application of "DevOps" is as a joint project between Development and Operations to optimize shared processes. Once the friction is minimal, you don't need a team there anymore. So we have a developer embedded in our meetings, and have more or less full control of any infrastructure.

I think there is a fundamental disconnect between what DevOps is and the classical business organization structure.

DevOps is not a department and it will not work if it is a department DevOps is a way of doing things to achieve maximum efficiency and everyone on development and operations must embrace it for it to work.

If either development or operations resist the change then either the resisting members should be eliminated or the switch over to the DevOps way aborted.

DevOps doesn't work without cooperation and effort from all team members.

So in essence, devops treat every projects as if it was a startup? That is, to hell with departments, managers and lines of communications, one project, one team, one office?

Not quite, Managers can be separate, accounts can be separate, advertizing can be separate etc. but Development and Operations must be treated as one unit no separation.

In other words anything that can have a line of communication and still work can stay as it is but in order for DevOps to work development and operations must essentially become one with both developers and sysadmins working together on architecting a flow that works for both of them.

One office and one project is not really a requirement but it helps a lot in the beginning however if you can keep the cooperation a culture alive even working remotely could work.

The key is people who are always willing to learn and try new things and putting effort into change and staying on the same page with their teammates.

So who gets the production system and database root passwords?

So this is the thing. There are three things that really go against a true "devop" system:

1) cost.

Ops are cheaper than devops, sometimes twice as much. Yes yes, flexibility. But thats a misnomer. If your dev teams are doing donkey sysadmin work, because they don't have the experience, or are up all hours fixing things, you're going to endup with a high staff turn over.

2) division of labour

As a company grows, they can employ more and more specialist staff. This gives them a competative edge as they are able to make nuanced decisions based on skill, or experience. A decent storage admin will be able to build a storage system that will scale relaibly, cheaply and with less downtime than someone who is not a specialist.

3) burn out.

A human is not designed to multitask 14 hours a day. Coding, testing, supporting, infrastructure procuring, for anything other than the smallest company is too much for a single role.

There are great things that come from a devop "culture"; greater personal autonomy. However like all things, its not one size fits all. There is a great deal of arrogance where people assume that a sysadmin never automate anything. or that they can't program.

They can and do, you know why? because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.

I was with you until "because they want to carry on searching ebay, scamming $someone boasting about network speed, or just plain getting shitfaced.", but I'm guessing you're just being cheeky.

I was thinking it was a BOFH ("Bastard Operator From Hell") reference. [0] The BOFH is a fictional sysadmin who is worse than any you have ever had, in entertaining ways. He basically does exactly the things the OP mentioned, while dealing with clueless management, undeliable vendors, and a Sith Apprentice^W^WYoung Coworker who learned well, and is constantly trying to stab him in the back. He's like a mix between Catbert and Wally. ;)

0: http://www.theregister.co.uk/2015/02/27/bofh_2015_episode_2/

I was bending the truth a tinybit.

The idea is that a good sysadmin is an idle sysadmin. (having offloaded tech support to the juniors)

unless they are playing with new stuff.

I am a big believer in DevOps to the point where I often consult to "implement DevOps" at organizations. The frustrating part if that organizations seem to want to create "DevOps Teams" to augment their other teams. An organization cannot have a "Development Team", an "Operations Team", and then a "DevOps Team". DevOps is a set of good principles which help to guide the process of making, releasing, and maintaining good software products. These are by no means the only set of principles and organization should live by, but Developers and Operations should work to implement these principles because it will make their process more efficient. It is certainly not a matter of simply tossing in yet another department full of people and hoping that more cost efficient work gets completed.

When I work with clients the inevitable question comes up: "What Principles should we implement then?" and the answers are usually more questions really. How long does it take developers to set up their working environments and get to coding? How is code tested and prepared for deployment? Are stakeholders feeling secure about deploying this new code to production users? Can we deploy effectively every time without deployment related issues? When something bad happens in production, what is the recovery procedure and how long does it take?

The answers to these questions and many others often show that there are a few issues. First, developers and operations aren't speaking to one another enough and aren't really working together. Second, there is very little trust in the current environment and that lack of trust is made up for by adding more human intervention. Third, there is very little automation and this causes the lack of trust but few people what to change their habits to fix it. And finally, there is such a great divid between engineering and operations that the pains felt by one group and not shared by the other.

DevOps has the principles needed to start to solve these problems. However, we don't need a department to implement those principles. We have enough departments.

Well said. I think this is what the author of the OP was also getting at - it's better to grow the culture (while sharing responsibility) where it belongs rather than purchasing it wholesale and shoving it down dev throats.

As an employee of a large software focused corporation that does many of these things right, I had no idea how bad things can get for smaller, newer companies that haven't figured all of this out.

Even small companies can have momentum and unanalyzed patterns that are hard to change.

Replace DevOps with QA and an equally valid article.

Dedicated QA teams should be avoided unless truly necessary. Otherwise, you have the same problem of developers throwing over under-tested code to a group of separate people that are smart but treated poorly.

A QA guy who loves to write automation gets the same promises... "oh you'll be programming mostly; manually test maybe once a month". But when they get into the door, it's the exact opposite.

Developers should all be passionate about QA (read as: the quality of their product), as well as how their product is actually deployed (read as: executed in the most important of environments--production!).

Last month I went to the UK PHP Conference where Jeremy Quinton spoke about DevOps. The biggest take-away I got was:

'Do not set out to create a DevOps team or employ people for that role'

My interpretation was the role simply doesn't exist. If your company has a team of developers and a team of people who keep the wheels turning - in an ideal world you already have a DevOps team as long as they collaborate. If you also employ smart people you will get 'developers that know how to do sysadmn' and 'sysadmins that know how to develop' - definitely this indeed should be embraced but not expected.

Obligatory lusis quote and link...

"Devops means giving a shit about your job enough to not pass the buck. Devops means giving a shit about your job enough to want to learn all the parts and not just your little world.

Developers need to understand infrastructure. Operations people need to understand code. People need to fucking work with each other and not just occupy space next to each other."


>You can’t have DevOps and still have separate operations and development teams. Period.

I cannot disagree more. You can't have developers of your product knee deep in DevOps and be effective (at a certain size, smaller teams with less demanding operations needs can).

DevOps should have very clear demarcation of responsibility in an organization. What the OP describes is an blending of responsibilities and that is the root of his problems.

DevOps /is/ operations. It is just that simple. It is design and automation of support systems. Nothing more, nothing less.

If 'DevOps' is operations, that what is the point of the term? Ops teams have been doing automation of support systems for a while, so what exactly does it mean to be DevOps? I think that's part of what the OP is getting at, if your fancy 'DevOps' team just ends up being separated and isolated from the dev team you're back at square one, you've accomplished nothing and the same divide exists with a new label.

As someone who works on the dev side, the ops side, and the mixture of the two I feel the OPs pain acutely and have had mixed experiences. People on the dev side need to understand and willing to learn about ops, it's not rocket science and you don't have to be knee deep in it to get the basics down. When ops becomes this black box and becomes a bottleneck is when you start to see dysfunction.

>If 'DevOps' is operations, that what is the point of the term?

It is taking more of a gestalt systems approach to the entire support structure. This means more intelligent support programs and treating it like designing and delivering a software project (or, more accurately a series of subprojects) as opposed to a loose amalgam of scripts.

This also requires the product development team to be keenly aware of that support architecture and how their decisions factor into it especially in terms of testing, deployment, and module interactivity.

> Ops teams have been doing automation of support systems for a while, so what exactly does it mean to be DevOps?

DevOps has been around for a while -- way before there was a word for it. I think DevOps has come into its own, recently, though because platforms have become hugely more flexible while at the same time a consensus built on what it means to manage those systems. The difference, I think, is whether things are built to remain somewhat static or if change is designed for. Are development ideas like encapsulation and decoupling etc designed into the system? Smart sysadmins have been doing this stuff for a long time but now organizations are taking the approach into account all the way up and down the chain. And, yes, that means that product developers need to be fully aware of operational needs but, in my opinion, I don't think that means they build the operational tools themselves.

>your fancy 'DevOps' team just ends up being separated and isolated from the dev team you're back at square one

Demarcation of responsibility doesn't mean demarcation of information. The product development team will need to be intimately aware of what is happening in operations and vice versa. In a DevOps implementing environment it is unavoidable.

> the dev side need to understand and willing to learn about ops, it's not rocket science

These days it's getting pretty close. I don't think it is fair to product developers to say "Listen, we like the idea of what Docker can do for us so you are going to learn it inside and out." Somewhere you have to say "Hey, we probably want to implement Docker. Ops, dig in. You know how our product is currently structured and how operations currently works with that. Try to figure out what needs to change and what doesn't. X and Y from the dev team will be made available for deeper discussions and testing as needed. When the overall gist of what has to change is known, we'll all get together, ops and produce dev, to get everyone on the same page." It's a bleeding of information, not a full on 'shake it all up' scenario.

Neither side can be a black box anymore if the current state of software development and delivery is going to be taken advantage of. But neither can everyone know and do everything or context switch as needed.

Keep in mind: I am not a DevOps theorist -- maybe I have it completely wrong according to the people that came up with it but this is my practical understanding of the movement.

> DevOps /is/ operations. It is just that simple. It is design and automation of support systems. Nothing more, nothing less.

I disagree; although there's a huge value to consciously automating and designing support/deployment systems, there's even more value to developing products to be deployed ab initio.

You need automated production systems, and you also need developers who understand how their code is deployed and run. Really, you need smart, productive people, not interchangeable Java monkeys.

> DevOps /is/ operations. It is just that simple. It is design and automation of support systems. Nothing more, nothing less.

Design and automation of any systems is "development". It is just that simple. Nothing more, nothing less.

So, from your description "DevOps" is really development whose application domain is operations.

(Though I'm not sure I agree with your description -- I think that's part of DevOps but that DevOps is somewhat broader; in the same way that many approaches that fall under the general agile/lean umbrella have pushed erasing role-based team boundaries and, even within the lower level teams, role distinctions within the broader development realm -- e.g., between programmers, analysts, testers, etc. -- DevOps is about erasing, or at least blurring somewhat, the team and role boundaries between development and operations; this doesn't mean that people don't have distinct competencies, but it means that people with those competencies work together at lower organizational levels and in a more tightly integrated way focused on overall mission, rather than throwing things over the wall at each other.)

Well, no. DevOps is (or should be) as the OP describes it. Just because you disagree with the concept doesn't make the definition wrong.

I mostly but not completely agree with your opinion. But, put it into context, there are more than one "right" definition of devops, just like there's more than one "right" definition of "Cloud" or "Cloud Computing."

I think the author's story is more an admonition of blind foolishness and deceit than it is an indictment of the automation.

hmm I've worked with some of the people who invented the term and this is a bit off. devops is bringing dev practises into ops as you describe - e.g. replacing ad-hoc bespoke shell scripts with more standardised automation frameworks leading to full infrastructure-as-code phoenix-pattern deployments. It is also breaking the dev and ops wall in the same way that agile is intended to break the bus/dev wall. An example of this would be continuous deployment and automated testing (e.g. the kind of things that IMVU did with automated performance testing on commit to live with rollback if problems started to appear as changes cascaded through the infrastructure). and like agile it is meant to describe a mindset rather than a role or department. and also like agile, it is horribly bent out of shape by most orgs and comes to mean different things to many people...

Former embedded developer, former systems developer, former language developer, former application developer, former senior systems administrator and now DBA. (30 years) Simply put, the problem IS TITLES. They cause the "not my job" problem. If you've been called, help fix the problem. If you don't understand the problem, stay with it, maybe you'll learn something. Just because your a "title" doesn't mean your s--t don't stink. Unless your are designing your CPUs, writing the languages that your write your OS and the applications that run your hardware, you are riding on the back of someone else. Do what's ask of you and ask how you can make everyone's job better. Then you'll be employed and paid well for a long time.

I think the most striking point made here was the one about culture and how without everyone buying into the DevOp paradigm, its just another unused tool.

I say this because I've lived it...the last 3 (three!) companies I've contracted at were pushing dev code to live servers! Even after I setup svnserve and created the repos, no one was really interested in changing workflow. I choose svn because, lets face it, it has a much lower learning curve then git.

It wasn't until I absolutely refused to work on the live server that other the devs grudgingly started using SVN...I'm sure they are all back to FTPing to the live server now that I'm gone.

Without the proper culture from above, things just aren't going to change.

I'm devops and in my opinion the title infers this; my job is to keep the communication between developers and the operations teams in sync and copacetic. Usually, Ops doesn't know dev and dev doesn't know ops so if there's nobody to translate and explain things in each others technical language things can't be done quickly and reliably. The wrong environments will be configured, things won't have backups, it will be like the wild west and things will be stressful for both sides.

My job, just like it was when I was a sysadmin, is to keep things running smoothly. To optimize both sides of the coin and to take things off their plate that I myself can deal with that might be a blocker to them finishing tasks and to us getting features out or server migrations done.

Some devs know ops and some ops know dev, those guys, more often than not I just check in with and see that things are running smoothly. For the other folk I try to create systems and workflows that get blockers out of their way.

I try to be a useful and reliable tool to both sides as much as I can be.

Hopefully the way I've been doing this for the last two years since I moved into a more Devops related role has been helpful to everyone.

Completely OT: "copacetic", new one for me - where have you come across this word; the etymology is intriguing to me.


Boy that's interesting, I've never thought about it. I'm an American who grew up in Germany but have an Italian and German family who grew up in the projects of NY (the Love Canal, Niagara Falls). My grandmother (german) and grandfather (italian) married and their families constantly fought and pretty much hated one another.

I remember them fighting (mostly arguing) and the men going "Ey ey, lets keep things copacetic." (They might have said capasetti)

I always thought it was a pretty and a smart sounding word. I guess not.

>pretty and a smart sounding word //

Sounds good/smart to me! The wiktionary page mentions possible Italian-NY etymologies so you fit in there. Funnily enough I was going to try and be clever and guess you were a New Yorker [based on that] but noticed elsewhere you appear to be in TX.

Interesting, thanks for responding.

Army brat, I've been everywhere!

Great name for a startup: copacetic.io

I first heard this word when this song came out: http://www.metrolyrics.com/bound-for-the-floor-lyrics-local-...

I was recently looking for a new job, and from the job specs and interviews I have realised that 'DevOps' means very different things to many people. I am a system administrator who likes to automate things and write code, and also spent decades supporting developers and good Agile environments, etc. and production ops environments. Some companies think DevOps is a developer who knows how to deploy services on Ubuntu, and their interview tests the implementation of algorithms covered in recent CS courses. I removed any mention of DevOps from my CV, that made things less confusing for everyone.

It seems OP is using the word DevOps to describe what is traditionally referred to as "special projects". Essentially, the non-technical CIO hears a new buzzword and hires a team to implement it in an ivory tower without fully partnering with existing team members. From OO to SOA to Agile to DevOps and everything in between these initiatives almost always fail. That of course doesn't mean that OO is bad, or that DevOps is bad. It is just that the "special project" methodology is terrible.

To me DevOps is a cost cutting measure whereby an organization can save a few hundred thousand dollars by having one individual not only do the development but also handle the deployment and the testing.

In theory that will eliminate the need for an additional tester and a second individual to do the change control.

But those short term gains expose the organization to the massive risks that the one and only one individual will make a mistake, with no second individual available to do a double check.

The reality is people make mistakes and by eliminating people from that double checking process may well save money in the short term, but in the long run it greatly adds to the of risk failure.

While there are short term gains, the risks to the organization can be massive:


That is actually an example of the old style of IT.

The person deploying the software had no idea how it worked on an internal level.

The person who wrote the software had no idea how it was actually deployed in production.

They didn't have a quick rollback in place (this is the exact reason you have quick rollback ability)

The deploy was done by hand, with a user typing. That is probably the biggest risk you can have in a production system. It means that fully repeatable builds in test & pre production are impossible.

I am not saying "DevOps" would fix this, but a bit of empathy / comunications between the dev team and ops team might have averted it.


Well, look, I already told you. I deal with the goddamn customers so the engineers don't have to!! I have people skills!! I am good at dealing with people!!! Can't you understand that?!? WHAT THE HELL IS WRONG WITH YOU PEOPLE?!!!!!!!

I have been referred to as a devops person since I have been developing distributed systems since 1998 and liked knowing and working the full stack, including hardware.

My definition of devops: small teams totally own their project, iterating on design, implementation, deployment. I have no experience with one giant devops team for an entire company.

It seems that the main problem here was not DevOps but rather the culture in which he was asked to do his work. Creating a devops team whether "by fiat or by force" doesn't sound like a recipe for a good experience.

My experience is different: DevOps saves me many headaches by letting me easily deploy things for small clients. A larger company I work with uses Chef to manage their 40+ servers; the IT department doesn't touch these servers. Rather, 3 or 4 developers share in the work of writing cookbooks, adding instrumentation, deploying code and maintaining the servers. It would be handy to have a full-time guy as well (OP mentioned the amount of work needed to keep up with just your basic security patches) but we -- as developers -- love the access we have to our entire stack.

I don't want to take away from your point, but you touched on security - I mentioned it because it was a constant pain point for both myself and for our teams. Like operations, it's something you can be good at on a superficial level, and do OK for most things, but really getting it and being good at it takes experience and time.

I don't think it came through in my post, but having a security expert (or any specialist) in the mix can be a huge help in making the product better - I know this from personal experience - you just have to make sure they're integrated, challenged, and respected in the group.

Don't know if the concept of DevOps is bs but I know that it's unfortunate if a developer never did system administration: there are many of this "git push heroku" kind of devs who pee in their pants if they have to deal with bare metal and the production servers. It's not that they wouldn't be able too handle them, it's that they are afraid because they never did and besides, good system administration knowledge leads always to better application and system design.

I blame Java much more than heroku. Corporate drone Java developers have a much greater tendency to only write code (too much code!) and not be interested in learning anything beyond the confines of their comfy auto-completing IDE. It seems to me Java developers will howl "not my job!" the loudest whenever anyone suggests that maybe they can mange their own environments.

In the enterprise Java world, the comfy auto-completing IDE is a smart strategic choice when "managing the environment" means weeks of frustration at installing, configuring and fitting together by trial and error an application server like IBM Websphere and the application. Funny tools like Maven, Subversion and certain Eclipse plugins add extra puzzles and unpredictability for the true amateur, but the normal unforeseen complications are usually enough to forfeit any hope of good automation.

Sounds like what has been around since the mainframe days and batch computing. Or maybe one could liken it to Star Trek, with ops being the people keeping the warp core going even when faced with borgs and black holes?

I'm a dev who learned everything by experience. I'm in my 20's and have been doing this for ~20 years, so I have a bit of experience with both the new and the old world.

My take is that the code quality of whats allowed to be in production has gotten worse, that's it. It's an effect of an abundance of RAM and virtualization. The abundance of ram lets you just trash it and do things in a horribly inefficient way. Sure the extra memory can help with time memory trade off if you have something that doesnt need to scale a great deal, but processing things inline will still be faster. Developers have forgotten how to handle streams. Usually forcing a dev to learn a highly functional language that's immutable can fix these issues (it must involve a large dataset, where you know they will tax out the memory, and then code review to help).

I mention virtualization because a common pattern I've seen is to write poor code, and just put more boxes behind the load balancer when things start failing because the technical debt is so high it's an impossible mountain to climb. This obviously isn't the right approach and will leave your devops guys worn out chasing your everyday distributed systems problems that happen because it wasn't built right for distribution and the dev has never read the fallacies of distributed computing.

And while you're fixing this technical debt a team of competent engineers builds a competing product and trumps you hard (second mover advantage).

Maybe people paid a bit more attention when things cost 10x as much as they do now and every answer (both bad and good, and you can't tell the difference) wasn't a stackoverflow search away.....

So you're in your late 20s and started working when you were in elementary school? :-)

Good point. Currently I am working on parsing a 5.5 million line STEP file. I've learned more about memory usage and high level overhead in the last week then I had in the previous 10 years of .Net dev.

Unfortunately the term DevOps has been diluted in the job market. Now most DevOps job postings are actually just plain ol' sys op roles, where "development" means writing tools for the actual developers. If you want to spend at least 50% of your time writing code then avoid any job postings that mention "ops".

I'm not surprised by his experience as it mirrors a bit of mine. This was previously discussed on HN: https://news.ycombinator.com/item?id=2674841

Although in his post, he says:

"Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow."

I would argue that cultural change has to also start from the top. You can't change the culture if the leaders of the company don't work with you on it.

The tendency of people at the bottom is to eventually start behaving like their managers as the managers project what is acceptable and not acceptable behavior. Trying to start change from the bottom up is a hard, hard thing to do, with a low probability of success if senior management isn't on board as well.

> I would argue that cultural change has to also start from the top.

This is where the "breathing room" comes in. I think that fostering an environment where people are free to organize themselves without edicts and policy produces overall better processes (and ultimately better quality).

I think it's OK for management to lay a framework, provide a particular kind of training, but it's best to provide goals that forward the company, step back and let the people closest to the work decide the best way to achieve them.

Whether you want to or not, it's your choice. However, the push towards DevOps as opposed to Dev is based on efficiency; it's easier to hire a developer that knows operations, instead of a developer and an operations employee. Even without DevOps, a lot of positions are Full Stack, so you need to know front and back-end development and technology. I don't think it's merely a trend, it's more or less a push towards saving money and time. If you don't want to be a full stack eng or a DevOps eng, then you're limiting yourself and you're limiting the opportunities you'll have. It sucks for those who want to limit themselves to one specific group of tasks instead of branching out, but it's just how it is.

If you want to make a devops omelet at a place that has an over the wall mentality, you have to break some eggs (and walls).

I was part of a successful transition to devops from a thoroughly silo'd operation, where the boss basically gave the dev team the responsibility and authority to make it happen. He was also able to snag some of the traditional ops team and make them part of the new team. After a period of storming, the new team came up with some innovative ways to get the job done. Once the team realized that they were it in terms of getting it done, it got done. Quality and job satisfaction improved, cost was reduced.

I've also seen various unsuccessful transitions. In every case, the responsibility was divided into various silos, which sandbagged everything.

DevOps may be hyped, but it's not outright bullshit. DevOps is just the realization that you would rather have a fewer people knowing everything, than many people knowing disjoint areas. The same force underpins the full stack crowd, or the notion that pizza size teams are more efficient. While this is not always possible in practice, in my experience it is better if the project is small enough that you can pull it off.

The memory bandwidth in one single person's brain is simply much higher than the bandwidth between people. To maximize performance you want to run your project on as few brains as possible.

How does one reconcile DevOps with the Sarbanes-Oxley act? http://en.wikipedia.org/wiki/Separation_of_duties

With difficulty. For example - you can still have an Ops team, and they have the required access. The developers access to production is done on a case by case basis, with some sort of tracking of who logged in. and why. Also, doing full pre - merge code review helps, as it is yet another set of eyes on the code that will be pushed to production...

In short - its not a "one solution fits all", and it is something that needs to be discussed with you SOX auditors, to find a working solution for your company

Thanks for the response! We keep a distinct separation of duties, but we try to keep both groups in constant contact to minimize animosity in both groups. We pick and choose some DevOps ideals to make it fit, was just curious about other people's experience.

SoD simply means your systems or processes require at least two different people to make any changes. If your entire tech team is DevOps then it simply means the dev who tests and deploys a given code change is different from the dev who wrote the code. These different roles can be per task, it doesn't have to be full-time persons assigned to full-time roles.

DevOps doesn't mean people doing multiple jobs. You can collaborate with separated roles and be productive. Furthermore you can build abstraction to simply remove manual, unrecorded interaction with systems to satisfy auditing and regulation.

It seems, from reading the article, like the need would be satisfied by having two dev-ops engineers manage something, rather than one developer and one ops guy.

Having been around since the inception of this space, when things were much more tools focused, to now, when it's a lot more about "culture" (all good), I feel 95% of DevOps talks are people discussing what DevOps is. As such, it's now a metasyntactical label for a lot of things that can by good in various scenarios. I agree it is time to retire the label and instead talk about those specific things -- automation, culture, continuous improvement, etc, in nearly all instances.

A senior systems administrator can and does write programs for automation, monitoring, configuration management, and all the little things that are not customer-facing applications.

I used to be able to use "senior systems administrator" as a short-hand for those skill sets, but about 2010 that stopped working.

Now I need to use "DevOps" as an imperfect substitute in the title; otherwise only junior people apply, even though the rest of the ad goes into the same depth about skills and abilities.

So that’s what DevOps really is. You take a bunch of really skilled, passionate, talented people who don’t have their heads shoved so far up their own asses that they can take the time to learn new things.

What you want to look out for is the formation of "walls." People have an instinct to do this. People will form cliques -- which effectively are invisible barriers to understanding and honest collaboration.

In a 5 person startup, this is easy. You have one group. Most of the critical information flows are inside this boundary, and those which aren't, the group is already intensely aware of. (Investors, customers.)

However, when headcount has started to grow, and when you are forming a separate group to take on a task that requires intense collaboration, maybe you're doing the wrong thing? Maybe what you want is a separate role for people who get embedded in a variety of different groups? What if DevOps people belonged to DevOps, but also, and primarily, belonged with a particular group of developers? Then, they could have firsthand experience of intense collaboration, but also meet once a week with all the other DevOps role holders and communicate to standardize and avoid duplicated effort?

A generalization of this: Make sure you're building bridges and not compartments.

Funny and, unfortunately, relevant, especially the last tweet: https://twitter.com/devops_borat

DevOps is not a job. It is marketing speak.

Sure, there's this general idea that development and operations teams should work together, or of a programmer also having some experience adminning systems or vice versa. But this is as much a job description as "non-domestic car mechanic" is a job description. Generalized ideas that don't say anything but an indication of the background expected of the person.

"DevOps" or not makes no difference; all corporations face the same problems with multiple segregated teams and people doing whatever they want in order to get their jobs done, as well as a sort of tunnel vision that prevents one from properly supporting your teammates or a product. To this end, there is only one thing that will make a significant difference in the work done, no matter what you call your team or position.

The best engineer in the world becomes a janitor under the wrong leadership or teamwork.

I think the right thing to do is to firstly get rid of the term because it's become so meaningless and abused in so many ways. At least "cloud" has some semblance of at least meaning "whatever that we used to do in the 90s for IT is BAD" and implies great upheaval, but DevOps is a really, really weak movement in comparison because most companies that are dysfunctional try to contain whatever DevOps is into their existing (typically broken, so let's propagate that error 100x, right?) practices.

While I agree that sysadmins should be developers first it is increasingly less likely you'll find developers willing to become computer janitors and deal with the ugly realities of operations in, for example, larger companies. I wouldn't think that John Carmack ever liked having to figure out the installer scripts and keep them working across all the supported platforms through all the BS that Microsoft and different Linux distributions could throw at you, yet that's what a more "ideal" DevOps organization described by many model would have John Carmack do.

What should have happened though is that instead of hobbling the John Carmacks and dragging down their productivity, you should have fired the developers that refuse to cooperate with operations and those that keep actively putting off writing "boring" code or never documenting things that do matter for operations to figure out and learn through a failure that something broke. That kind of behavior is the developer equivalent of not being a team player at all. It's true that none of these things actually make you a better programmer in itself, but these things are what contribute to making you more PROFESSIONAL instead of a cowboy renegade programmer that doesn't care if nobody else can use his code because "clearly, if you can't figure it out quickly you don't deserve to use it" (not an exaggeration at all I've heard).

Having been in both development and operations in rather large (and small!) companies now, I'd far, far, far rather stay as a hermit-crab developer than deal with the constant dysfunctions of enterprise Faustian nightmares that are required for you to deal with as an operations engineer. You can generally ignore a lot of the noise in a super-siloed off R&D organization, for example, but you cannot ignore your pager going off at 2 am because a random server crashed causing someone to get very, very angry and demand immediate resolution when it is your JOB to keep those things going. Hardly going to write anything really meaningful with that ruining your work week pretty much every week.

Is it just me or are development practices (sem ver, git flow, test coverage, in-house code guidelines, even vcs and CI) the kinds of things that go out the window when a DevOps team forms, because they're "too slow"? For example, who has time to ensure 100% code coverage for your script that's going to fix the problem with your load balancer?

The exact same thing I'm noticing is happening with the "automation" teams that are cropping up. It's like folks are discovering ways of getting around process, forgetting completely why the process exists in the first place.

One thing I generally insist on, is that the DevOps Engineers are held to the same standard as the Dev's when it comes to version control, testing and documentation. After all - that goes hand in hand with the 'Dev' part of the title.

Devops is a professional movement to codify knowledge about how web companies operate at scale and speed. It gets misinterpreted sometimes (more lately as it becomes a fad). Doesn't mean its bullshit. Same problems with Agile, Lean, Six Sigma, or any other codification of how humans could or should work together to deliver a product.

Many of the lessons learned here aren't exactly universal, for example some of the better Devops cultures (e.g Google) do have separate dev & ops teams, they just have good collaboration.

I think DevOps is a bit of a fad anyway, because a huge chunk of the problems people are solving will soon be solved by off-the-shelf standard cloud solutions. It makes no sense for everyone to be writing their own deployment/configuration Chef and Docker setups when they can just use increasingly powerful and flexible PaaS options like AppEngine/Heroku.

This piece is so confounding about what Devops is and isn't, so much that I'm ready to just flip the bozo bit on anyone who uses the term credulously. This is silly.

Now you know how I feel.

tl;dr because waking up in the middle of the night to debug someone else's app really, really sucks.

A smart, meticulous, better than other people person like myself can't get past the hubris, sorry.

Once again, we have the stylish new white background with barely legible text design pattern showing on HN.

I switched back to the "2010" theme, hopefully people will have a better reading experience now. Thanks for letting me know (letting me know in private without the snark would have been better, but beggars can't be choosers, so I'll take what I can get).

What does HN have to do with the theme of this guy's blog?

If you're struggling to read it, try https://www.readability.com/shorten

Agreed, the font is very light weight. The way it renders on my screen (Macbook Air) gives me a headache.

I'm glad it's not just me. looks like he changed it to something else

The difference between this font color and #000 is actually really really minor at a larger scale. The issue tends to be exacerbated by smaller print.

I don't bother reading it from the source, push it to pocket, and off I go with their stylish style :)

It's just a common Wordpress theme:


It seems like a mere attempt to adapt labor-division best practices from assembly plant industry. More managers, bigger budgets, easy staff rotation - that kind of crap. Also a fresh new meme to ride on. Nothing special.

I will sum this up:

> We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it.

Followed by:

> What matters is good people working together without artificial boundaries.

Former will never produce latter. Always work with engineering before building anything that impacts their day to day.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact