Hacker News new | past | comments | ask | show | jobs | submit login

I’ve long held this opinion but I consistently get drowned out.

DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways.

All “devops” as a job title has done has muddy responsibilities and given many folks the wrong impression of what an operations discipline should be.

There is also a lot of rewriting of history that gets thrown in, similar to how when people talk about cloud then the only alternative is to start making CPUs by hand and begin building your own nuclear reactors. It’s the idea of what came before, not the reality, that people seem to be defensive of.

It’s honestly exhausting to discuss.

So instead I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.

Roles so simple that you already have a clue what they do, which is sort of the point of job titles.




Agreed, anytime I talk with someone about DevOps ... we end up having to hash out the entire process to really know what either of us are actually talking about. Otherwise you have these situations

"Yea the DevOps guy messed up the widget and nobody notic---"

"Wait, what is the DevOps guy doing even touching that widget.... what is even DevOps to you?"

"Bro that widget IS DevOps."

-silence-

Same applies to the topic of "micro-services".


>microservices

I never miss an opportunity to share my favorite piece of comedy this decade:

https://youtu.be/y8OnoxKotPQ


it's too close to reality.

There is a good talk that is also comedic as it comes from the perspective of someone who wants to fail at doing microservices.

https://www.youtube.com/watch?v=GWgRw5jiYy0


Thank was pretty funny, thanks.


SOA is just Spaghetti Architecture. I haven't seen an implementation that convinces me otherwise.


What is your definition of Spaghetti Architecture? Netflix had a good SOA that enabled rapid development and had strong cut lines between services, with no way to access the data of a service without going through the service's API.

I think that's where most people go wrong. They put a bunch of services in front of a shared database, which means that they don't have to go through a service's API to get to it's data, and that's what breaks everything.


This is really important, and I've gotten a lot of quizzical looks when making this assertion over the years: data is owned by one and only one service. If two pieces of code assert ownership of it by mutating that data or looking past the public encapsulation of that data, then that code is the same service.

If you see a queue between two services, that is usually an indication that some ownership is being transferred (even briefly), and that is a critical point where you need introspection and observation.


> Data sovereignty per microservice

> An important rule for microservices architecture is that each microservice must own its domain data and logic . Just as a full application owns its logic and data, so must each microservice own its logic and data under an autonomous lifecycle, with independent deployment per microservice.

https://learn.microsoft.com/en-us/dotnet/architecture/micros...


The big issue I've seen is that takes a lot of work, so people cut corners. The problem is of course when you cut corners with microservices and rely on a shared database for instance, suddenly you're dealing with 40% of the costs of a microservices and 0% of the benefits.


That's completely untrue and glosses over the very real costs of transaction management in such an environment.

Using a shared database allows you to punt a lot of that complexity to a system that's been specifically designed for it, and working well for probably 20+ years.

Too many people think microservices don't have their own, severe, downsides. The likes of netflix, google, et al, can afford to pay people whose entire job is to manage the complexities of these approaches that flat don't exist in other scenarios.

But it's a hell of a lot simpler to use a single database if you can get away with it.


Shared database can be a reasonable microservice boundary, especially when using database-as-queue or database-as-mucroservice. In the former, services a, b and c can insert and query but only D can update, i.e. any service can creat a work-order but only D can mark it completed. I. The latter, nobody can read or write and all access to tables is done through stored procedures.

I don't recommend either, and it's still a code smell, but with clear definitions they can work.


The benefits of the microservice pattern are you can build separate teams responsible for different business logic, and they can have their own deployment schedule. You don't get either of those benefits when a b and c have to coordinate on their work-order creation business logic, and a, b, c and d all need to be deployed at the same time anytime the schema changes.

Btw some of the confusion might be what I mean by shared database. I didn't mean two services sharing a limited set of tables, using the database as a rabbitmq replacement. I meant sharing the backing database of the microservices. It sounds like we probably agree I just wasn't very clear about what I meant by sharing a database.

(Ironically I am literally using a database as a queue to share data between two services we are running in prod. I don't think of it as a microservice because it's not separate teams, it just because our monolith is hosted on a platform that doesn't support certain libraries so we had to role those libraries onto a separate platform that does support them.)


I think we're in violent agreement - A database can be a queue and that doesn't link things as the same, but it's easy to break those promises and you need to be clear in that promise to begin with. If two "microservices" have to move in strict lockstep, they're not microservices, they're components of a larger service.


Yeah, services should be formed around the needs of data, and you shouldn't run multiple services in the same image/container just to split the code up. It shouldn't ever be about the code.

Amazon's SOA architecture at least all started out about pulling bits of data spread over hundreds of servers into services that could take advantage of caching.


This is it exactly!


I have seen the "data service" design pattern that tries to work around this, by having a CRUD microservice to front a database, providing hundreds of APIs to read/write the data for 50 other microservices. If one of these CRUD microservices goes down, everything breaks.


I feel like it has its place in large organization. But being a large organization means the odds of spaghetti are approaching 100% no matter what you do.


Any general purpose architecture allows spaghetti, there's no way around it.


Has there been an architecture movement that formally embraces the spaghetti, aiming for peaceful coexistence? Detractors will certainly point at SOA and shout "that one!", but I mean one that openly admits..


I’m not sure if this architecture has a name other than “Event Sourcing” but I think giant org wide shared message bus with all services freely pushing and pulling data from it is the closest to embracing spaghetti.


Microservices should be resource-oriented, not service-oriented. NetKernel takes it to the extreme, it’s a shame it hasn’t seen broader adoption.

http://resources.1060research.com/docs/IntroductionToResourc...


"Hey, did the front-end new-hire upgrade the Linux distro yet?"


Side rant:

There was an angry Ask HN post a while back about how terrible the "new guys" are that are out there.

It then complained that the new hire wrote a horrible authentication service.

I thought it was an intentionally absurd post about the expectations put on new some rando new guy to write something important / they shouldn't be working alone on ... but they were serious.


this devops is also a dumping ground for anything else not happening on a developers computer which they should control but wont because "That is devops job" mostly this happens alot with node projects for some reason..


"anything else not happening on a developers computer which they should control but wont"

Ok, I've seen this, but IME (24y in industry, the last 6 as a consultant) in the vast majority of cases, it's more like "things devs should control but CAN'T [bc CICD etc are silo'd and owned jealously by an overburdened ops team unable or unwilling to facilitate self-service]".


Some of the 'jealousy' may also come from bad experiences.

It only takes a few instances of people royally messing stuff up in production for their deployment rights to be stripped away – and other groups catch the fallout too, as new "procedures" get implemented.


Also, some developers do not seem to know what kind of impact a production issue can have.

Having something not work during the development cycle is annoying for one person or a team. Having something break in production usually means people get paged and (depending on what you do) the resulting issues can have major societal aswell as economic impact.

In the ops world, this is especially visible if you go further down the stack. (from applications down the the network).

Network architecture moves at a toad's pace compared to webdevelopment, which is a good thing considering breaking the network will usually break every other system inside a IT department/landscape.


"Safety rules are written in blood" applies to parts of the organisation too.

Sure, there are some overbearing procedure monkeys who really want a process on everything, but a lot of "protection" rules are there because something really bad (and expensive, financially or reputationally) has happened before.


Humans are kinda terrible when they can just "not my problem" things.


>>the only alternative is to start making CPUs by hand

Agreed. For some applications the cloud difference is significant; for many (most?) others though, "Cloud" is just rebrand of "Hosted". And even for more cloudy offerings, while I'm in a very specific and different part of IBM, some of the old timers/architects/powers-that-be keep trying to explain "We had that in 1969!!!" :-D

Agreed also at rewriting of history when it comes to development/support/operations models. My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)


> My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :)

As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all.

My personal favourite is how many large multi-nationals are now building in-house clouds?

WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?


> WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?

A couple million in salaries and bonuses.


I think the difference is the interface.

The interface to the shared-use datacenter, if you're lucky, is a spreadsheet that declares the static resources you own and a remote hands guy that can tackle things beyond the capabilities of your remote KVM. If you need more capacity you need to work with the datacenter folks to order physical machines that might show up in a few months.

The interface to the in-house cloud is an API. In most instances, developers are completely abstracted away from the physical infrastructure and don't need to take a lock on some human in the datacenter to get their work done.


I still think that's a bit "rewriting the history". E. G. Vmware enabled fast self-provisioned VMs and nobody called them cloud. Heck in 1999ish or so, while at university and way before I was an IBMer, I could sign up to some ibm development program as a student, get an account, and provision Linux VMs on mainframe.

Not to say your scenario isn't valid and real, but I live that scenario every day today with in house cloud and virtualization too - it takes months of approvals and solutioning and security assessment and network engineering and procurement and costing and whatnot... To deploy a windows vm.


> Vmware enabled fast self-provisioned VMs and nobody called them cloud

Because it’s missing all the glue. I need to self-service the VMs, the database, the load balancer, the dns records, certs, and hook them all up so I can receive production traffic all via an API I could theoretically do in Terraform.


> WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?

An in-house cloud sounds like the mainframe installed in the raised floor computer room at the school district office I worked at in the late 90s; of course, the 12 foot long Unisys mainframe was replaced with a Unisys 4U pentium pro box pretending to be a mainframe, and then there was a lot of extra floor space.

If you're running your own 'cloud' in a (shared) colo, I dunno that that's really in-house. I guess it's still 'private cloud' though.


  “There aren’t any new problems. Just new engineers”
  - A sig I read a long time ago


> WTF is the difference between an "in-house cloud" and a shared-use datacenter from the 1990's?

Can't call a mainframe a bunch of buzzwords like "Hyper-converged, high availability, on-premise software as a service cloud platform".


I'm pretty sure mainframe guys are calling it exactly that:

https://www.ibm.com/ca-en/products/z16

warning: buzz word alert!


Turn-key cloud technology.


The only possible value over the last iteration is if they cleaned house and the new team manages to be more permissive than the old one. I'd much rather have on-prem, but the sad fact is that for cloud I just need a signed check. For on-prem I also need buy-in from other divisions before I can even start experimenting with a new service.

I still have the exact same problem with respect to never having exactly the ratio of CPU to memory that would make my app happy.


I had a couple of telco customers that ran their own internal cloud, and I’d say the main difference is that it’s way more expensive than the public cloud, heaps less flexible, and when you need compute it has to go through an approval process.

I mean … sometimes I shake my head in wonder, and other times I just shake my head.


An in-house cloud will just be a bunch of commodity servers running a hypervisor that gives you an API that allows you to automate the provisioning of infrastructure.

I am guessing that in the 80s you weren't writing Infrastructure as Code to define exactly what resources you needed for your software, having it all set up automatically, and so on.


Isn't that what JCL did?

https://en.wikipedia.org/wiki/Job_Control_Language

True, they didn't call it "Infrastructure as Code" but it was used to define mainframe resources for jobs to use when they ran?


The two of you have validated my very existence as a coder. I keep jokingly telling everyone that we’re often going around in circles. No one ever believes me (lol!)


I don't understand how seemingly 90% of developers, can't do anything outside of their narrow scope of experience. Writing a shell script? Throw a temper tantrum, that's devops job! Have to work with legacy code? I can't believe this!

Especially seniors and beyond. They force leetcode interviews they somehow pass or are grandfathered through and gatekeep "trash devs" by slamming gotchas about how the whiteboarded code discussion doesn't technically compile and how "returnAverage()" isn't a method ("did you know you have to write methods before they work? What does returnAverage() supposedly even do? Return a random character in the alphabet?")

I get excited about working in different tech areas. I'm exceptional at fixing other's bugs and maintaining code. Absolutely nobody seems to be hiring for this though. It's all about college exam trivia and leetcode.

When I do get into these companies, I have to work with people throwing around casual racism with HR joining in and f-slurs like it's corporate 4chan. I ask if anyone can help look at a critical bug I discovered, and nobody speaks up. Even when it turns out it was their last code change that caused it and they are the sole master expert in this area, and they were just working on organizing their desktop instead.

"I can try to look at the SQL problem, since nobody else spoke up." Then I'm explaining the basics of SQL to some guy sitting around blank faced and it turns out he was hired because of his 10+ years of expertise in SQL. His whole job is to tackle problems like that.

Meanwhile I'm out here looking for jobs when panic cuts happen and it takes forever because I'm drained from all the gotchas and gatekeeping in these interviews.

There's no way I could live with myself if I put this experience onto others. I lose sleep when I fail to call out someone talking over a quiet person on my team. I've never seen anybody else stand up for anybody, however.


I learned SQL because my coworker was stuck on a bug she couldn't figure out and 'knew' I'd be able to solve her problem.

I just kept asking questions until I found the one she missed.

I also ended up becoming a bespoke VxWorks admin because the guy who volunteered was never available, and someone asserted my code wasn't working well on VxWorks, so of course I had to know enough to do benchmarks. I fixed a few problems but the real issue was the hard drive wasn't doing DMA due to the kernel not recognizing the processor revision number.

Somewhere between those two events I realized that if my part of the project is great but the whole project is on fire, nobody cares about my stuff. I don't get points for being right and the team being wrong. I mean, I do for some people, but I still feel bad at the end of the project, and those are the 'points' I have to live with the most.

The way that played on on the latter project is that once we got our shit together, we started work stealing from our peer orgs. Volunteering to carve off little pieces of interface between us and 'take care' of this bit of data handling here and that one over there. I came to realize that the Org subconsciously knew this and any past success they had was due to self-organization and collective work-stealing.

Throw people at it, turn a blind eye to precise mission statements, and hope for the best.


You hit the nail on the head - at many places, the hiring process seems to be great at blocking people who are great at solving real world problems that aren't as good at whiteboard leet-coding.

If you truly enjoy this part of the job, you can go for a freelance career - build a network and reputation as the person you call to solve the hard stuff, and do mostly that, no leetcode interviews required...


I like to build up my teams with people that act and think like you. Sometimes I need a sniper and when I do I'll get one. But my teams need to be excited about building and running the systems that we make.


To be fair, shell scripts are pretty horrible to write and maintain from a programmer's point of view. They're good in that they're portable, but they're not a good programming language. I don't object to writing them, but I'd rather there be an alternative.

Anyway, it sounds like you've worked in some really shitty companies / shitty people.


Sounds like medium or bigger sized company office politics.

At smaller companies the title tends to be only a title and many people are basically full-stack and know a bit of everything. And you're trading hats to get jobs done.


> I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.

This is a great way to do it. There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity. Being direct about roles goes a long way towards simplifying corporate complexity.


> There seems to be a correlation between unnecessary product complexity and unnecessary corporate complexity.

That's Conway's law.

"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."

https://en.m.wikipedia.org/wiki/Conway%27s_law


Most of the useful insights in the Devops movement can be derived from knowing Conway's law and thinking about organizational patterns holistically. As usual the success stories get cargo culted by others without understanding the principles and thought processes that led there, and usually fail because every organization is different and has different needs. Change is hard. This will happen with every shiny new movement like Devops or Agile.

I'm sure we're due for a new one soon which will follow the same path, since people seem to be admitting that Devops has problems on here more and more. Will that cause deeper reflection? Probably not for many orgs, because soon some consultants will dream up a new brand of silver bullet to slay the immortal monster of organizational dysfunction and that's much more exciting.


Conway's Law is sorely overlooked. It really does have broad applicability in the industry, and sometimes you can use it to evaluate vendors better (read: this product is a bit haphazard...the vendor probably is also. Warning.)


Totally agree. The best succinct summary of Conway’s Law I've heard is “you ship your org chart.”


I like "the org chart is the asymptote," i.e. it's the best you could possibly do, and your reality will actually be worse. (see Casey Muratori)


Which is a good way to word it because if your org chart is screwy enough you won't ship at all.


Roles are muddy because life is muddy...and because people can do more than one thing at a time. Also because save for a few roles, businesses rarely need people 100% allocated to one role or task. Responsibilities are nice, and job titles mean people mostly do x or y.

My favorite example is that of a restaurant. You hire waitors, diswashers, bartenders, line cooks, prep cooks, hosts and hostesses, and managers. And your waiters might be 100% focused on taking orders during prime dining hours, but at the end of the night, they help clean up assuming other roles. Your bartender might be busy making drinks, but they take a table or two if they can. Your prep cook might finish the prep early, and come on the line to help out. Your manager focuses on expediting everything and keeping the ship running smooth, but also can fill holes at time.


Communication with the other roles you work with is key though in that case, otherwise you'll have chaos - e.g. because the waiters don't know that the bartender took the table, or because the cooks need things the next morning that aren't where they're supposed to be.


Exactly.

You wouldn’t expect a bartender to table every person, prepare all their drinks, cook all their food and clean up their table.

But that’s kind of what we keep trying to do, because we have new bartending tools that make it so that you need fewer and because we claim communication is harder than doing everything in one role.

Then when that person invariably gets overloaded; we hire more bartenders and complain they’re not good enough at cooking your specific dish.


That depends on the complexity of the development lifecycle, doesn’t it? If a single person can maintain it, having three different roles would mean unnecessary corporate complexity.


Not sure what you mean with "development lifecycle" exactly. Aren't maintenance and development both part of the product lifecycle?


Yep, this has been the hardest fight in our "culture shift towards devops" - getting the idea and wording right.

After a lot of discussion, we mostly realized that we get the most value if we define more specific operational roles. We now have the idea of infra-ops, and product-ops. infra-ops is providing a deployment platform - a container runtime and persistences. product-ops on the other hand is responsible for deploying and running the different products from development on this deployment platform.

And this is giving people good ideas. Some products are very simple without harsh requirements. In these cases, one of the backend devs just takes over the product-ops role by setting up a deployment pipeline, a job and some migration handling. Other products are bigger - for example we are providing some of our systems as essentially managed systems to invididual tenants and customers. In such a case, parts of the product operations is with the product team - such as writing jobs, releasing containers and artifacts, and other parts of the product-ops role takes place in the managed services consultancy. These products are currently looking at wordings to split up the product-ops role according to their needs in their context. And that's totally fine - the infra-ops role is also divided depending on the system the engineer is working on. A postgres admin is an infra-operator, for example.

And all of this results in a devops oriented culture of cooperating across team boundaries through automated processes.


"Sysadmin" got rebranded as "DevOps", because "techsupport" got rebranded as "sysadmin".


Yerp. As a former *nix admin, around the time that "DevOps" became a term, I read it as "system administrator, who also now has to fix developer code." Which, honestly, was already part of the job at a lot of places I've worked, owed to the age old problem of people who "test" something on their weird personal desktop environment, then hand it off and shrug off any questions with "well, it works on my desktop". That said, the part that I found offensive was that I was doing 2 jobs, but only getting paid for one.

Then came DevSecOps. Around that time, I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.


> I switched to just Security, and while I miss the ability to make and push changes to hundreds of thousands of machines to get things done, I don't miss any of the pressure or blame that automatically got lumped onto the sysadmin shoulders every time anything went wrong, and the complete lack of appreciation of all the times nothing went wrong, that were entirely the result of a tireless, efficient systems administration team.

This reminds me of a joke my networking collegues used to tell to the new hires.

Welcome to the networking team, where we are responsible for the network, aswell as anything that runs on the network because developers never bothered to pay attention in their networking class.


It’s a tough place to be when the bar is perfection


It's tough, until you stop caring. And obviously, hitting the "don't care" point is pretty sub-optimal, so far as being able to muster any enthusiasm necessary to keep doing the job.


Yeah, I've often seen "DevOps" described as "System Administration, but automating everything you can with scripting or whatever", to which my first thought was "WTF do you think Sysadmins do, if not exactly that?!?"

It's like they'd never heard a BOFH tell someone, "Go away now, or I will replace you with a small and uncomplicated shell script."


glad someone said it. I mean all the CI/CD pipeline are just in the old day *nix/bsd days bash/perl/awk/sed/python/ruby (fill in the blank) scripts. One problem when IT/Software development become more main stream is that everything got another layer of obfuscation especially in corporate cultures and amongst sales and recruiters and managements. Development Operation or pipeline sounds sooo much better bash scripts.


Just bash script isn't enough if there's no automated process that run it every time developers make changes to your main/release repository trunk. If you still run it manually to verify your build, then it's not a CI. If you're still doing manual release deployment, then it's not a CD.


The extent of devops for your developers should be:

1. Push code to an automated pipeline

2. I understand that the automated pipeline may need input from me in order to run successfully.

Beyond that, you need other people building the "platform" that they're deploying to (for the most part). Probably the ideal example here is Heroku.

Your devs shouldn't be required to do all of the actual ops stuff. They need access to an ops platform.


To grossly oversimplify, there are essentially two views of DevOps.

1.) What's probably the traditional view which is breaking down the walls between devs and ops, developers carrying pagers, etc. I.e. at least in an idealized world, there are no devs and ops--only DevOps.

2.) As you suggest (and which probably more closely matches how "DevOps" works especially in larger organizations), an internal (or external) operations team provides a platform that developers can use. Developers are still going to be exposed to some operational details, but a lot of them are abstracted away.


3) The way I first heard it described years ago, bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc


> bringing developer practices into ops: version control, testing, code reviews, continuous integration, etc

In my opinion, this last subset just doesn't work well for a lot of branches of operations.

Sure, it might work if you run stuff in AWS and have an operations team managing that, But what if your team is responsible for things like Storage Arrays, networking equipment?

Doing continious integration and code review on some of these is hard if not impossible


> Your devs shouldn't be required to do all of the actual ops stuff. They need access to an ops platform.

This becomes a lot more apparent once you're dealing with B2B customers and their nonfunctional requirements.

We've had situations in-house when devs built some simple microservice to handle some connection to a customers BI/DWH system - it's just 2 days of spring boot chugging to wrangle APIs around without state, nothing bad. But then, the customer started blasting that team with SLAs, backup questions like RTO, RPO, retention, regulation adherence of retention, the whole stack of IT security down to the physical access control.. That poor PO was caught just like a deer in the headlights.

This is some operational responsibility we're taking over for our dev-teams. We're providing the persistence, and this includes a defined backup and recovery strategy, a security strategy and such. And this also includes experience in dealing with these insane questionaires, and correctly pricing absurd requirements for custom backup solutions. And in fact, "we will have to schedule a discussion with ops about this" has ended quite a few of these requirements with "oh.. it's not that important". Intimidation with long job titles and appeals to external authority do work.


Highly dependent on the product. Anything infrastructure related you’ll have a really bad time with that model


I expect my developers to troubleshoot a failing pipeline too. The pipeline is part of the "ops platform" they're expected to work with.


The article has a solid solution ;)


DevOps is part of my job description. I was fuzzy on what it meant before, after reading this article, I have no idea what it means. As a general term, it's far too vague, when in practice it seems to just mean that you'll be splitting your time between writing regular application code and writing infrastructure as code. Oh and you get to be on-call.


i'm working on an RFP response that has the word "DevSecOps" sprinkled around here and there. That's an even more subjective and ambiguous term than devops :/


It means you're gonna be doing dev, ops, security, compliance, UX, UI, cost optimization, build, test, release, and on and on... good luck!


I'm convinced the use of the term "DevOps" continues only because of its incredible polymorphic vagueness...

Two intelligent, sincere, experienced tech people can discuss devops, in detail, for a good length of time, and still be talking right past each other without really noticing.


It exists so people who dole out money can make one person do two jobs.


You hire build engineers and release engineers seperately? Does your product demand that level of specialization for these 2 functions? Just curious


I work on AAA video games, so unfortunately yes.

It's ok to have build-engineer doing release engineering if it's simple enough, but doing build engineering properly (Source control, artifact control, shared caches, dependency management of the compiler outside of the standard toolchain) is quite a large job.

Release engineering itself gets complicated when you take into consideration the different target platforms, most games get published on Playstation, Xbox and one of a handful of PC platforms (notably steam), plus the online systems and of course the development environments and internal release systems which are ubiquitous.

Build Engineering usually is the step between developers and QA, and release engineering is the bit after QA.

Both interact a lot with QA so could be rolled into the same role.


Do those different breeds of engineers work in one team or in separate organizational branches?

The problem the term 'devops' is addressing, the way I understand it, isn't that of insufficient individual jack-of-tradesness, but that of too much separation between (internal) organisations. It should be perfectly possible to build a "devops" team from deeply specialized experts and turning it into a job description seems quite a stretch to me.

But you might actually want some jack-of-trades types nonetheless, because those isolated organizations weren't completely without merit: they are good at solving that problem affectionately called the bus factor. A single ops guy in a devops team is a single point of failure and the mitigation preparation of keeping some of the not-so-ops peers sufficiently in the loop will be much more dependent on organic motivation than the counterpart in a specialist company branch. There, substitutability would be much easier to ensure with formal process.


> Do those different breeds of engineers work in one team or in separate organizational branches?

It doesn’t matter as long as communication is open.

Proper ops is all about reducing the bus factor. Procedures don’t depend on people, they depend on being good at communicating.

If your ops person got hit by a car, I would expect that everything would continue running, I would also expect to find some clear documentation, this is mandatory.


DevOps to me is all about bridging what used to be a huge gap between operations and development. Developers need to know top 10 OWASP. One cannot simply say "it works locally". Ops people need to understand how your tools and programs work at a high level at minimum.


Developers and Operations have been working together for a long time.

Some teams historically worked fine together, others didn't.

I think the tools we have these days are much better (for ops) to ship what works on developers machines.

But if your organisation had a culture of not working together then devops didn't really do much.

and anyway, you've leaned into my point about it meaning different things to different people.


These two talks will change your view on DevOps forever:

- https://www.youtube.com/watch?v=zwSNjVTF168

- https://www.youtube.com/watch?v=MnyvgFDh-kw


not if I don't watch it.


I get called an "Infrastructure Engineer" sometimes, I've also had the title of SRE and SWE. I don't really feel like any of these actually fit me.

I do work on Cloud Infrastructure at times, but honestly it's the smallest part of what I do. It's usually addressed in the architectural designs and I have to touch it incrementally. What I do much more often is writing tools, daemons, and services for distributed systems. I end up calling myself a Distributed Systems Software Engineer to reflect the idea that what I work on is systems and software, and most of them are non-monoliths (from a systems perspective).

Do you have any thoughts on Systems Engineers or the title that I prefer to call myself?


In the games industry we have a role called “tools programmer” which is a person who works on tooling to accelerate development, sometimes these can be extensions to the editor for a specialisation for a game (say, a tyre builder for a racing game) and sometimes they can be more broad (a distribution tool using libtorrent so that a new build of the game can be shipped worldwide very quickly).

There are people who might refer to you as a backend programmer (if the focus is services and daemons), or platform engineer (if the focus is developer velocity).

Largely it depends what your primary focus is.


I don't know, every time I see a title split people start saying that's "X's job", you can't do that because it's "y's job", or I don't want to be involved in "Z".

Fewer titles can avoid these discussions.


Looks like you already identified the people that should be in those roles.

If they're not doing it or was overlooked for some reason, then that's an issue but not really an argument to not do it.


Since when are managers incentivized to put people where their interests align? particularly when there is a perceived role transition.


Sounds like a bad culture.

That won't be fixed with a new job title.


You hire those engineers and then you have them implement and support Continuous Delivery for your business, yes? DevOps is a silly term, but one that companies seem to use for "infrastructure engineers who help us continuously deliver value to our customers, efficiently and pain-free". Give your people whatever title you feel comfortable with - but essentially they are "doing the DevOps". :P


Did I get downvoted because I'm right and the truth hurts or is there a disagreement somebody chose to not share? Hmm...


You basically assert that the other person is categorically wrong without argument but by rereredefining words. It doesn't read as productive to me.


> I hire infra engineers, build engineers, release engineers and: backend engineers.

I always thought DevOps the “function” just meant this, and being a DevOps engineer at a small company meant you did these with decreasing emphasis, where by the time you’re in the backend it’s just helping enforce logging, tracing, other observable components.

Has worked for me in hiring and being hired and almost everyone I know understands this.


I think everyone can agree that for most complicated things there are many sides to a story that makes up the present state of the system. I think taking a very opinionated stand in any complicated subject by itself negates the possibility of a mature and complete analysis of any subject. So the title itself of this article is unfortunate.


Shouldn't a mature and complete analysis result in a conclusion, i.e. an opinionated stand backed up by correct reasoning?


Yes it should result in a conclusion but the conclusion should not simplify it in a way that the true nature of the issue is lost.


Some time ago, i was talking to an MBA people /project manager guy was ho had a completely different definition of what "devops" meant. His meaning was some kind of project management approach.

We have a way to prostitute words in this field... agile, QA, devops.


I agree with this but I’d add that it’s also important that the roles aren’t overly siloed. A front end engineer, for example, should be able to and even encouraged to look at backend code, and even submit PRs for it.

And more importantly, the backend engineers should be encouraged to be grateful for the PRs (even if it’s not acceptable for some reason). The more eyes on the code, the better.

I think developers naturally tend to specialise (at least for a time) but ultimately we all need to understand and contribute to code regardless if it’s front end, backend, build, test, …


>DevOps has different meaning depending on who you’re talking to

1000% this.

I think it's an instance of how the conversation in software always talks about solutions as if they're end-all-be-all answers, rather than explain a problem in specifics and why that made the solution the right answer at the time.

Then that "solution" becomes a buzzword version of itself, and popular buzzwords are tools leaders use to overcome institutional inertia. Which is a good thing, to get over that inertia. But then what comes misses the insight and understanding that eventually turned into a buzzword.


This is the answer.

I am interviewing right now and my previous experience includes devops roles and SRE roles, so I get contacted for both by recruiters. After hearing about the responsibilities and examples for these roles I can only come to the conclusion that titles are a waste of time. Even the "level" of a title of Staff, Principal, Lead, etc are a waste of time. Just call the job what it is and if you can't decide on what it is at least to an 80% level, maybe you're asking too much for that job?


Are you hiring? I was hired as an infrastructure engineer and I'm currently writing typescript for the frontend of an internal product.


Yes, I am, the project is mostly in stealth mode but please feel free to drop me a CV:

jan [at] competition [dot] company

If you want more information on the project we're working on the site is https://rennsport.gg

We're building a hard-core racing simulator game with a backend which can persist car ownership in a way that feels authentic. (IE; not just tied to a game).


> DevOps has different meaning depending on who you’re talking to, even some definitions that appear similar are different in nuanced but important ways.

This is always a dead giveaway that something is a buzzword

Same for rest. Sometimes when people use it, it just means JSON + http requests. Other times it's supposed to be some kind of architectural style


You can always point people to the original paper that defined REST: https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arc...


Should one also point people to the original vision of OOP (by Alan Kay) when they mention the term?


Yes, at least when there's disagreement about what OOP means. You and I can disagree about it, but we can more easily answer "Does it match what Alan Kay wrote about?" with a yes or no.


I agree - I think it it is much better to use titles like these and with even a short JD describing what technology is used (many of which could be "devops" tools for infra engineers) everything becomes much more clear.


Agreed; I have always thought of it as a rationalization of bad practices.


I suspect most places just use the term DevOps to try to get two jobs out of one person.

Then they also tend to have to do their own security and spec out their own tickets and then do Project Management.

Most tech people should be called DevSecOpsBizAnalystProjectManager.


>> I became CTO so I can solve this mess properly, I don’t hire devops, I hire infra engineers, build engineers, release engineers and: backend engineers.

This makes a lot of sense, but you need to realize that this is only a tiny part of a successful organization. There are many setups like that which you describe across many industries that have failed reasons beyond the role definitions. It's not enough to establish the organization. You have to keep the behaviors in check over time when you hire a build engineer with different aspirations.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: