There are some other dynamics that are worth mentioning here that can stand in the way of undertaking any of the steps the author recommends. I don't think they touched on any of these things so I'll list some of them out:
* Technical staff in an organization feel they have no authority or respect within the organization. This can happen when companies become too sales-driven.
* Senior technical staff feel they have no ability to delegate, either because of the previous point or because they don't trust the technical ability of less senior staff.
* Those in an organization who do have authority to delegate lack sufficient understanding of priorities or lack technical knowledge.
These things highlight the classic balance between technical knowledge and managerial skills. If you don't have people that bring both to the table, you're in for trouble. Not sure if I'd go so far as to say this is what it's all about. However, reading over the author's advice, I thought to myself, "None of this could have happened at some places I've worked." You can come up with a good plan, but if no one will listen to you or take your advice, it's worthless.
The author probably was that person who brought what was needed to the table. As such, perhaps the plan is not what ended up mattering so much as having someone who was willing and able to push it through. Also, having the right people from top to bottom. My feeling sometimes is that effective teams occur almost randomly. This might be because organizations rarely have the patience or the resources to assemble them deliberately.
Particularly: "perhaps the plan is not what ended up mattering so much as having someone who was willing and able to push it through.". This is very true. It was a combination of having tenacity to grind through a lot of politics and repetitive conversations and doubts that everyone had, while also addressing extremely real technical and process issues. Previous leaders often brought one or the other to the team: they either knew what needed to be done technically but couldn't push a plan through, or were politically savvy enough to get changes implemented but they just ended up moving deckchairs around. When I stepped in, I inherited a team that did this twice previously and everyone had zero trust and learned helplessness. I wrote this up and tried to emphasize the slowness of the process and the fact that it wasn't a "one simple trick" type approach because that was the case.
Or: "If no one will listen to you or take your advice, it's worthless." This is true. We had a few really brilliant developers cursed with a cassandra complex: they correctly identified the problems but given their way of communicating, they actually fostered resistance to the changes. Communication of the plan and building consensus is as important as the plan itself, if not moreso. Every plan will need to be adjusted based on reality and context, but as long as you build consensus and goodwill, you'll be able to have that flexibility.
With regards to your point about technical staff, we were an enterprise sales organization where decision making was concentrated in the finance and sales departments. It was extremely sales driven. The reason we were able to push this through was precisely because the company had lost faith in product and engineering to accomplish anything after a series of poor launches and chronically missed deadlines. I actually think the problem is that if engineering leadership did have more trust, they would have pushed back on this. It didn't look good to "give up and try to just do one thing at a time" as we did. For most organizations, that's painfully radical. But in the words of Munger/Buffet: "Don't just do something, sit there!"
After the MVP is found, then the real work begins in building a company. We all understand that if everything is a priority, then nothing is, but often times, business resources don't. Support wants bugs fixed to keep customers happy and to re-sign. Sales whats features to land their whales. Engineering wants to work off tech debt, refine, and innovate. Etc.. I've found that a formal "Intake" process works wonders. Engineering needs an advocate to take these disparate priorities and refine what, when and how to address. It's really that simple, and that complex. Finding someone to own it. Sometimes it's impossible depending on the Org and personalities involved.
> Engineering wants to work off tech debt, refine, and innovate.
I know I'm not the only one who is backing away slowly from the concept of tech debt, and I'm not the only one trying to drag other people with me.
Tech debt has turned into hippie-dippie bullshit in the minds of many people. It was an attempt at enlightened self interest by appealing to the bean counters, but if you're talking to bean counters you've already lost.
The developers want easy things to be easy so they don't have to spend half their time slogging through code that does everything the wrong way, half their time explaining why everything takes so long (heading toward infinity), and half their time implementing new functionality. And yes that's 3/2 which is why we have people doing 60 hour work weeks.
A long time ago I saw a speaker claim that it's not tech debt, it's wear and tear. I don't know if I agree entirely with that characterization, but I definitely think that it's less wrong.
A lot of what people call tech debt is just working code that they don’t understand (maybe because all of the people that did understand it are now gone). Or it’s complex, which may just be because it’s dealing with a complex issue and accumulated lots of bug fixes over time. Or tech debt is just nitpicky design choices which may very well be not-great but aren’t actually causing any problems. None of that is worth investing in beyond a fix-as-you-go, in my opinion, due to there usually being problems to solve with much greater business impact.
On the other hand, you have tech debt like: a complex set of dependencies/services with poor interfaces that makes what should be simple changes very complex and risky. Something implemented so inefficiently that it’s slow enough to impact UX or wastes a lot of computing resources ($$$). Or code that is just so, so bad that making any change is like walking in a minefield - and while a lot of engineers are way too quick to put code in this bucket, I do think it’s possible (eg a complete ball of mud application with tons of global state, very poorly applied concurrency patterns, extreme lack of parameterization in favor of copy-paste).
That said, I have often seen “tech debt” used as a rationalization to work on some nitpick with no business impact and without a clear improvement. In my opinion, this is partially because working on such tech debt can be very easy: it doesn’t require understanding more complex business requirements, it might be very easy to test with existing integration tests, it has little risk to production, and it doesn’t have task/planning dependencies with other people.
Here's the stance I take: there are many reasons why something might be labeled tech debt. However; assuming the remediation of that debt is implemented in good faith, by reasonable skilled engineers; there is near-always business value in remediating it. It may not always be the same $/hr return on investment, but there is usually positive business value.
Secondary point: there is "near-always" positive business value; near-always; and its actually impossible to predict if a given tech debt mitigation project will result in negative business value. You can only know in retrospect.
An example you give and label as bad: some crusty old system, outdated language, legacy, everyone who knows it is gone. But, it works.
Mitigation: rewrite it in something more modern that everyone on the team knows. Do that, and out of thin air your organization just minted a handful of new Experts on the system. They're forced to dive into the old code. Document the new stuff. Projects like this can be pretty fun!
Counter-point: The old system works. Why change it? We just need engineers to dive in, learn it, keep the old crusty libraries and frameworks up to date, lets hire and plan for that. And people fuckin wonder why engineering retention is so bad! "No one who works here knows how it works" Huh I wonder why they left! Let's pass all our problems onto the new generation; they don't get to own anything, their job is to fix shit and bolt on new functionality at a tenth the rate they feel should be possible, enjoy being an indefinite renter of the system until you get as fed up as the generation prior, move companies, and do it again.
Business value? You're providing cool projects to your employees. That's it. We'll make up reasons it benefits customers. "We'll be in a better position to scale and reduce API latencies" bs. "Feature velocity will be higher" maybe. Throw out some more tactile reasons like it'll ease hiring because when people hear PHP they run for the hills. Yeah that's more correct, but the biggest reason is expert retention. Have you got experts? If yes: don't rewrite. If no: maybe rewrite. Let engineers rewrite, they feel ownership and responsibility over the system, they probably stay longer. That's it.
"Complexity" is similar. Its complex because the business problem is complex. Sure! Engineers just need to dive in and learn it. When? Is the organization willing to literally block off time for the engineers to do this? Lol. Has anyone ever worked at an org that did this? It could work! But what they may block off time for is: rewriting it to make it less complex. The end result probably won't be less complex. But you'll have gained an SME! And the organization can feel good perceiving the problem area as "now less complex" (whereas in reality all you did was train up an SME via way of a refactor).
When does it go wrong? Obviously; if a legacy system works, or solves a complex problem, and the rewrite doesn't quite capture what it does. The old code is right there; that's the spec. If the new doesn't match the old spec, it probably comes down to someone not fully understanding the old code. Guess what: that was gonna cause a problem in the future. You just expedited and uncovered a potential problem; but in a context where it can usually be remediated significantly quicker because you're building off a new system with tons of experts, rather than an old system no one likes working in. That's business value!
And sure; there are much rarer cases where the entire project goes way over time, the new system sucks, gets caught in development hell, people leave mid-project... that sucks. It can happen. But I'll reiterate: its literally impossible to know with any reasonable certainty that this will happen before taking it on. This means its an extremely poor reason to not to do it.
To be clear: I am not asserting that addressing tech debt should always be a priority. I'm asserting that there is near-always positive business value, and that People are extremely bad at estimating what that positive business value will be because it doesn't have a dollar-sign attached to it like CS and Sales orgs have.
> just working code that they don’t understand (maybe because all of the people that did understand it are now gone).
Well in that case the real debt is the lack of documentation or clarity in the code. If the code is not written in a way that someone other than the original author can easily understand it, that's a problem. It also indicates gaps in the process of transferring ownership. There should be a team of owners for each piece of code that brings new members up to speed when people leave. Knowledge mustn't leave the company when people leave.
OP here. We had a lot of discussion about where Tech Debt falls on the spectrum of projects to work on. That merits its own post. But the gist of it is that resolving tech debt has value. The challenge is that most engineering organizations are awful at describing and communicating that value.
For example: "If we refactor this code and migrate to a new server, we'll be able to deploy in 15 minutes instead of 6 hours, and we'll be able to ship features faster to customers with lower risk. This means that we'll get back ~90h of engineering time per week (6 hours x 5 people x 3 teams), and our defect rate will go down by 50% (We see 10% of customers churn due to preventable defects).. Therefore, there is $702,000 savings in engineering efficiency (more time available) and would decrease churn by 10%, resulting in [X] more value retained."
Once our engineering team started thinking about the business case for why tech debt should be prioritized, they started to make really compelling arguments for why something needed to be worked on, and we could evaluate it against explicit business/feature requests. This value rolled up into the form of decreasing costs or increasing revenue.
Again: resolving tech debt has value, and the engineering leadership needs to work on figuring out what that value is and how it stacks up against all of the other valuable work that the company wants to do to grow. Engineering leaders who don't understand this often struggle at the executive level and ultimately fail the teams they represent.
The problem I think is engineering haven't figured out good ways to measure software productivity, and so it's very hard to make the kinds of estimates that we need to properly justify work that otherwise delivers zero immediate business value.
(I also suspect that not all refactoring work is equal, and that most refactoring work don't result in any positive engineering or business improvements.)
- 1: Keep asking "why is this valuable" until you get to the money questions. It's often a few layers deep. Not being able to get at that (missing an ability to quantify) was in itself something that engineering leadership needed to work on. After all -- how can you prioritize between two things if you can't figure out what value they create?
- 2: Most engineering teams know that they might be able to save some time by refactoring. We figured out the average hourly cost of engineering effort and used that as a multiplier for time saved by a new project's implementation. You can take it a step further and subtract the cost of working on a project from the time it ultimately saves you to remove "dumb" refactors that take longer to deliver than time they save.
- 3: If you can't get to a tech debt's value, that's a red flag as to if it actually creates value. This was true for features and business requests as well. We also said no to features that "felt good" but ultimately created zero value (some redesigns included). "Rewriting something in a new language" because a new sr manager with strong preferences joined the team was a big culprit of these sorts of projects.
- 4: We also often asked if a tech debt project needed to be done independently of other work, or if it could be included as part of a new feature that we were working on. An example was that one team pitched rewriting a microservice because it had no API. On the other side of the company, we had a project to get rid of that feature entirely and replace it with a different functionality. If tech debt projects were handled entirely within the domain of engineering, we would have spent weeks rewriting something only to throw it away a month later. Hence the need for bubbling tech debt up to the single stack rank, and thus treating tech debt projects the same as any other project we were working on (with appropriate visibility, buy-in, and evaluation of value/urgency).
> "why is this valuable" until you get to the money questions
Often this is one layer too far though. Business relationships aren't 100% about business, or we wouldn't put an adjective in front of 'relationship'. We'd just call it 'business'. If you know engineers are trying to put together a proposal to take on a project that might not be immediately obvious to the business, we already know all we need to know here. Developers don't go out of their way to put together proposals for frippery, and most of them don't put together proposals for clear business cases either. They put together proposals because something is pissing them off, one way or another.
If you say no, after they've stuck their necks out, then you've damaged a relationship with people who are not comfortable being clear or direct about these sorts of problems. People who you are paying to more clever than you. Which means when the consequences come, you won't see them coming, be it by apathy or malice.
We aren't allowed to say, "if you don't give us time to fix some of this stuff I'm going to quit, or worse," and so everyone tries to sugar coat the real reasons with things that are objectively true but specious motivations at best.
As far as I'm concerned, the original notion of tech debt was intended to be about borrowing on good will, asking people to do something they don't really want to do, but with the idea of an eventual end. That kind of sacrifice can often be quite healthy, even transformative. But what happened instead was advanced game theory. How long can we stay on the verge of social and/or technical bankruptcy without ever crossing over? And every new kind of 'check' they learn to float is a whole new world of delaying tactics.
Getting a divorce is very expensive, but you don't avoid divorce because it's expensive. If you're smart, you don't even mention that it's expensive, because if you weren't having an argument before, you are certainly having one now.
Reading these comments and replies has me very interested. I think talking about technical debts and ways of quantifying it is definitely worth another post. I'd read it!
You might find "The principles of product development flow" interesting. A large component of the book is about quantifying all work to be done, and then managing the resulting queues and WIP. Many aspects of the article reminded me of this book, but with concrete implementations of general techniques, which was very helpful.
> I also suspect that not all refactoring work is equal, and that most refactoring work don't result in any positive engineering or business improvements.
We have lots of old code in production, some 25 years old. Much of it written during the startup crunch, with little thought to extensibility and so on.
But it works.
So we have a rule to only cleanup/refactor code when we have a specific case which requires us to change it. If it requires a bit more than a minor change, we'll do a cleanup, refactor or rewrite depending on the circumstance.
Remove-tech-debt-as-you-go is how I approach the problem. Doing so ensures that removing tech debt is not done in a vacuum, disconnected from business needs.
My software work is centered around security and I find it almost impossible to persuade managers that security is worth anything.
The exception is when a major hacking event unfolds in the media. Then, suddenly everyone is attentive.
I suspect it is the same with militaries, except the USA and a few other countries. It is easy for politicians to cut military budgets and spend the money on civilians, and some intellectuals will even praise them as prioritizing peace, until suddenly tanks adorned with white letters roll over Ukrainian borders. Then, countries panic because their underfinanced military force is inadequate to the threat, which just became very real. (I am looking at you, Germany.)
Once the engineering team started thinking about business cases, they started wasting time they could be dedicating to engineering. I know where you come from, on a similar journey, but when things don’t work, adding things to do is the worst option.
1. If you don't properly evaluate the work and be selective with what you work on, actual engineering time will be allocated to too many projects. People spin their wheels working on potentially insignificant things, which is a morale killer. So thinking about the business cases was a way of reducing work, not adding work.
2. I tried to emphasize that thinking about business cases is something product and engineering management/leadership need to own, not necessarily IC engineers. ICs can add ideas to the backlog and identify opportunity, and it is the job of PMs and EMs to evaluate those ideas and sequence them for being worked on, relative to other projects. The feedback loop that is important to ICs is the impact of their work in terms of user metrics and goals the project was supposed to accomplish. I don't want to get too heartfelt about it (because Capitalism, Yay!), but there's a difference between churning out features into a black hole and knowing how your deliverables affect the people for whom you build things, and letting that continue to inform your work.
I think we are more aligned than what our exchange may superficially imply. But to clarify, from a value generation perspective, I don’t see tech debt as something whose payoff would generate any value other than in terms of technical flexibility and long term engineering resources availability. If a tech debt item is adding a feature then it’s not tech debt. Tech debt is born out of suboptimal support for features. So the feature is in place, but not in the proper technical way (I.e. against target architecture, held together with manual processes etc.). As such my approach with tech debt is realise the value by having cadence of paying tech debt off, not being afraid or thinking too much about the value of bringing things up to date. It’s more of a keep the engine running approach, at least in big engineering organisations where that makes sense from financial perspective as they resemble more a locomotive than a car in economics of restarting vs running ongoing. I am also confident both approaches may equally work or fail depending on organisation dynamics.
My analysis is that the problem is "tech debt is bad". Just like real debt, there are times when you sign your name to it on purpose because it's a good decision. For example, early on in a project when the requirements are fuzzy and undiscovered.
Like financial debt, you pay interest on tech debt, so you do want to strategically pay it down when the stars align, maybe you've recently put out some new features, and you can buy the political capital with customers (and by extension, management).
Debt isn't a bad analogy but an appreciation for the nuance would be nice, I don't commonly see that.
Technical debt is unavoidable, but letting technical debt grow without managing it is a great way to kill a project.
Explaining the benefits of paying down particlar pieces of technical debt is important, but another important piece of the puzzle is talking in advance with stakeholders about when, where and how much technical debt you want to take on to launch a product. When the existence of technical debt isn't talked about until it needs to be repaid and thus comes as a surprise, the conversation becomes a lot harder. Additionally many of the decisions about how much technical debt to take on to launch something can be improved by bringing stakeholders directly into the conversation, at least on a general level.
There’s a whiff of victim blaming here that bothers me.
When taking on tech debt you often get a lot of pillow talk. We have great reasons to do the wrong thing for the right reasons, but that is all forgotten after the crisis is over. Either immediately for those really bad at it, or when the next crisis hits.
I do agree though that when neither transparency nor awareness are good, there will be several crises in the middle that are triggered by the tech debt. I’ve seen that play out a couple of times, typically when I’ve joined a project in progress, and still building trust before I can start pointing out brutal facts.
Some developers are blind to different forms of tech debt. And as long as there is dissenting opinion the business will doubt any information that suggests they can’t have their cake and eat it too. I’ve known a handful of jaded engineers at least who, instead of being afraid or awed by titles, end up treating managers like willful children. It… does not yield consistently good results, but then again I don’t know anyone or any process that does. I’ve seen worse, and I’ve seen better.
> There’s a whiff of victim blaming here that bothers me.
Huh, what are you even talking about? What victim am I blaming for what?
I am sharing an important part of the technical debt management process that I don't see mentioned as frequently. Planning out where and when your organization should take on technical debt seems as important to me as planning out where and when your organization should pay down that debt.
Treating people like children is rarely a good approach and to me indicates someone who is either lazy or unwilling to take the effort to communicate well (edit: including when working with actual children).
If an organization can't resolve dissent effectively, that seems like a deeper underlying organizational disfunction that will cause all kinds of problems.
The truth is, the line for what is technical debt is kinda fuzzy and a there is a lot of other stuff that isn't really technical debt, but rather process or organizational debt.
The problem with "tech debt" is that in the heads of non-developers it is equated with financial debt. When you take on financial debt for a business it's a long-term investment (a buy-to-rent mortgage on a property, or a business loan to buy a factory plant for example). So OK, we're a startup that needs to find the right market, so we'll take on X amount of tech debt to get there, and then when we are in a better place we'll pay down the tech debt in installments of X amount per month.
Mostly what I've seen of "tech debt" has not been this kind of deliberate decision making of tradeoffs. It's more like:
1. Non-tech founders want to start a tech company.
2. Seeing the high salaries of developers they opt to hire their college roommate or relative.
3. Friend/relative is horribly inexperienced and builds a pile of crap.
4. Said pile of crap nevertheless gets market traction.
5. Customers complain of endless bugs & sales want long list of features.
6. Friend/relative is promoted to CTO. Time to hire experienced devs!
7. Experienced devs say "this is a pile of crap, throw it away and start over".
8. Founders tell devs no, you have to fix bugs and add features. Hop to it.
9. Developers have no leeway or political capital to fix the underlying problems, do what they can, and eventually leave.
So it's more akin to an 18 year old with a gambling habit who has just discovered payday loans.
At some point you have to decide if integrity means knowing the right thing to do and doing it, or knowing the right thing to do and asking permission to do it.
If you can’t get your coworkers on board, then they can make you look bad faster than the project will reach an intervention. If you can, you can stop having abstract conversations with managers about being Good and just start telling them that this is how long things take now.
Relentless refactoring is a set of skills that are interesting but difficult enough that people lose the plot. It’s really about amortizing all difficult work across all of the work, not just the difficult bits. This is actually a strategy from real-time systems, where maximum reaction time matters, and something that XP knew and Scrum does not. A thing I’ve noticed is that there’s often a lot of gaming by management once the engineers start to say “no” but haven’t mastered yet. The bargaining is expensive in time, energy, and good will (once people realize they’re trying to trick us into doing something we said was hard by going at it sideways).
When you add a day, two days, or week to all estimates, then the ratio between the smallest and largest stories decreases. The incentive to waste time trying to trick people into saying yes to something that violates the project triangle is drastically reduced. They won’t like it, but they weren’t liking anything else we tried to tell them so that’s not really a new situation. What’s new is that they see we’re serious and they don’t like it.
The counter-point here is that while you can argue about the name, there is a very real issue that poor architecture will slow progress not because it's "hard" to implement features, but because organizationally it becomes impossible to get them merged.
We have an endless parade of commentary on HN about what and how much should be in commits and pull requests and what have you, but the thing I've always found credulous about these ideas is how they survive contact with a code base where a "small" feature is going to have a very wide-ranging pattern of changes necessary to support it.
It doesn't take much before their are features you functionally can't implement, because there's no possible way to build a neat set of commits for them and get that through reviewers - particularly if there's a lot of activity, because the immediate effect is usually that you'll break a lot of other minor patches with the changes (or they keep breaking you).
I'm not sure though that I've seen anyone actually prioritize fixing technical debt in these terms though.
I am using a low code/no code tool for some back office services and we're investing into the enterprise version. The enterprise version is on-prem only. To migrate from cloud to on-prem, we have to export the cloud data to the on-prem service. Each "section" has a download to json button but there is no download all button. So we have to go to all 50+ sections, click download, upload to our on-prem server, and click in a few places to "reset" things.
The fact that these guys cannot build a migration button smells like major technical debt. These guys aren't stupid since they have built an incredible tool. Somewhere along the line, their product became fragmented enough, where it would be too costly to build a feature that would lower the barrier to entry for their enterprise service. We're talking about high 5 figure to a low 6 figure yearly contract. They're telling us we need to go click for an hour to do the migration.
So if you don't think this is technical debt, I'm curious as to your interpretation of the situation.
I've found myself holding a bit of chagrin at the term 'technical debt' lately as well, mostly because it feels like saying it, pointing to it, quantifying it, and shoving 'it' under people's noses and proving its there (not to mention the usual ceremonies of what I'd like to do about it) isn't doing me any favors anymore.
And at that point....
....I don't know that simply calling it something else is going to help.
Yes I'm a bit jaded, grab a drink, next one's on me
Begin by defining technical debt. Write down an actual definition for the phrase. Once you have a working definition that everyone agrees on you will find all your problems with the term will disappear. (Hint: good luck finding an agreed on definition).
Tech debt and refactoring are the two most overloaded terms in software development. Most often "tech debit" isn't - it is code that met the need when it was written, but the need changed later. Wear and tear seems much more appropriate.
And re-writing from scratch isn't refactoring, but that seems to be the most frequent usage of term these days.
> A long time ago I saw a speaker claim that it's not tech debt, it's wear and tear.
It sounds like you agree with the idea in general, but just don't like the current metaphor. Can you expand on what about this alternate metaphor fits the phenomenon better?
I can. Even if you design a good system up front, the requirements graph changes over time until your great ideas have turned into hindrances. This can be scaling issues, but mostly it's where reality turned out to be different than your abstractions were built for.
Maybe you made one module very generic but now it's cumbersome to add features to. Maybe you didn't add enough flexibility in another place which resulted in a buildup of suboptimal code. Maybe extreme concurrency was planned for but never needed and now the person has left and no one else understands how that part of the system works.
So now, over time, this emergent phenomenon that some call code rot causes your beautiful, elegant greenfield project to turn into a monster that no one wants to touch. It's hard to modify, easy to break, and takes forever to work on. It's legacy code.
I've begun to see tech debt as business processes that are too expensive to replace. You can either encapsulate the process with modern thinking or sell your company to someone who has investors willing to pay to create new processes.
Just swapping tooling shouldn't be considered tech debt if you can actually do it without breaking income.
There should be someone in the company, with visibility across the entire organization who says "yes" and "no" and explains why.
I've worked in places like this and the clarity is amazing and very motivating. The goals from the top down were 3 things. Everyone in the org could understand and apply it to their own work. Anything else was deprioritized. There was no confusion or swirl.
Lots of people piss on management as useless, but when done well, it can contribute a lot in terms of work-life balance, employees feeling motivated and organizational cohesion.
Think you nailed it by breaking it up by groups of people with different lists.
Literally every group/dept/team has a list. Someone/process has to arrange that into a single list for the group that builds. If you have more than one group that builds stuff, then more output lists and more complex the transformation from input lists to output lists.
Guess this is a simplistic summary of what PMs and Eng managers do.
This is one of the main things a good leader should be doing. Not just execs, but up and down the chain, one of the most valuable things a leader can do is get everyone pointed in the right direction. Make decisions about priorities, communicate those decisions to the team and the rest of the org, and then commit to getting them done. Ruthlessly prioritize and be willing to say no.
It's not actually easy, for a number of reasons. One is organizational. If your boss isn't doing the same then then it's going to be harder for you to do it. If you're responsible to multiple people who don't have any common chain of command then it's going to be really hard. And it involves saying "no" and standing your ground.
> Be really careful with the yellows: if there is low impact, you might be confusing work for progress. If there is low urgency, you might be overestimaging the value of your work.
On the flipside, if only high-urgency projects get done, you end up becoming a 500+ employee IT service provider without a proper Identity and Access Management System (IAM), without a Privileged Access Management system (PAM), a haphazard, ad-hoc secrets store and so on.
That's all fine and good for a 10-person company, you can get by like that when you're 100, above 200 maybe it starts to get icky...
Yes, I understand that if really nothing progresses, you have to prioritize ruthlessly, but you cannot keep doing only high urgency projects forever.
As someone who have gone through this there is one Achilles heel: appraisals.
Unless the management agrees in letter & spirit on prioritisation this is a disaster.
When everything is a priority the message from top is "We want best of both worlds" i.e. we want productivity, quality and less complaining.
Now you have prioritized in seq: A B C D... The team who has C or D to deliver will have there deliverables pushed by say 6 months. Guess who will be pissed-off with this?
But the team accountable for D will say that this is what was agreed and signed on but the top still had those flawed expectations back of their mind.
There is also cross-team impact. The product manager who was championing project D now has his/her appraisal in question. He/She will point fingers to the engineering and complain to his/her boss.
In other words: Every one in the chain of command has vested interest. Unless they are aligned to large extent its hard to get out of this situation.
This is a really valuable point and we faced a lot of ground-level resistance because of this. Teams and people were incentivized to care about the wrong things. Those things made sense in the past, but the structure and incentives of the engineering org did not keep up with the changing business needs. Again: not a knock on historic decisions (which made sense at the time), but a reflect of necessary change. Changing the incentive structure was implicit in the reorg we did and it was one of the most painful parts of the process.
That said, the reorg allowed for more effective parallelization by making teams more independent, and thereby avoiding the "I'm sitting here and unable to get anything done" problem.
But yes, if people are not in consensus, you'll have sabotage. The Lippit-Knoster model for managing complex change does a great job highlighting this.
The author seems well aware of that and wrote than all of this is only truly actionable if/when you're at a director-like level, and you can handle expectations from below and above.
No real comment other than damn, what a solid-gold piece of writing and advice. Felt like I was just nodding along to every other sentence. Awesome stuff.
I particularly liked breaking down “priority” into the intersection of urgency and impact.
I wonder how much that helps, obviously you can try to assign some expected value or maybe few different scenarios on how value might realize, but then you have two different knobs to play with.
If your organization has lots of politics you might be willing to "adjust" urgency of your projects by setting tighter than normal timelines or boost projected value such that your project gets chosen.
More simple low-mid-hi model doesn't give that detail so people understand that it's just a guess, not exact calculation.
If nothing else, the psychological difference is invaluable. "Priority" is irrational and often arbitrary. Impact means money. Urgency means timeline. These things can be put in numbers, even if inaccurate, this is why it's much easier to build consensus around them.
Regarding politics, the author mentioned aligning incentives in another comment. It seems generally necessary regardless of the method used.
This was a really great article and case study. The problems (and eventual solutions) will be familiar and unsurprising to readers of The Phoenix Project: https://itrevolution.com/the-phoenix-project/
If you only add features, you gain rigidity and fragility. To go fast you must go well.
Teams that balance maintainability against features have a competitive advantage and will beat you every time.
Going "fast" immediately slows you down. Right then and there. You think you're going fast but you're just fooling yourself. The DevOps Handbook goes into this - you need to spend 20% of the teams time on clean code and maintainability.
Not 19, not 21, 20. That's how you remove rigidity? Most places who apply handbooks end up applying handbooks to the detriments of everything else.
I feel we need to embrace the fact software is a fluid monster and we must work at it dynamically: cleaning up when it s unbearably crappy, adding to it when clients and users depend on it, and a balance between both when clients can wait a bit or when the dirt hasn't settled yet.
Then you d have a team that delivers rather than state that we must lose the client because his need overlap our 20% "moving classes around more logical namespaces" clean code.
> Then you d have a team that delivers rather than state that we must lose the client because his need overlap our 20% "moving classes around more logical namespaces" clean code.
Which sounds like something that happened zero times ever. The problem is rarely that there's too much tech debt work, it's that client requirements take over dev work right up until the product goes up in flames (at which point the client you meticulously built features for disappears along with the rest).
Since you're rejecting handbooks, how do you define "unbearably crappy?" Perhaps it's when the devs have already started looking for new jobs? And then, if it's unbearably crappy and you start cleaning up and a client comes with requirements, do you drop the cleanup work?
> Figuring out the impact and likelihood was something that a product management org would normally devote the bulk of its time towards.
Here’s a question: why weren’t they devoting the bulk of their time to it? I’ve been in sessions like this as an engineer and they end up being incredibly productive and informative for everyone as to what the priorities actually are. These aren’t things that engineers are supposed to be doing better than product managers.
Why do product orgs _constantly_ fall on their faces when it comes to prioritization? My guess is that the job is highly politics centric and PMs have a hard time getting out of the way of their own aspirations or ego to let the progress take precedence.
And no this isn’t trite PM bashing. I would like organizations to re-evaluate the incentive structures of product owners and make them look more like tech stack performance metrics (literally, as in how much is our software getting out of the way of our customers usage and allowing them to be “productive”, whatever that term may mean for the company). Also, give product owners some credit for backing off on new features to make headway for tech debt pay down.
Lots of really good points in here. I have worked at various companies who have struggled after the initial growth. Hubris can take over and we feel the need to over-extend, to sit at the big-boys table, to court the corporates with their fat (but very tight) wallets.
I think a common cause is simply fear. There is a lot at stake when you have 50+ employees. Maybe those one or two markets are not big enough to support what we need any more so instead of razor focus and discipline which helps, we try and build everything for everyone and a lot of it is simply wasted effort. We also start having teams so instead of a design decision being taken by the entire management (and can get shouted down), it just gets done because "the design team" regardless of the value it is actually adding or not.
In extremis, we see the FANG companies who have so many engineering staff, we can only hypothesize about what most of these people even do for 40 hours per week on massive salaries.
I wonder how much lost productivity comes from people saying "if only it had feature X" to salesmen to be polite instead of a simple "we're not interested and won't be buying".
It's the difference between being a feature factory and having a real product strategy.
If you actually have some kind of customer in mind, and you have ideas of problems that you're solving for them, then this can be valuable feedback even if it's a lie. Digging into the feature might expose you to a new customer problem that weren't aware of. You can talk to other customers and prospects and understand if this is a problem they have as well. You can take that information and extrapolate it into a real set of use cases, think about the size of the market of "customers who have this problem," and decide if it's something that makes sense to tackle with your product. Then you can design features to solve the real problem.
If you just have a feature factory then all you'll do is build the feature as the prospect specified. You'll go back, they'll still say no, and now you have no idea how to market this feature or if it would even be useful for anyone else or it's just built around the customer's internal process.
"The Mom Test" is an awesome look at that -- how bad questions and worse feedback leads to a lot of wasted effort. Highly recommend it. http://momtestbook.com/
OP here -- something I didn't touch on is that this is really common when a company transitions from early-stage to mid stage. The operations of a small company and the operations of a mid size company (and the operations of a large company) are completely different. The problems start to happen there is a mismatch between what you are and what you think you are. If you implement big-company practices for a small startup, you're often going to be overburdening the team and losing agility. When you are a med/large company but operating like a small company, you'll see chaos and inefficiency. Part of the problem is that there's no clear line. Observant engineering and product leaders tend to have an intuition for when practices need to change... I think team size and revenue/contract size (for B2B) are good heuristics. But more often than not, it's a stumble through the transition and folks feel the pain before figuring out the solution.
> Part of the problem is that there's no clear line.
This is what baffles/wonders me about the state of the art in building companies. It seems so anecdotal and ad-hoc. Case studies and good advice, while they exist, are from "previous generations" and "we do it differently" because "culture". It feels like society has to continuously reinvent solutions to managing people and projects rather than building upon proven and effective strategies. I guess, perhaps, I am betraying my own lack of knowledge of this domain. Maybe there are tried and true tactics and I've just never experienced them. Related, it really feels like there should be a sub-field of "sociology" set on improving the efficiency of groups of humans using objective methodology that yields consistent and reliable results...
> it feels like there should be a sub-field of “sociology” set on improving the efficiency groups of humans
Yes there is. Its called organisational behaviour. Most b-schools have some faculty specialising in it. Cornell,I think, has a whole department in the school of Industrial and Labor Relations. They’re pretty good.
Related to this, and something I should write up in more detail one day: Priority is an order.
"High", "Medium", "Low" are not a priority, they are useless.
A priority is an ordering of tasks. Once you free yourself from trying to assign labels to the priority then you free stakeholders from worrying about putting things "very high" in case they tread on toes and you enable an honest conversation about where it belongs in the list of tasks or projects that need finishing.
After all, no-one ever marks things as "low" priority because they understand it'll never get done, and no-one ever marks things as "high" priority because they worry about the politics of doing so.
The result is you get "medium", "Just above medium", "Just above just above medium" and "JAJAM+" in your JIRA. (True story! And a shout out to anyone recognising JAJAM+).
A lot of this is, IMO, very basic - a project is not done until it’s tested and running in prod - I am surprised anywhere with engineers with any experience at all would act otherwise.
I don’t think having 40 people work on one thing at once is good at all. It’s like solving scaling by just getting a faster machine. Eventually you may not be able to purchase a machine fast enough to do everything you need to do… And you’re going to have a ton of deadweight.
I hate “agile” as practiced but, in theory, it solves this. If you scope out a project well, you should have a general sense for what cross-team dependencies exist and be able to put it on the other team’s plate. You need a decent amount of “trust” but ideally the other team will actually appropriately prioritize it, ie if it takes only two days of work to unblock a project with big business impact, they won’t push back that their Epic Rewrite v3 is too important and maybe they’ll get to it in 6 months.
The biggest problem I’ve seen is with bucketing all projects in long planning cycles (3 months, 6 months, 12 months, etc.). Every cross team dependency then takes about one planning cycle to be accomplished, and unexpected additional work/dependencies become nightmares. Also, while “ownership” and specialization are good to an extent, fixing a particular engineer to a specific domain over long intervals almost always causes problems because X% of engineers either aren’t skilled enough to be able to handle owning a domain, leave the team, get delayed, etc. If everybody is fully booked for a long time, there isn’t slack to help them, and it suddenly becomes a Big Deal to assign someone off a domain (because the person struggles for months and messes up a big project, rather than identifying they are messing up after weeks and course-correcting by putting them on something they can be more productive working on), and every project with more than a few people gets delayed because there’s no way to fix one or two engineers on the critical path dropping the ball.
What both this article and agile touch on is the benefit of having a flexible planning structure and not over-committing or trying to plan too far in advance. IMO, that is the key to getting shit done. I think you can do that without taking it to the extreme of having only one project in-flight though.
To those who are interviewing these days for a leadership position, the article has details for answering questions related to managing conflicting\multiple priorities
It's company wide - many things which aren't the product still require or affect engineerinig.
I'd bet that almost every company with 40 engineers has 300+ projects. Every company I ever worked at tried to do waaaayyyyy too much at once. It might be the single biggest problem in business. Steve Jobs is lauded for his design skills, but I'd guess it was the focus thing which made the difference. Warren Buffett similar. Bill Gates too. They have all commented on it, and it rings true to anyone who has spent time in a few businesses.
My gut reaction to the title of the post - fire 80% of the managers. Then announce skimpy buyouts for any employee who feels that they won't get enough face time with managers and/or time spent in meetings with the new manager/worker ratio.
> This routinely caused the team’s focus to change to whatever was the problem of the week (or to solve for whomever was yelling loudest).
My mind immediately goes to a lack of leadership and strategy. Why do we have pirates going around from team to team yelling, inventing new feature requests?
A lot of what I can read here sounds like someone (goopthink) finally took ownership and seized authority from a leadership team that had rotated out twice and had mostly given up.
> we had cycled through two previous heads of product and one head of engineering
> In fact, I volunteered for the role because no one wanted to take on a project
> The problem is that you need a conflict resolution and effort coordination mechanism
> Figuring out the impact and likelihood was something that a product management org would normally devote the bulk of its time towards. Not having that information was one of the failures of process we were dealing with
> adjusting the stack rank continued to be a deeply political process even after it was created
> “you’re a team of 40 people… surely you can do more than one thing at a time?”
> There was no way to scale those conversations. These 1:1s also allowed us to get ahead of influential senior managers who could have been saboteurs if they didn’t buy in.
This whole thing screams that there is no acting leadership in the sense of building a team focused on one goal. Allowing many managers to go around and create new business lines, not holding them accountable to drive outcomes but rather just be busy with some new idea. The issue your engineers have is identical to the one going on outside of engineering: no unified mission, nobody willing to say "no" to "some new idea that could totally make us some money and we should just try it and see".
It is as if the executive allowed a rabble of random managers to do whatever they wanted like gladiators in a combat arena. It's like CV driven development: "hey let's use kafka and http2 for our internal CRUD app" but instead it is like "hey let's start a new division of the business that has an entirely different workflow but exists in the same industry vertical".
The leadership team is absent from every point of this story and things could only improve when they got out of the way or were engineered out with pre-messaging in 1:1s.
Good that OP could improve all of this (which in itself is a huge accomplishment, something like 60% of change management efforts fail) but honestly until that L-team is rotated out entirely the problem isn"t solved, those pirates will go back on their habits that created a dysfunctional org full of "yeah but I have a cool idea" leaders. These leaders tend to want all of the decisions and none of the details, making everyone live in their mess. OP dove head-first into the details and this is why they could fix things.
* Technical staff in an organization feel they have no authority or respect within the organization. This can happen when companies become too sales-driven.
* Senior technical staff feel they have no ability to delegate, either because of the previous point or because they don't trust the technical ability of less senior staff.
* Those in an organization who do have authority to delegate lack sufficient understanding of priorities or lack technical knowledge.
These things highlight the classic balance between technical knowledge and managerial skills. If you don't have people that bring both to the table, you're in for trouble. Not sure if I'd go so far as to say this is what it's all about. However, reading over the author's advice, I thought to myself, "None of this could have happened at some places I've worked." You can come up with a good plan, but if no one will listen to you or take your advice, it's worthless.
The author probably was that person who brought what was needed to the table. As such, perhaps the plan is not what ended up mattering so much as having someone who was willing and able to push it through. Also, having the right people from top to bottom. My feeling sometimes is that effective teams occur almost randomly. This might be because organizations rarely have the patience or the resources to assemble them deliberately.