Part of the problem is that software is never really "done" -- older software needs a ton of maintenance and support, and new features are restricted by existing code's technology choices, design decisions and architecture.
So pennies in later systems never fall onto the truck and get delivered to the customer, rather they fall into the final state and randomly teleport back into earlier stages as teams that implemented them start getting support work.
Unfortunately you're not going to be as productive when you're developing a new product against a background of supporting multiple existing products / customers, as you were when you were developing your startup's initial product without much else going on.
> Part of the problem is that software is never really "done"
Absolutely. The cognitive dissonance is bad enough that we might be overdue for a change in vocabulary, starting with the word "done". Even hypothetically perfect code needs to get ported to a new architecture, OS, VM, and/or compiler at some point.
I think we, as technical experts, do a terrible job explaining that every piece of code old and new has an ongoing cost. The discussion points should be about which costs are higher and whether trade-offs are worth it. Are the ongoing costs (is the team even aware of what they are?) lower than the cost of change (how do we know it was worth it? can we change our mind if it wasn't?).
While the article brings up many good points about bottlenecks, estimation, and allocation of resources, I find that bad mental models of how software development works costs far more, albeit in subtler ways, than "Heather can do 12 units of work but we only gave her 8 and she blocked a lot on task Alpha instead of moving on to something else".
Continuing that line of thinking, one of my epiphany moments when I first arrived at a dominant player was the acceptance that we would ship with bugs.
The way to stay at the front of the race is to prioritize your bugs and fix the right ones in the time that you have because if you think you'll get them all, you'll never ship. Get the ones that have the highest probability of causing your customers pain and know that you aren't ignoring the ones that are scheduled for your next maintenance release. Debate intelligently if you think something needs to be reclassified as a higher priority and learn to accept that there might be a couple that you don't win so you can get on with covering as much of what remains as possible.
> we, as technical experts, do a terrible job explaining that every piece of code old and new has an ongoing cost.
It's always hard to find good metaphors for programming programming as a craft, but I liken it to a factory or campus.
Even when it is "built" and producing value, there are always issues with maintenance and renovation and replacement.
"Technical debt" isn't just maintenance costs, but the restrictions placed on what you can do next. Ran an electrical line straight across the lawn? Now you have to black-out a building to progress.
that is why software development is not engineering (yet).
constructional engineering has the benefit of clear physical limits, you cannot fiddle with a building after its release too much - add more stuff on top and it will crash down. people get that, by now.
same with software - everything gets added in, email, chat, video, whatever. then it becomes a bloated mess, and a new product, lean and simple, comes along. rinse, repeat. they will ruin Slack, for example, guaranteed, and not even that long from now.
the thinking of constraints, of boundaries of growth, has not been established in software yet. UNIX had it, but that mindset got lost as moore's law kicked in. but the constraint is the human mind, groking what the fuck those millions line of code actually do.
Software can be "done". It depends on the software. Any embedded software that's not field-upgradable is done when the device ships. To take an extreme example, the software on a space probe that gets launched into space and cannot update itself definitely needs to have a "done" date.
> Software can be "done".... Any embedded software that's not field-upgradable is done when the device ships.
That's a perfect example of an that proves the rule.
Saying that, even within that ecosystem, there are libraries, ports, oops-it-blew-up patches, builds against emulators because the hardware isn't ready, etc. You don't write a new codebase from scratch for every device you ship. Parts of that "done" code end up in next year's model.
> the software on a space probe that gets launched into space and cannot update itself definitely needs to have a "done" date.
Actually, those get updates too. As scary an idea as that is. NASA engineers are some crazy folk.
But I don't disagree that software can be done. It depends on the software, and the supporting ecosystem just as much. A microwave doesn't need or get updates, but it's also not networked, or rendering across different browsers, or needing to support a lot of users.
But I think in the larger sense, done is relative. There is always a vision, and growth. And deadlines. This isn't such a problem so long as you are always working at making it better. But if all you ever do is ship features and never address the underlying design... good luck.
* > But I don't disagree that software can be done. It depends on the software, and the supporting ecosystem just as much. A microwave doesn't need or get updates, but it's also not networked, or rendering across different browsers, or needing to support a lot of users.*
That's for the copy of the software deployed on a specific microwave. Presumably though the team that wrote that code are doing some ongoing work adding new features and fixing bugs for new model revisions and brand new models. As well as changes to keep their tooling and development environments up to date or at at least supportable.
Wouldn't this ongoing support requirement in software be analogous to customer service/maintenance departments at an auto dealer, or the customer service hotline for other manufacturing products? I wouldn't say this problem is unique for software.
Add to that the problem that "productive" is not the real goal. Businesses can survive and even thrive with sub-optimal processes as long as those processes deliver value that exceeds their cost. (We see this in early stage startups, i.e. "Do things that don't scale".) Needing a development process to run optimally in order to deliver value suggests that the marginal value may be far too small for a custom software solution.
No! No! No! No! WHY??
We spend years saying "software development is not manufacturing", and now it the name of "agile" we allow all this to crept back.
Your team is not falling behind, your estimates were wrong!!
That response is ineffective, because the managers will simply pressure the engineers to give better estimates.
The penny game (where I work we call it the beer game) is a good learning exercise, but in software development you really don't want to be having different "stations" handing work off to each other in the first place, because that leads to delays and quality issues. Each station will develop its own little backlog and pretty soon a feature cannot make it from start to finish without spending weeks waiting in the design queue, then the dev queue, then the test queue...
Instead, you need to have a small, cross-functional team focusing on the same piece of the product at the same time, collaborating and giving rapid feedback on each other's work--that's what agile and XP and scrum are for. And you need a pull system instead of a push system, to regulate the amount of work in progress--that's what kanban boards are for.
But doesn't specialization naturally push towards the creation of stations? If your cross-functional team contains both a database engineer and a UX designer, I don't see how either can help with the other's job.
I find it's helpful to at least be aware of what is going on with other 'specialists'. That way you can hopefully get some mental (or physical!) prep done long before it gets to you.
It really can help if they are actively communicating with each other--the user interface is how the data gets into the system, so if the database engineer understands the interface it can provide context for where the data comes from, what it means and how it is used, and that can inform data modeling decisions.
Understanding how the data model is laid out, how much structure is needed, what fields are required/optional and what types they are etc. can also inform UI/UX design decisions.
And this is an example of two roles that are at opposite ends of the frontend/backend spectrum. For roles whose work products interface directly, there's an even stronger need to collaborate.
Specialization is good in that it allows people to focus on using their best skills, but it's bad if it fragments the product development team and inhibits communication and decisionmaking.
Certainly active communication and the prevention of siloing can help speed development and reduce defects. But the Penny Game illustrates that backlogs and queuing issues will occur even if each station is in perfect communication with all the others. So you can't eliminate queuing issues with close collaboration and rapid feedback. You can only eliminate it by eliminating dependencies and specialization, which I don't think is possible in modern software development.
Well yes, on a basic level when you have specialization and division of labor, you will always have dependencies.
But when you tighten the feedback loop and shrink the batch size of the work, you make the dependencies smaller and more granular, so the queue gets smaller and fluctuates less, and you get better flow of work through the process.
Here's a simple visual that illustrates why this is the case:
And yet we have existence proofs that Kanban works, in the sense that it allows many teams to quickly deliver value to customers. So what don't you like about it? We shouldn't reject innovations out of hand just because they happen to originate in manufacturing.
I've usually assumed that when people say "software development is not manufacturing" they usually are referring to the "Western" model of manufacturing that the first half Goldratt's book Critical Chain spends tearing apart.
There are fluctuations to be sure, but I think you can compare against estimates under two different methodologies and see whether you seem to perform better under one or another. (Of course you have to weigh the benefit of such measurements against the costs of making those estimates).
To my mind the costs of context switching are pretty well established at this point and the value of limiting work in progress is clear enough from experience, but that's just my experience and maybe you disagree.
I don't disagree with the cost of context switch and I see value in limiting the work in progress, my argument is against focusing on throughput and measuring it by units of work, and particularly using estimates as the base of the measure
> my argument is against focusing on throughput and measuring it by units of work, and particularly using estimates as the base of the measure
That's not what I'm advocating. All I meant to say is that when you have a lot of work in progress you get less done, whether or not you have deadlines.
No, I know about pull, Toyota, Lean, and all that.
Still manufacturing and production of repeatable widgets...
Will you apply the same approach to make a bunch of mathematicians prove a theorem?
Most computer programming (in the private sector) is not research. It is product development. It is much more like designing and prototyping a physical product like an automobile, than it is like proving a theorem.
The number of times I've had someone bring up a feature at 9:02 am (usually on the phone with six other people, of course) and wonder why I won't "ballpark it" at 9:08 is mind boggling.
Having been both a Manufacturing Engineer and a Software Engineer, there are similarities between the two scenarios: work-in-process within a manufacturing facility and whatever features/functionality a given software development department is creating.
I would create detailed step-by-step job instructions for each station in the manufacturing line, along with expected times each unit should take at each station. It was trivial to compare expected versus actual, then dive into the details of what's going wrong if the two values are not matching up (new vendor causing quality issues? machine troubles?).
With software, those "job instructions" are little more than the specification hammered out by (hopefully) someone who knows the product, domain, technical capabilities of the software development team (including design and QA). It's rare. Very Rare....any single person can estimate exactly how long a sprint or anything larger is going to take. One wrong technical hurdle, and every estimate is out the window.
For the "production worker" on the software dev team, just following those specs wont get them to the finished product. Their knowledge, how well the team can support lack of expertise, shifting priorities...these all create logjams.
The difference is, writing code IS the specification process. The code itself is a functional design spec.
The persistent assumption that writing code can be handled like a production process is flat-out wrong. The software equivalent to production is the build/test/deploy phase which any intelligent developer will automate.
The concept of a 'production worker' programmer seems ridiculous. Somebody more experienced researches the domain, comes up with solutions, specifies all those solutions, and the 'production worker' programmer is left to to what exactly? The engineering equivalent is handing a draftsman a hand written drawing and telling them to implement it in CAD. It's good for practice and to get new people up to speed with the process but it doesn't provide much value.
The reality is, unless the team is using an easily repeated process like churning out Wordpress sites from a scaffold generator, there's no such thing as 'job instructions' on a programming team. Each and every developer on the team is assessing the domain and trying to hammer out a specification for the production process. Any part of the process that's easily repeated should -- and ideally --will be automated away.
It's not an easy process to measure because specification requires extensive research and discovery. Bad assumptions are made and corrected. Some approaches require new and novel approaches. A significant amount of the work comes with a great deal of risk.
Throw a few programmers into the mix who lack the battle hardened experience and wisdom of previous successes/failures and they'll generally drag everybody else down.
In short, software development is not a production line. It's hard to define because it requires a lot of unknowns and even more unknown unknowns.
my problem is that all the manufacturing techniques I saw being attempted in software development assume each "unit" of production is somewhat equivalent but in my experience thats a very rare occurrence.
The book Principles of Product Development Flow by Don Reinertson has an excellent chapter on queueing theory, giving an overview of the math behind it and how it applies to the development process.
Perhaps the main problem is treating a team of professionals like an assembly line. Execs and PM dream up the idea, throw it over the fence to design who mocks it out, who throws it to the backend to build the api, who throws it to the front end to piece together. What could go wrong with this process? Oh, maybe you have a meeting so people get their input in, but the problem is you don’t really know the important parts you are missing until you start building. This isn’t the way to organize an elite team. Good people don’t want to be a cog in a machine.
Not to be confused with Penney's game, which is a different kind of fascinating thing in itself: if two players each pick a sequence of heads and tails and win if their sequence shows up first, the player who picks the second sequence after seeing the first sequence has an advantage.
For a more detailed explanation of the importance of minimizing work in progress (i.e. inventory) I highly recommend "The Principles of Product Development Flow" by Donald G. Reinertsen. It contains a thorough economic analysis based on queuing theory.
This is a fun game because it also is related to a lot of counterintuitive things about constraints and premature optimization.
If you optimize something that isn't a constraint, the problem isn't just that you are wasting effort - it's that you might also be unwitting increasing the backlog of another part of the system. So it's not only "not helpful", it could sometimes even be harmful.
That's why Goldratt et al emphasize constraint analysis so much - analyze the entire system (usually using causal analysis; why certain parts of the system explain other symptoms), identify the current core constraint, invest all spare efforts to alleviate that constraint only (and not other inefficiencies), and then re-analyze.
I don't believe this is necessarily counter to Agile! It's just that a lot of people implement Agile badly. Which isn't a problem with Agile exactly, it's just that there's a lot about complex systems that are counterintuitive. You can't really be reductionist about it.
This is something that more managers need to understand - that even slight variations in how long it takes for work to be produced can have a major effect on throughput.
The first thing that comes to mind is to allow different stations to do each others' jobs when needed. Sure, productivity will be lower when doing something you aren't used to, but isn't it strictly better than being idle because there's no work on your level? (Barring problems because you didn't do the work well, but this is a simplistic model anyway.)
I wonder how the game would end up if stations could work on other levels for a slight penalty to production for both stations, only kicking in when one is idle over some threshold.
I think that's what SCRUM tries to achieve. A lot of tasks in TODO? People start working on these. A lot of tasks in QA? People work on that. Tasks get done but not deployed? People will start to cooperate on deployment. Each one is less productive but the whole team overcomes more of these big scale threshholds.
The different stations don't represent developers, they represent discrete tasks. So that could be designer > coder > qa, or dev stack 1 > dev stack 2 > dev stack 3. A coder waiting for a design won't just go and design. A developer with Stack 3 might be familiar with Stack 2 but might not know the best practices from that stack, introducing bugs or funky code that will create more work later on. If all your developers can do all the tasks, the graphics in this story don't apply.
As always: communication overhead. This developer coming in to a new thing after finishing their last thing has to spin up on all sorts of stuff, maybe they don't even know much about what we're working on, but even if they do, they don't know what the current status is. All that spin-up has cost both for the developer and the new team. It is often more efficient to spend that idle time working on lower-priority tasks in their current area.
You can't just break development tasks into subtasks at will, they always have a natural structure. Also, you can't go reassigning developers all the time, they should always do their tasks from start to completion or their productivity suffers badly. Neither can you assign several developers to the same task and get a productivity improvement.
I think this pretty much cuts to the heart of what's usually wrong with the whole "agile" thing -- by micromanaging programmers to such a degree, instead of being able to use that downtime to do things that practically everyone knows needs to get done, you have people sitting on their hands because everything has to go through the "backlog."
IMO the solution is to stop treating programmers like cogs in a machine, and actually give them some amount of autonomy to make decisions.
On my agile team, we will say, "We see a problem here." We create an issue and start working on it, and alert the product owner as to why it is a problem and needs to take priority. We have that respect because we deliver working software. We are not the "fastest" team, but we get the latitude to make things better because we work with adults.
We refactor as we go, even if the business value could have been delivered "faster" by compromising long term value for short term goals.
But we also let them do their job and trust them to understand the strategic goals to the business better than we do.
Yeah, I recognize there's some places that do agile in a good way (hence the qualifiers I put in); I've just been burned enough that I tend to be deeply skeptical of it. What I've noticed is that some managers fall in love with charts and measurements (because they can use those metrics to impress others in the organization), and anything that can't be superficially measured ends up either being devalued or outright banned.
I remember one place I worked, they brought in consultants to "help" us create an agile process. After about four months or so, they declared the operation a success because the "velocity" had increased by 4x. (Which would be incredible if it were true). In reality, we were just measuring four times more things, and development was going much slower, but those charts sure looked nice.
> instead of being able to use that downtime to do things that practically everyone knows needs to get done, you have people sitting on their hands because everything has to go through the "backlog."
Oh man, this happened at one of my stricter scrum managed companies. There was a bug that people were noticing that should have meant that no other cards could pass QA.
I recognized it as the small change that it was, made the branch, and had a peer ready to merge it, but got "a talking to" from the scrum master and product managers because they never recognized it as a bug.
A similar example regarding quality is Dr Deming's famous Red Bead Experiment. It is a fascinating insight into how dysfunctional management systems operate and perpetuate.
I like the simplicity of the simulation. My take-away is that each station needs to plan for average throughput, not for capacity.
The problem is that you don't know what average throughput is until you run the system in production. So you might have either too much, or too little capacity.
It's been quite a few years since I read The Goal. I can't remember now if I found the whole family narrative interesting and it kept me engaged in between the operations education, or if I found it a distraction.
So pennies in later systems never fall onto the truck and get delivered to the customer, rather they fall into the final state and randomly teleport back into earlier stages as teams that implemented them start getting support work.
Unfortunately you're not going to be as productive when you're developing a new product against a background of supporting multiple existing products / customers, as you were when you were developing your startup's initial product without much else going on.