That is hard to say and it would be a book worthy to explain what could come next. All I know is that it is unsustainable.
The complexity is simply too high and it's not getting better. One of the reasons I think why you see the fail fast movement be so successful.
Once you accept that failure is part of the process, once you abandon the "zero mistake" policy that many large organizations instill internally and externally you will begin to approach projects differently.
The truth is that "zero mistake" organizations make as many mistakes as everyone else, they just have the financial strength to ignore them as long as economy of scale works in their favor.
I could write forever about projects that went wrong not because the developers where bad but because the premise that fuels product development is broken.
I blame primarily business schools and large parts of academia for this. But it could extend all the way into the way the stock market is structured.
If you buy my premise that post-industrial is different than industrial age. That project definition is primary and time is secondary today. Then it does put some doubt at least in me about whether the stock markets focus on growth and Q's is sustainable.
Nature seems to be doing a good job as pacing various processes. It takes nine months to give birth to a child. One cell at a time. But the process is ongoing.
Nature is the ultimate continues deployment strategy.
It sounds like you've thought deeply about this - have you written more on your ideas here elsewhere? Would be interested to hear how this would work practically as well as your ideas on how the focus on perpetual growth hinders successful project delivery.
I'd love to see you write more on this as well. You've clearly thought about it prior to this conversation in a way that I don't think a lot of us have.
I think you will find reading Hayek and Coase to be rewarding.
Hayek introduced the idea that no one individual planner can make flawless plans because the information to do so is widely dispersed; instead success or failure in the market gives rapid feedback about how to allocate resources. Indeed the market is a discovery process that unearths this information in a way that a planner never could[1].
Coase asked the question: if this is so, how do large firms emerge? He identified the cost of transactions to be the key. The higher the transaction costs, the higher the cost of loosely coupled economies. The size of the firm is thus based on the relative costs of transactions (searching, identifying, validating, negotiating etc) vs the inevitable waste caused by planning.
In essence, large companies are islands of command economics in the sea of the market.
[1] In a related argument, Mises said that even if a planner could know all the variables the resulting problem would simply be too large to solve rationally.
So things like prototyping, UML, use cases, and Agile were not a "serious attempt" to answer that? I think it's also unwise to tell your manager "it'll be done when it's done."
Largely those are things done by developers, not by the business. Perhaps in cases where the business is on board with them (eg. Agile and rapid iterations), but mostly they'd rather handwave and hope for the best.
Instead of replacing estimation, I would say that you should embrace the problem in estimates - uncertainty.
Convey the uncertainty you have about a task to those you work with, and then you can start to factor in the risks of uncertain tasks.
I build project management software for a living at LiquidPlanner. Everything we do be it development, design, or marketing is based on ranged estimates. We don't always get the estimate right, but that's the beauty of a range, I it takes into account the fact that you will miss it some of the time.
The other thing that can replace an estimate is... lots of estimates. If you're planning something, update your estimates as you gain more knowledge about the problem.
I used to work for big oil, and they would have 100-200 million dollar projects. Not software projects, actually building plants. Similar to software development, these were basically prototypes. Yes a lot of the technology was known, but they weren't exactly building cookie-cutter houses.
So the first round of estimates would be something crazy like +/- 80%, both money and timeframe.
If that seemed good, then they'ed fork over the money to get more specifications and more details, and come up with a new estimate. Maybe +/- 50%. Then they'd re-evaluate the viability of the project. And do that a few more times, spending more money and time each iteration, before they actually committed to the project or dumped it. At that point, the estimates were quite accurate.
So basically, the software version. How long will that feature take?
An experienced developer can spitball an answer. About a week. Then they start working on it. After a day-and-a-half, they're going to have a better idea if that's actually a three day project or a two-weeker or if it ain't gonna happen.
Or maybe they need a day-or-two or a week to do a spike to be able to provide that estimate. All well and good if you let them do the spike. Not so good if you demand an estimate when the developer has clearly said he has no idea.
I absolutely agree that interval arithmetic is the best way to deal with this, accountants and their budget systems always require estimates to be reduced to fix figures.
