
Requirements volatility is the core problem of software engineering - luord
https://stackoverflow.blog/2020/02/20/requirements-volatility-is-the-core-problem-of-software-engineering/
======
rossdavidh
I am reminded of how linguists, when trying to document a language from one of
the last living native speakers, don't just ask them to tell them the rules of
it. They have to task questions like "how would you say this?" Then they try
to reverse-engineer the grammar, syntax, etc.

In most cases, the people who are giving the requirements either: 1) don't
actually know them (e.g. they are in upper management but the software will be
used by line workers) 2) have internalized the details of the process so long
ago that they are essentially subconscious now. They would tell you the real
requirements if they could, but they just can't, they will instead tell you
what they think the requirements are. Then, once you build it and they try to
use it, like the native speaker of a language hearing you speak a sentence in
it using the grammar rules they just told you, they will immediately know that
it's wrong.

So, the best way to find the _real_ requirements for the software, is to build
the software, and have them try to use it. Stop planning and endlessly
documenting someone's fantasy or best guess at what the requirements are; the
only way to really discover the requirements is to build the thing.

~~~
cecilpl2
Here's another good example: what is the correct way to order multiple
adjectives in English? If you are a native speaker, you probably don't have
any idea, you just know that you say "an old big brown cardboard fridge box",
rather than "a cardboard brown fridge old big box". The order is very specific
and any deviation is obviously incorrect. Try it for yourself!

But any ESL student can tell you the order is quantity, opinion, size, age,
shape, color, origin, material, purpose. We'd had to reverse-engineer the
rules, and then teach it to newcomers.

~~~
tigerlily
Then it must be "big old brown cardboard fridge box".

~~~
__blockcipher__
Incidentally that’s exactly how I would have worded the sentence as a native
speaker. But only because “big ol’” is a common phrase by itself

~~~
funnybeam
To me, also a native speaker (UK if it makes a difference), “big old” implies
size alone(fairly large) rather than age, while “old big” would definitely
mean old and large

------
kabes
I often hear people say that software engineering is a joke compared to civil
engineering or other kinds. But they forget that when a civil engineer designs
a bridge he doesn't start with: "We don't know exactly what weight it should
support yet, but just start building the bridge already". And he doesn't
finish with a: "Ah, turns out we wanted a tunnel instead of a bridge, can we
change that next sprint?".

~~~
clarry
> I often hear people say that software engineering is a joke compared to
> civil engineering ...

I think your example points out exactly why some think software engineering is
a joke: they rush ass first to implement something (and they implement the
wrong thing) instead of sitting down and figuring out the _actual_
requirements first like any real engineering job would have you do.

And it looks like everyone is pushing it more into this ass-first direction
with agile. Let's not even try to plan ahead! You know what you want next
week, so this short-sighted focus leads to chaotic development and once the
real requirements are figured out 9 months into the project, well too bad it's
too late to redesign and rewrite it all properly. So it's forever going to be
a bumpy and crooked bridge on the verge of collapse, held together by more and
more duck tape, hot glue and occult ceremony. And there's a plastic tunnel
hanging underneath it. Only the brave will crawl through.

~~~
markus_zhang
Manager and Director are the people who are pushing "agile". They want quick
results, and SE can only do what they want. You simply really can't "gather"
requirement and finish a feature and do proper testing in a 2-week sprint.

I do agree there are a lot of bad practices in SE as well, but let's not blame
everything on them.

~~~
clarry
Yeah I'm not really blaming individual software engineers (I'm in the same
boat, fighting a slow trickle of feature after feature without really being
given a chance to step back and consider the overall design & its future
evolution), but the industry as a whole. As I said in another comment in this
thread, all too often SEs are just tasked with cooking sh*t to vague spec (as
laid out by sales & other non-engineers) instead of being active at the
earlier stages where engineering should already begin.

It's like sales people together with the customer decided the shape and colour
of the bridge, as well as the number of pillars it has, and how many lanes it
has, and what material it's made of -- here's the stack of things you build
with -- and what other fancy features it has, and now finally we can call the
engineer so they can hop on a bulldozer and start engineering with their
hammer for the next two weeks and show the result so we can adjust the shape
if it's not as pretty as we imagined. Actual engineering didn't happen.

------
asplake
> Requirements change. Every software engineering project will face this hard
> problem at some point.

> There is really only one answer to this: you need to find a way to match the
> requirements-development-delivery cycle to the rate at which the
> requirements change.

Honestly, I believe that much of the problem is with the word “requirements”
and I so wish we could kill this use of the word. Bottom line, for systems
built for humans, ploughing through backlogs of requirements is the path to
mediocracy (or worse) – as the OP discovered.

A less misleading framing: you need to find a way to match the delivery cycle
to the rate at which understanding changes. In other words, your development
process is a learning process.

~~~
artsyca
100% agree learning is the core problem people in large organizations really
don't know how to learn and the typical corporate environment is the perfect
quasi academic environment geared towards pursuing metrics that amount to
chasing grades

------
jokoon
This quote is old and explain very well why early software was often high
quality:

> “Walking on water and developing software from a specification are easy if
> both are frozen.”

― Edward V. Berard

The more you plan ahead and the less you change goals while writing software,
the less problems you have.

Software is the only field of engineering where you can change things at any
point in time. But it doesn't mean you HAVE to change things. Software should
really take lesson from other fields of engineering. Companies always want to
cut costs, and with software, they really can.

~~~
BlueTemplar
The ongoing discussion in the EU about a legal obligation of X years of update
support comes to mind...

------
bostik
The article mentions how one part of change is that regulations may have
changed.

I work in a heavily regulated industry, and am dealing with a set of
regulators nearly all who depend on just one particular auditing company to
... effectively write their regulations for them. To the point where a new
jurisdiction happily refers to their draft regulations as documents prefixed
with the auditing firm's name on them. (This is not a joke or hyperbole.)

That particular company has trained their own workforce to evaluate acceptance
criteria according to their internal workflows and practices. This in turn
means that almost _all_ industry regulations are geared not towards
functionality, but to make sure the auditing company has the least amount of
training to do. As far as I am concerned, this is a particularly perverse form
of regulatory capture: it's not even the industry at large writing their
regulations, but an already entrenched gatekeeper acting on _their_ own
interest, crippling both the regulators and the companies in the industry.

Now, I have no problem with regulations as such, because for most parts the
rules tend to make good sense. But I do have a problem with regulations shaped
after the desires of this particular auditing company. Instead of focusing on
results, quality, transparency, accuracy, recoverability and functionality,
they have a fixated view on particularly dubious change management processes.

I am _convinced_ that at some point in history, someone has done extensive
research into 70 years of software engineering best practices - and gone to
extreme lengths to expressly reject as many of them as possible.

Two examples: they insist that even unit tests _MUST NOT_ be written by the
engineers who work on the code; and they are mortally afraid of the idea that
engineers who work on a problem could ever _understand and evaluate_ the
problem. (Not to mention that the very concept of a workflow tool is
completely alien to them.)

The end result is that you can either try to do software well, or you can try
to do it according to regulations.

~~~
BlueTemplar
Understanding and evaluating the problem is the main job of an engineer?!?
(Compared to a technician...)

~~~
bostik
It's definitely part of the job. If you don't understand what the problem is
that you're trying to solve, how can you verify that the constraints you are
working with actually make sense?

------
fouc
Greg Young described a good way to deal with requirements volatility: optimize
from the beginning to be able to delete your code, and structure your code so
that any part of it is no bigger than 1 week's worth of coding. So that any
part can be re-written in 1 week.

[https://vimeo.com/108441214](https://vimeo.com/108441214)

~~~
StreakyCobra
That was an insightful talk, thanks for sharing!

------
EliRivers
The only software project I ever worked on in which the customer didn't report
so much as a single bug had the requirements done properly, up front, and
agreed all round. Of the thousands of identified requirements, I think less
than ten changed over the lifetime of the project, and each change underwent
significant examination for consequences.

The project was waterfall all the way.

The quality of software that can be built if you know before you start what
it's meant to do, and that doesn't change, is _astounding_. Truly astounding.

~~~
matsemann
I have worked on huge government projects that has been specced and planned
for years before arriving at my team's desks. It takes a few days before we
will have questions. Things not defined good enough, inconsistencies, glaring
holes etc.

I think having all reqs upfront is a utopia that will not happen. Those
projects I was involved in would have been better if the reqs and scope was
done in conjunction with us implementing it.

~~~
EliRivers
We had some good requirements people. Really good. They didn't just write
things down. There was a lot of examination and analysis and plotting and so
on, and they were good at it. They'd all begun their careers as software
engineers and gradually specialised into being really good at requirements.

 _I think having all reqs upfront is a utopia that will not happen._

What can I say? It happened to me. Only with one company in seven or so worked
for so far. I would not be surprised if people worked two dozen jobs and never
experienced it.

You do need high-quality customers, though, and they are hard to come by. I've
worked for a company that has turned down customers on the grounds that the
potential customer just isn't good enough; they worded it differently, but
that's what it was.

------
drenginian
It’s not a problem of software engineering, it’s a property of software
engineering.

~~~
cjfd
Yes, that is very true. If changing and complex requirements are a problem,
you are doing it wrong. This is the main reason why code quality must be as
high as possible. It is the best chance there is to be able to accommodate new
and changing requirements in a reasonable time frame. It is also the reason
why automated tests are a necessity. Otherwise it is way to easy to satisfy
one requirement while not noticing that at the same time one is destroying
three others.

~~~
bryanrasmussen
If the subject is Software Engineering and you said "If changing and complex
requirements are a problem, you are doing it wrong" the it in that sentence
should be Software Engineering - but the changing and complex requirements
seldomly come from the Engineers.

~~~
cjfd
No, they do not come from the software engineers. It is like being a DJ who is
taking requests. You are not playing music for yourself but for other people.
It the DJs task to make them sound good. On the other hand, one sometimes also
needs to clarify what is wanted because the customers will ask for a rock song
with a slight edge of metal in it and as the DJ you will need to know what
song fits that description.

~~~
bryanrasmussen
It seems that when I say that requirements seldomly come from the software
engineers you have somehow misunderstood me as meaning that requirements do
come from the software engineers?

~~~
cjfd
No, it is just that your post sounded to me like you were saying that it is a
problem that the requirements are not coming from the software engineers. My
post attempts to say that this is not a problem and that it actually is how it
should be.

~~~
bryanrasmussen
How could you possibly interpret what I wrote in that manner?

I quoted your statement "If changing and complex requirements are a problem,
you are doing it wrong" and pointed out that it was not reasonable to say
someone is doing Software Engineering wrong due to changing requirements, as
the engineers are seldomly in charge of requirements.

~~~
cjfd
Ah, now I understand what you are trying to say. Yes, the 'it' in my post is
indeed referring to software engineering.

It is not so much that they are doing software engineering wrong due to
changing requirements as that the changing requirements make clear that the
quality of the engineering is not that great. Another engineer who is working
with a higher quality perhaps could have handled the changing and complex
requirements without much problems. And in another project the requirements
are perhaps quite simple and there is not much requests for changes and both
engineers would have done equally well because their skill was not tested
much.

Of course, there also has to be some limit in changing and complex
requirements beyond which it is no longer reasonable for anyone to be able to
keep up...... but we do call it SOFTware as opposed to HARDware because it is
supposed to be changeable.... and if that which is supposed to be changeable
actually is not, there must be something wrong.....

------
Tainnor
By now, I'm rather convinced that the "core problem" of software engineering
is that it's too easy. Yes, despite the myriad failures in delivery, once you
actually get some working software onto the market, its value immediately
multiplies as a sole function of your marketing (and, incidentally, I think
this is one of the reasons why US based companies are more successful than
e.g. European ones: they can start out with a big single market). And even
when you're not profitable, VC money is being pumped into the most ridiculous
of companies right now.

Of course, when faced with this kind of situation, why bother getting things
right and professionalising? I've seen big companies with the most insane
development cultures where things would constantly break, and yet it didn't
really have any big negative impact on profits. Maybe the well will dry up at
some point, but so far it doesn't look like it.

------
danielovichdk
Developers who believe they can forsee the future are more dangerous than
missing business requirements.

The problem with missing business requirements are that developers often are
either to focused on writing the code instead of actually understanding the
business.

You cannot build something you do not understand.

Understand small and expand your understanding. Code is not the goal, it never
is. It's a means to a goal.

Another thing about software is, that a lot of it should never ever be
written. It's the last thing you do, is write code for overcoming a challenge,
because rarely you really need it.

~~~
ausimian
Nicely put. In my nearly three decades of software development, the most
successful of my various colleagues have been the ones motivated by
understanding the problem and its wider context, rather than the
tech/process/language _de jour_.

This is a multiplier because it allows them to detect and ignore non-problems
and work on problems whose solutions _will_ deliver value.

------
atoav
It is funny to compare that to traditional product design. Let's say your goal
is to design a chair. The most important tools in your toolbox are variants
and iterations. The worst you could do as a designer is constantly working on
one (final) chair. Because every change that goes aginst the initial concept
will have you saw pieces away and glue them to other spots till you end up
with a completely irrational whole.

The problem is of course, that a low effort design prototype still looks and
feels like a low effort design prototype. But the customer sees 5 variants of
the low effort prototype and can decide which one is iterated on further.

If you program a low effort software prototype, it should still kinda work and
this is where the customer would decide to run with it.

Also software (like graphic design) is a field where people without any skill
tend to have strong opinions about how things should look or work, which makes
manipulazing the customer into taking a decent solution part of the job.

~~~
taurath
> Also software (like graphic design) is a field where people without any
> skill tend to have strong opinions about how things should look or work

Business too. You do not have to be a good strategist to “succeed” as a VP in
a medium or larger sized organization. In fact, you can be downright terrible.

~~~
steve_g
But when you're writing business software to solve business problems, many
more people have a "right" to an opinion. Usually the business people know a
lot more about the problem domain than the programmer. They are also the ones
who will use the solution.

I've been thinking that this is one of the key differences between software
and bridges. When you design a bridge, very few people have a "right" to an
opinion on the bolt sizing. But when you design an order entry system, the
business users (and users of the downstream data) have a valid right to an
opinion.

~~~
atoav
Agreed, if it is their profession they know it better. The thing I was
initially refering to is stuff like people who do a horrible layout in word
and ask to to literally do the same thing without having any qualification on
the matter. Or people who want you to put a whole page of text nobody will
ever be able to read onto a roadside sign.

This feels like if one would ask a electrician to use wires without insulation
because they like them better — you can totally have your taste, but some
stuff is just plain stupid and won't work as intended.

Graphics e.g. is about communicating certain things, and a graphic designers
job is to be aware of what this communication does and how effective it does
it. And so just like with a bridge (but arguably less dangerous) there can be
design that does the job and design that does the polar oposite of what your
customer intended.

------
ctdonath
The key difference between software vs all other engineering: building it is
instant & free & repeatable, vs expensive & slow & singleton.

If bridges could be replaced in minutes for pennies, we’d build them
iteratively too.

~~~
ken
Buildings certainly _do_ change. I just finished 3 months of work on a
structure that was "finished" 20 years ago. We had the client coming to us
with changes to our changes several times a week.

The problem with software is that we don't build it in a way that makes
changes easy (or at least not all the types of changes that clients frequently
want to do), and we do a terrible job of communicating the costs of changes.

Give me any program, and I can change a couple words in its requirements which
would cause more than half of its lines to need to be changed. That's just how
we write software today.

(We diligently made 'time_t' a typedef, and then we embedded the physical size
of it in every file format and API and protocol so we can't actually change it
without breaking everything.)

Maybe we need a new type of specification system, to encode the distinction
between "this is a decision which is immensely expensive to change and will
necessitate rewriting half the program" and "this is just part of the facade
and we can change it cheaply at any time".

------
moron4hire
After 50 years of activity in attempting to improve software project metrics
with little to no improvement, we have to admit that we've been looking in
completely the wrong place for effective change.

It's not how the developers work that is the problem. We've changed languages,
team sizes, project cycle lengths, everything. Except one thing.

The one thing in common with all of these failed software projects is
corporate, top-down management. I would say it's even gotten worse, to
counterbalance the real productivity improvements we have actually gained from
improved tools. You look at all the companies that _can_ deliver software on
time and budget and they have one thing in common: they aren't ran by Harvard
MBAs. You look at all the once-great companies that eventually faded away into
obscurity and they also have one thing in common: imposition of MBA-imposed
corporate management structure to "cut costs" (aka, "throw out the baby with
the bathwater"). It killed Bell Labs. It killed Xerox PARC. It killed Apple
once, which then was reborn from its own ashes, and is in the process of
killing Apple again.

It's impossible to be a great developer in an environment that denies that
there can be unique skills and knowledge to individuals. Anywhere developers
are "resources" they will be treated as interchangeable cogs.

------
bob1029
I will say that our experience is that the happy path for virtually any
customer is to ship the MVP as soon as humanly possible. This also means each
new/revised feature after the fact. Only once the customer is actually
screwing around with your app or feature will they start to develop a true
understanding of what is unfolding relative to their daily life or business.

Our customers have extremely high expectations for the quality of output from
our application. Their entire business hinges on whether or not we marked a
specific checkbox correctly on one of many insanely-complex forms. We started
out on this project with a mentality of "its gotta be perfect and we gotta be
done at some point". As a result, our development cycles dragged on for months
with the customer testing everything all at once on a single "drop dead day".
Thankfully, we were able to get away from this mindset. We went from 'almost
killing the company' to 'everyone is extremely happy' simply by delivering
features to customers on a daily basis. Having that nearly-immediate feedback
on new code from the customer means that the responsible developer likely
still has most of the abstract concerns loaded into their mental cache and can
quickly iterate if needed.

One other aspect that really started putting leverage behind us was building
our own integrated tools for easily investigating exceptions or other issues
in customer environments. If a user of our application experiences an
exception, we are typically reviewing it within 5 minutes (without them having
to report it to us explicitly), and if its a trivial affair, a build with a
fix would be available for update that same day. Having the ability to trace
user events through our application is probably the single most impactful
troubleshooting tool available to us aside from the raw logs.

Ultimately, all of the above is the acknowledgement by us that software
development is a very hazardous affair, and that we need processes and tools
available for rapidly engaging these hazards when they arise. The sooner you
realize you have the wrong answer to a problem, the sooner you can go off and
find the correct one.

------
zuckluni
I disagree with the title. I've used waterfall and still found engineering
hard.

Even you have a bulletproof requirements document, implementation and
engineering still produces unforeseen eventualities and unexpected
consequences.

I think software engineering is hard primarily because it's the coalface
between abstract and concrete.

We're wired to be great at concrete, but capable at abstract. With software
it's like we can get ourselves into a perfect storm of intractable-for-us
problems.

Other forms of engineering, civil, mechanical, etc are still really hard, but
I think we're better wired for them because they're less abstract, more
concrete.

But in software, abstraction is part of your bread-and-butter daily-grind
tool-kit.

------
physicles
If requirements volatility is the core problem, the next question to ask is,
Why is this the core problem here and not in other engineering disciplines?

I think it stems from a mismatch between the perceived cost and the actual
cost of changing software, which one of the first comments on the article
captured:

> That’s my take – the perception that software is simple to change, which
> stems from the lack of understanding of its inherent complexity by users,
> and the fact that it is, up to a certain point – is what makes requirements
> volatile.

Part of the problem too is that the actual cost of changing software varies by
an order of magnitude or more, depending on how fluent the developer is with
the tools they’re using, and how good they are at accommodating change. If
that’s not something even we as a profession (if you can call it that) have
been able to reliably quantify, expecting a client to understand it is beyond
the pale.

I’ve been programming for most of my life, so if I see an app then most of the
time I can get a sense for how long it’d take to build. But I still have this
nagging feeling that building software — the act of translating what’s in your
head into working code — seems to take way longer than it ought to.

~~~
xorcist
> Why is this the core problem here and not in other engineering disciplines?

Software is the blueprint of a process. Software engineering is basically the
art of exact specifications.

Once you've _unambiguously_ specified what it is you want, you're done. The
rest is pretty much an appropriate language an a compiler but we have plenty
of those.

------
32gbsd
There is a whole industry of people who are in the business of telling
programmers what to do but not how to successfully complete a project because
they do not know anything about how to actually finish the things they tell
people to start. "Just use agile everything will be OK!"

~~~
32gbsd
Oh and lets not forget the project managers themselves that blame the
client/users instead of their own inability to meet the requirements;
[http://owensoft.net/v4/item/2368/](http://owensoft.net/v4/item/2368/)

------
mrdoops
This, along with the cost of development, is why I have a pet-peeve for hard-
coded business logic directly in the code (in most cases). I.e. when a
frequently changing business process like a finite state machine for an
approval procedure is only change-able by a developer that means there's
always the cost of that developer's "need-to-know" for a straight forward
change in the business operations. The cost of a given change is greater than
a declarative capability that took into account operational changes in the
first place.

That said hard coding business logic is often so much quicker to deploy that
in many situations you just want to hard code the logic, deploy, get feedback,
then introduce flexible to the kind of change needed then continue. If you
have engineers who know how to ask the kind of questions that get to the real
requirements (usually not juniors) you can have a maintainable code base.
Worst case scenario is if you have the "yes I shall do anything and everything
with my new code power" developers who just implement out of eagerness and
conflict avoidance; this can be the worst of every world: hard coded business
logic so convoluted only the junior knows how to make changes.

~~~
crgwbr
> This, along with the cost of development, is why I have a pet-peeve for
> hard-coded business logic directly in the code (in most cases).

I’ve built systems both ways. Requiring a developer to make changes definitely
has costs, but it does have benefits as well. Systems configured in data
rather than code will almost invariably be untested and prone to configuration
bugs. Overall, I’ve spent probably double or triple the time debugging
configuration of complex business rules systems than I would have if it was
just written in code and unit tested.

~~~
mrdoops
Agreed. I guess my point is that we need to focus more on making declarative
business rules systems more tenable as an approach in general. If we consider
the framework + CRUD generators as the most productive way to build a web app
(probably has been since Rails) - that's a result of tooling built around an
approach. What happens when we focus similar efforts of tooling towards these
Rule Based components?

------
29athrowaway
Agile is about giving developers the illusion of control, so they are more
comfortable committing to an endless cycle of nonsense.

~~~
corporateslave5
No agile is just micromanagement. Control every task the developer works on,
make them report status on a daily basis.

~~~
nineteen999
Ask 10 different developers, Agile salesmen, and product owners and you'll get
10 different explanations of what Agile is.

Occasionally you'll come across somebody with a healthy dose of skepticism.

[https://news.ycombinator.com/item?id=5406384](https://news.ycombinator.com/item?id=5406384)

~~~
hliyan
I think a lot of people are conflating Agility with SCRUM. If you look at the
official SCRUM guide, you'll find ZERO references to agility. And if you look
at the original Agile Manifesto, you'll find ZERO references to stories,
sprints, story points, velocity, standups or any other thing that we consider
agile.

For me, the essence of agility is: break down large projects into smaller and
smaller projects until the "mini-project" becomes simple enough for you to
execute. But each such mini-project _must_ delivery something that works.
Also, don't show documents about the software, _show the software_.

------
chiefalchemist
The two most common patterns I've experienced:

1) Poorly communicated goals. Sometimes not communicated at all. If you don't
know where you're head and _why_, it's damn near impossible to get there.

2) Wants being confused with actual business needs.

3) Clients / stakeholders not able or willing to put in the time for essential
discussions and critical decisions.

------
perlgeek
The article talks about the one contract that had the requirements fixed --
but then fails to talk about how contracts should be laid out to embrace
changing requirements.

People have known for a long time that volatile requirements are a core
problem, but that typically hasn't really affected the way software and
software development services are procured.

This might be different in other parts of the world, but my impression is that
even custom development contracts are procured the same way that you would a
big shipment of physical goods, which is just insane.

Once you work as a developer in a project that was procured in such a way,
it's already too late. The knowledge that requirements will change and will
totally disrupt your development process is there, and it doesn't really help
you.

------
ellius
As an addendum to this, I would frame the problem as "complexity," one aspect
or symptom of which is requirements volatility.

Complexity comes from two sources in software projects. The first is the
domain. As other commenters have pointed out, domain experts are usually half-
aware of their domain requirements because much of the complexity has been
buried under their expertise and intuition. Domain requirements also
inherently change over time as markets, professional standards, and technology
evolve. So the process of development involves a shifting discovery of new
aspects of the domain as things which were unconscious become consciously
learned.

The other aspect of complexity is the technology applied to the problem.
Computer technology is inherently complex, and the imperfect mapping between
domain problems and technology adds a third angle.

The best books I've seen for dealing with these problems are "Clean
Architecture," "A Philosophy of Software Design," and "Principles of Product
Development Flow." The first two explain how to build good systems that
encapsulate complexity and are geared towards changeability. Such systems are
flexible as new aspects of domain complexity are discovered over time. The
latter book takes a higher level view of how to design projects to cope with
the discovery process in a way that is economically viable and justifiable to
the people financing projects, and also in a way that arranges work so that
developers can actually be productive and not become swamped by the complexity
problem or bad processes and practices.

------
Gibbon1
> As a consequence of this, software is never finished, only abandoned.

Ditty of mine from around 1988 or so. The hardware is old and obsolete but the
software never is complete.

One thing I realized a long time ago was that you need enough resources to
keep up with the requirements drift or you will never finish. That's what's
behind YAGNI and Minimum Viable Product.

There is this other thing which is a successful product 'creates it's own
weather' as far as requirements go.

------
pbhjpbhj
Is it primarily requirement volatility, or primarily ill-defined requirements?
(If it's the later, that's some super-irony.)

That would explain at least a couple of further issues mentioned here. Like
rewrites: software doesn't always define it's own outcomes well; the
requirements don't change they're just poorly defined.

Requirements changing, and feature creep are subtly different too -- with
static ill-defined requirements features could change as the requirements in a
particular facet of the software become more clearly mapped out further on in
the project.

For example, in the Berlin airport situation (in a comment here), adding a new
floor could be due to a realisation that a static requirement for floorspace
want going to be obtained, because (say) a static requirement for through-flow
hadn't been fully defined and expanding the capacity to meet that requirement
then limited the available floorspace. None of the requirements needs to have
changed.

Perhaps that's to much of a philosophical distinction.

(Not a [software] architect, or project manager or anything, feel free to tell
me I'm talking rubbish.)

------
hden
> Requirements change. Every software engineering project will face this hard
> problem at some point.

Rich Hickey calls this a 'situated' program in his talk from Clojure/Conj
2017, where the program has to deal with information and real-world
irregularities for extended periods.

While agile disciplines help you from one perspective, your tools (e.g.
program language) might also help you solve some of the problems. In this
framing, Clojure certainly tries to help where the it matters most.

Highly recommended to read the transcript or watch the video, even if you are
not interested in the language itself.

Video:
[https://www.youtube.com/watch?v=2V1FtfBDsLU](https://www.youtube.com/watch?v=2V1FtfBDsLU)
Transcript: [https://github.com/matthiasn/talk-
transcripts/blob/master/Hi...](https://github.com/matthiasn/talk-
transcripts/blob/master/Hickey_Rich/EffectivePrograms.md)

------
robocat
Don”t rewrites also fail all the time, even if requirements are not expanded?

Rewrites would mostly be successful by the articles premise.

~~~
Nasrudith
Well rewrites have two conflicting goals - simplifying the code/improving
performance and consistency with the old version. Even if the requirements are
nominally the same "unwritten" ones pop up

~~~
robocat
Sure, but “unwritten” requirements pop up all the time for any requirements,
so the two situations are still broadly equivalent.

------
Ericson2314
Honestly the desire for simple unit economics and legacy property rights
really fucked it up. People pay for "a software", but what they really need is
a change in the collective software commons. Of course, no one expects any
change order to be the final one, so the the volitility is natural and
expected.

------
temac
Just to nitpick, there are counter examples of finished but not abandoned
software. As usual, Tex comes to mind.

Or they just could. E.g. we could probably take the current Python 3 and
freeze it and debug it for 20 years, at which point it would be basically
finished, and maintenance could still continue (but way much less would be
needed)

IMO a good part of requirement volatility is a self inflicted wound. Some are
essential, of course, but some changes are for the sake of changing and giving
PM, UI, and software engineers jobs, and I even don't know lot of people who
appreciate too many gratuitous changes in e.g. software GUI.

My general advice is to slow down. And fucking finish things before starting
new ones. We don't let half-constructed bridges around just because we start
to build new ones with fancier materials.

~~~
BlueTemplar
I guess that slowing down doesn't pay well in an industry where a startup can
get from zero to billion in a few years?

~~~
temac
Probably that plays a role. However, not everything is a startup. Accumulating
technical debt during even just a few years like there is no tomorrow is also
a non negligible risk. You can _also_ go from billion to zero in a few
years...

------
waterlink
My honest opinion is that software conceptually __was created __to allow for
volatility of the requirements.

If you want to have static requirements (or changing once per 5-10y), then
that’s what you have _hardware_ for.

Software has “soft” in the name for that exact reason.

Embrace the constant change.

------
sktrdie
A really big influence to me has been the work by prof Harel on Behavioral
Programming (aka scenario based programming) which makes coding similar to how
requirements change, evolve and sculpt the software.

His take is much more pragmatic than the Agile manifesto hinting at entire new
languages (live sequence charts) and paradigms (bthreads) where requirements
can actually be executed

This wonderful video by Harel himself is a good intro on these concepts:
[https://vimeo.com/167925763](https://vimeo.com/167925763)

------
x3haloed
>In an agile project, ideally, there is a working partial implementation
starting very early in a project, and observable progress is being made toward
a satisfactory product from the first.

Can anyone give a practical example of how this works in the real world? Like,
what's a complicated workflow that needs automating, and how would you build
it iteratively? Also, how could you predictably alot a time budget in this
case so that you are not endlessly chasing satisfaction on a specific product
or feature?

------
hwayne
As part of a project I interviewed a lot of "crossovers": people who worked as
both traditional engineers and software engineers. According to everyone I
talked to, requirements volatility happens everywhere. Two civil engineers had
to move a bridge. One "system engineer" worked for Boeing and was part of a
team who's only job was to figure out how to reconcile existing work whenever
they changed requirements.

------
mobjack
I change the requirements of my personal projects all the time.

Once I get a prototype working I get lots of ideas of how to improve things. I
also see that some of my initial plans did not work I out as I envisioned and
have to iterate on them.

If I can't get the requirements right on my own projects, then I have no
expectations that others could do so either.

Have the expectation that requirements are wrong on any project. It is better
to embrace that these will change rather than try to fight them.

------
mannykannot
Requirements volatility does not fully explain the frequency with which basic
security flaws keep reappearing, long after they have been recognized and
documented.

------
WalterBright
That's true of any form of engineering, not just software. Requirements change
because circumstances constantly change, and people simply are unable to
imagine it completely beforehand. One has to build it and try it out.

As software engineers, we build things to be adaptable to change. That's why
it works to build things out of encapsulated components.

------
vanniv
Requirements are fairly volatile -- but the real problem isn't that the
requirements themselves are volatile but that the true requirements are so
poorly understood (and, thus, the apparent requirements shift rapidly as
understanding of the problem domain changes throughout the software
lifecycle).

------
beefield
As someone somewhere said, writing software to a spec is like walking on the
water - Easy when it is frozen.

~~~
guggle
Still slippery though.

~~~
Aeolun
Better make it a rough spec then.

------
xamde
Requirements are hard, but the _core_ problem of software engineering is
complexity.

------
bryanrasmussen
This is somewhat triggering for me right now as I am just getting done with a
project that is quite a few degrees hotter on the Requirements volatility
scale than is currently assumed.

------
amelius
Hopefully the existence of this article will not make it legitimate for
managers to change the specs all the time.

------
heymijo
> _The goal of software engineering is to make that development predictable
> and cost-effective._

Is this true?

------
longtermd
"Requirements volatility" is just another word for: A/B testing software :).
Let's get used to it

------
carapace
The weird thing (IMO) is that computers are already universal machines, so why
isn't _any_ programming language already enough for _any_ task?

In other words, why hasn't Excel _fully_ defeated all comers?

It's widely known but seldom appreciated that _Excel_ is the most popular and
successful programing language in history. And not just a little. It wins by a
large margin. So why do people still pay you and I to "make things go"? (Yes,
that's a Pakled reference. [https://memory-
alpha.fandom.com/wiki/Pakled](https://memory-alpha.fandom.com/wiki/Pakled))

I think to answer this question you have to understand the difference between
what we have now and what we could have.

What we have now is a stack: OS, WM, "Desktop" -> Separate "Applications"
often with their own GUI (in addition to the OS-provided defaults) -> Web
"apps" where anything goes (up to and including "dark patterns" like fake
widgets designed to mimic OS-level modal dialogs.) All of this is conceptual
_overhead_ that adds nothing to the achievement of the prime tasks (that
humans are using computer for in the first place.)

What we could have: I have _two_ examples: The Oberon OS and Jef Raskin's
work.

[https://inf.ethz.ch/personal/wirth/ProjectOberon/](https://inf.ethz.ch/personal/wirth/ProjectOberon/)
[http://www.projectoberon.com/](http://www.projectoberon.com/) Try it live in
your browser:
[https://schierlm.github.io/OberonEmulator/emu.html](https://schierlm.github.io/OberonEmulator/emu.html)

[https://en.wikipedia.org/wiki/Jef_Raskin](https://en.wikipedia.org/wiki/Jef_Raskin)
[https://en.wikipedia.org/wiki/The_Humane_Interface_(Book)](https://en.wikipedia.org/wiki/The_Humane_Interface_\(Book\))
[https://en.wikipedia.org/wiki/Canon_Cat](https://en.wikipedia.org/wiki/Canon_Cat)
[https://en.wikipedia.org/wiki/Archy](https://en.wikipedia.org/wiki/Archy)

> These ideas include content persistence, modelessness, a nucleus with
> commands instead of applications, navigation using incremental text search,
> and a zooming user interface (ZUI).

The core idea is to get "applications" and even modes out of the way,
disintermediate as much as possible between the user and the raw computing
power of the machine.

In sum, we spend an enormous amount of time and energy working on things that
have _no_ intrinsic bearing on the tasks we hope to accomplish, largely due to
the lack of awareness that it's possible to dispense with _most_ of what is
today considered normal UI. We are "turd polishing".

If you get out of the users' way, they can implement their own requirements.

\- - - -

Here's RMS talking about secretaries extending Emacs with Lisp:

> programming new editing commands was so convenient that even the secretaries
> in his [Bernie Greenberg] office started learning how to use it. They used a
> manual someone had written which showed how to extend Emacs, but didn't say
> it was a programming. So the secretaries, who believed they couldn't do
> programming, weren't scared off. They read the manual, discovered they could
> do useful things and they learned to program.

[https://www.gnu.org/gnu/rms-lisp.html](https://www.gnu.org/gnu/rms-lisp.html)

------
DanielBMarkham
<rant>I don't think most people know what they're doing when it comes to
managing the delivery of software. The field is full of canards, facile
simplifications, and self-referential, circular reasoning. A few examples.

\- The problem is requirements. Is the system broken? Yes. Do the requirements
say for it to be this way? Yes. Must be broken requirements, right? Since
requirements are the "driver" of the process, if the process is broke so must
be something about requirements.

\- Software is never done. Bullshit. Do I need somebody to reprogram my
decades-old mp3 player? I do not. Would it be nice if somebody did? Maybe, but
it's not required for me to enjoy the software. The software is written, does
its job, and needs no maintenance. It's done. There's plenty of software being
updated today for pretty good reasons that for all intents and purposes is
done. Just nobody wants to admit it. If it's possible to add stuff or update
it, we do. It's what we do.

\- FTA, "...Requirements change. Every software engineering project will face
this hard problem at some point..." That's true, but woefully incomplete. Your
job creating software is not to meet the requirements, it's not to change and
add stuff as the requirements change. That's viewing the work, once again, as
requirements-driven. No matter what your process or paperwork, the work is
ultimately user-driven, not requirements-driven. Your job is to scope a
domain, anticipate change in that domain, and provide easy-to-use configurable
software that anticipates that change. You do that by knowing the users and
the work, not by reading requirements or responding to updated requirements.
That kind of thinking removes the key piece of the entire work, the user. By
the time somebody gives you a requirement, it's too late. Your design is
whack.

How do we make aviation and space travel gear safe? We fly the hell out of it.
What's the wrong way to go? Build one-at-a-time and ask for changes and
complete reworks for every new project. How do we build software today? Like
we used to build rockets; using the wrong metaphor. We need reusable,
configurable, capabilities-driven software, not endlessly worked-on huge hunks
of bytes from BigCorp feature factories.</rant>

EDIT: Just to clarify the rocket metaphor, we'd be far, far better off using
COTS tools and writing minimalist, simple code that only does a few things,
then pounding the hell out of it with actual usage to figure out which "axes
of change" we missed. Instead, more and more everything has some unique tech
stack, a byzantine testing and deployment deployment system, the entire thing
fun of tons of complex one-of code everywhere ... then we stick on new
features one at a time every time there's something a user wants that it
doesn't have. From a managing-complexity and managing-cost standpoint, it's
actually worse than the Big Project rocket situation we had for decades.

------
AmericanChopper
Perhaps that is the core problem, but that problem is caused by not knowing
the requirements to begin with. Which again is usually caused by not knowing
what your customer wants (which you should be able to do, even if they don’t
know themselves what they want), or not knowing who your customers are.
Contract work is the best environment for fostering a culture of not knowing
what your customers want, because it is the ultimate exercise in fence-
throwing (often to somebody who’s just going to throw it over yet another
fence).

Problems tend not to change so dramatically over the course of a project. What
usually happens is some stakeholder realizes at some point that they didn’t
actually understand the problem the first time around, and thus need to change
the requirements. Even significant regulatory changes won’t usually cause
that. In almost all cases, regulatory changes have a significant amount of
time between announcement and enforcement.

Making your development practices more responsive to change is a good goal,
but it doesn’t solve this problem. Because this problem is typically created
before the first line of code is written. If the defined requirements don’t
actually solve the customers problem, then they’re going to be delivered a
non-functional solution, whether it takes one year to deliver, or one day.

