
My 20-Year Experience of Software Development Methodologies - fagnerbrack
https://zwischenzugs.com/2017/10/15/my-20-year-experience-of-software-development-methodologies/
======
tw1010
I find it somewhat amusing that this article doesn't pick up on the fact that
it is furthering the root cause of this very issue in the act of basing it on
the book Sapiens. A core problem with these mythological trends is that they
reduce world views down to easy-to-communicate little stories. This is useful
for propagating ideas from person to person, but it necessarily has to be
pithy and avoid too much detail. What we aught to do is to notice this
historical pattern, and to take a stand against it. We should try to not be
too persuaded by the book, or methodology, of the month (Sapiens right now)
but to try to go back to the hard, tough, classics. We should try to build a
foundation based on actual psycological and sociological research, instead of
being steered too much by the, understandably attractive, mimetic pulls of
water cooler philosophy.

~~~
danblick
I used to think in terms of stories, but then I heard Tyler Cowen tell me a
story about the dangers of stories, and now I don't think in terms of stories
so much. (My story of personal growth.)

[https://www.ted.com/talks/tyler_cowen_be_suspicious_of_stori...](https://www.ted.com/talks/tyler_cowen_be_suspicious_of_stories/up-
next)

(PS: my comment is actually just a joke from the talk, but in any case I
really enjoyed and talk and would recommend it.)

~~~
dctoedt
Obligatory cite: [https://xkcd.com/552/](https://xkcd.com/552/)

------
mhomde
Nice article! I'd like to a software methodology more supportive of what I'd
call "Continuous refactoring" or a more "kaizen"-like mindset. "Agile" isn't
always as agile as I'd like, but becomes rigid in peoples interpretation of
the rules.

It's a constant fight to keep down the number of lines of code and complexity,
in large projects. Once a code-base passes beyond a certain point it becomes
difficult to see redundancies and patterns. Bloated code breeds bloated code.

If you on the other hand keep trimming and keep refactoring deep and wide as
the code and requirements change you can often keep things manageable and
understandable. "bad code" get's easier to spot, it's a little bit like the
broken windows theory.

Scrum IMO and a lot of the agile methodologies, at least in their
implementation, in my experience, tends to treat projects like brick-building
and lacks a way to support and enforce constant re-architecturing and removal
of code smells.

If code quality and refactoring is not supported by the actual methodology it
can become an bureaucratic problem where product owners doesn't see the value
and doesn't prioritize it (even though it leads to quicker development time
and less bugs in the long run)

Unit testing is supposed to make refactoring easier, but can become an
impediment to refactoring as well, as the cost of rewriting tests can be seen
as too high.

We're currently experimenting with having one hour meeting each week where you
can discuss things like that, and then dedicating some time to refactor, we'll
see how it works out :)

~~~
zwischenzug
That sounds like the 'healthy' team I talk about in the article. The problem
with the 'continuous refactoring' (in my experience) is that the people that
hold the purse strings cannot grasp the benefit, so cut this early.

~~~
user5994461
The problem with refactoring is that developers think it's a task of it's own.
They want time to do it, they want special estimate, they want it to become a
thing of its own.

Refactoring is low level work that is part of every task, it should not even
be mentioned.

When you ask your mechanic to change wheels, you don't expect him to explain
that he will have to take off the previous wheel and change the screws, that
will be an item on the bill consuming half of the budget.

~~~
ehnto
It often is a task of it's own. It's not always "low-level work", sometimes
it's directly replacing key components in a huge chain of events that all need
to be modified to accept your refactor. It really depends on the codebase,
features you're developing, nature of the business engagement and so on. There
is also risk in a refactor, especially if it's heavy in business logic.

If I'm doing a bugfix task on a legacy site, I might note that it would be
good to spend ~4h on a refactor, when the bug may have taken 0.25h to fix. In
that scenario the benefits for refactoring are low, so we don't do it.

Alternatively, if I'm doing a 6h feature, and understanding and working with
the bad code would take 6h, versus spending 4-6h on a wholesale rewrite, of
course I would just slip the refactor inside the original task.

You have to keep your expectations realistic and your project managers
informed. If your refactor suddenly blows out in time because you didn't fully
comprehend your problem space, then you both haven't completed the feature or
achieved the refactor. Maybe in your business a day extra is fine, in some
businesses running over by a couple of hours can mean the ROI doesn't make
sense on that task anymore.

The key is to be sensible and pragmatic. There is no one solution to all
thinking. If there's anything any of these methodologies are trying to do,
it's provide a framework for quickly communicating and reacting to the nuance
of software development.

~~~
user5994461
>>> It often is a task of it's own. It's not always "low-level work",
sometimes it's directly replacing key components in a huge chain of events
that all need to be modified to accept your refactor. It really depends on the
codebase, features you're developing, nature of the business engagement and so
on. There is also risk in a refactor, especially if it's heavy in business
logic.

And that's why refactoring has a terrible reputation and developers can't be
trusted. Now you're talking about replacing key components and altering the
business logic under the pretext of refactoring.

This should NOT be called refactoring, this should be called replacing key
components and revisiting the business logic.

Of course, management doesn't want that to be done without justification.

~~~
watwut
Refactoring may change how business logic works internally without changing
anything in input/output or features. If you need to change how key components
are implemented, without changing what they do, then you are refactoring them.
Changes may be big internally, but it is still called refactoring if features
don't change.

------
mannykannot
I am ambivalent about "the difficulty lies not in defining [useful software
methodologies], but in convincing others to follow [them]."

Currently, I see too much faith being put in methods of dubious utility. I see
techniques, that may be effective when applied judiciously in some
circumstances, being promoted to universal principles. In particular, I see
the failure of waterfall being taken to mean that any attempt to think ahead
is counterproductive.

On the other hand, ten years before the scope of this article, Barry Boehm
took on the waterfall model with an iterative one that emphasized frequent
reevaluation of where you are, where you are going, and what impediments you
face [1]. The fact that this article begins with waterfall supports the notion
that adoption is the problem. Perhaps Boehm's model was largely overlooked
precisely because it was not dogmatic enough to promote zealotry?

[1]
[https://en.wikipedia.org/wiki/Spiral_model](https://en.wikipedia.org/wiki/Spiral_model)

~~~
snarf21
Very true. I've found at every company that I've worked at that a "bad"
process followed consistently by everyone always outperforms the "perfect"
process followed haphazardly.

This reminds me of the recent threads about people using Excel instead of
"real" software. At the end of the day, you are trying to add value. As long
as you aren't adding technical debt or other risks, the how matters much less.

------
ozmbie
It's important for me to separate the principles behind these methodologies,
from the industries that spawn up around them to sell guides and
certifications and books and conference tickets.

Adhering to the ideas of clear goal-setting, team communication, autonomy, and
iterative development have never had bad outcomes for me. But formalising them
into a ruleset for any arbitrary team doesn't capture everything.

~~~
williamdclt
I do agree with you, but saying "I adhere to the ideas of clear goal-setting,
team communication, autonomy, and iterative development" doesn't lead you
anywhere: you need tools and actual processes to convert this adhesion to an
idea into the materialization of this idea.

And that's where those companies intervene: they build an implementation of
those ideas. The implementation is not adapted for any project and team and it
is lossy regarding the original principles, but it has the advantage of having
being seriously thought about, tested and scalable.

------
danblick
Okay. I'm not a zealot about these things, but I think the author
underappreciates the nicer aspects of agile. My favorite book on agile looks
at it from a mathematical / risk management / process model standpoint.

[https://books.google.com/books/about/The_Principles_of_Produ...](https://books.google.com/books/about/The_Principles_of_Product_Development_Fl.html?id=1HlPPgAACAAJ)

The concepts used in the book - batch size, uncertainty, feedback, control,
queue size and latency - these really are useful things to think about when
scheduling work.

~~~
SubiculumCode
That said, it seems the author is making a larger point than the efficacy of
any individual methodology: Convincing people to co-operate within any
reasonable methodology/framework has the larger effect than effects of
different methodologies. That having a religion/philosophy is more important
than which religion/philosophy in terms of national co-operation, etcetera.

------
SatvikBeri
My approach is "Risk Driven Development" \- find the most likely causes of
failure for your project, and structure your methodology around reducing
those.

The standard lean startup risk is building something the customer doesn't
want, so it makes sense that the way to mitigate that is by getting customer
feedback quickly.

But other projects can have substantially different risks - for example, one
project I worked on involved building a pipeline to process a large amount of
data every day. There was no real customer risk, the spec was pretty much "do
this thing that's valuable on some data, but for 1,000x more." Instead, the
major uncertainty was whether the libraries we considered would scale up. To
mitigate this, we built small prototype projects with each library on
production workloads, and found something like 7 out of 12 of them would work
with minor adjustments, while 5 had to be discarded pretty much entirely. We
then wrote the project with alternatives for those 5, or writing replacements
for the functionality we needed from scratch.

As teams grow, coordination becomes a bigger risk. One way to reduce this is
by having interfaces between modules, and enforcing those interfaces in code.
My preferred method is static types and functions, but of course there are
others like JSON APIs.

~~~
rileymat2
This is quite common. I am curious if there is disagreement about what you
said.

~~~
SatvikBeri
Many companies I know try to use Agile/Scrum for data pipelines, and focus
heavily on getting a customer-ready prototype. This is then often really
difficult to scale, because bad architectural assumptions have been baked into
the project.

In general I see most companies/teams picking on methodology and staying with
it, rather than adapting to the project.

------
cmrdporcupine
In all the years I've been doing this the only times I've felt things really
gelled and were firing on all cylinders had little to do with methodology and
everything to do with culture and motivation. When people felt cohesive, and
they were all working towards the same goal, and that their personal
priorities and the project priorities could align.

I do think that the 'agile' methodology focus on short achievable iterations
and daily stand up meetings yielded good results _in that context_.
Disconnected from a healthy organization or transplanted into a bad one, it
just yields epic fail and falls apart.

Another thing that I saw work well was embedding product management, testing,
and operations people into small per-project teams with developers, which I
found reduced conflict from territoriality and reduced "throwing over the
wall" syndrome.

~~~
mstade
In my soon-to-be 15-year career as a professional software engineer (and
several more as an amateur) I've noticed that the common trait of the high
functioning teams I have had the distinct pleasure to work with was always the
same thing – a great leader. (In one case, there were two, working sort of in
tandem.)

Trying to sum up these leaders' qualities is difficult, but if I try to
distill it I think it comes down to primarily these two aspects:

1\. Strong vision and ability to clearly communicate direction 2\. Mutual
trust

They wouldn't tell people _what_ to do, and certainly not _how_ to do it, but
they'd always – every single day – point in the direction they wanted you to
go and course correct where necessary. It wasn't like it was an all-hands
meeting every morning either, it was more like gentle nudging or little
reminders, but not unsolicited. It's difficult to explain frankly, but the
result was that we as a team never felt unsure as to where we were heading.

The most important thing however certainly was the mutual trust. You could
trust these leaders to have your backs, no matter what, and these people would
trust you to get things done. That sense of trust created an immensely
palpable sense of loyalty, you really did feel like your input and efforts
mattered, and you always wanted to bring your a-game for the team. Again,
difficult to explain, but you know when you're in it.

Not one of these teams worked like the other, in terms of methodology, yet
things worked and they worked very well. We had issues, for sure, but I've
also worked in dysfunctional teams and the same kind of issues in those teams
were amplified by terrible leadership, leading to constantly worsening morale
and in some unfortunate cases abject failure to deliver.

------
itsyogesh
‘The test of a first-rate intelligence is the ability to hold two opposed
ideas in mind at the same time and still retain the ability to function.’ F.
Scott Fitzgerald

I know I am taking this out of context, but this is best thing I read on the
internet since a while.

~~~
gtycomb
This line too in the article --

'So I’m cool with it. Lean, Agile, Waterfall, whatever, the fact is we need
some kind of common ideology to co-operate in large numbers. '

------
rjzzleep
Companies really really like the words PMO and portfolio management. That's
why Scaled Agile Framework(SAFe) is that nasty thing you see in that picture.

"People will benefit from SAFe, but not nearly as much as they would from a
true agile approach"

[https://twitter.com/martinfowler/status/440874515199844352](https://twitter.com/martinfowler/status/440874515199844352)

IMHO all those methodologies that try to somehow shoehorn classical project
management into new things are at least to some extent the acknowledgement
that things aren't working as expected.

~~~
nradov
What specific parts of SAFe do you find to be nasty?

------
dovik
...with a funny link to [http://programming-
motherfucker.com/](http://programming-motherfucker.com/)

Personally, I did not know this manifesto before.

~~~
auggierose
It has featured a few times here before, I love the manifesto :)

------
wirrbel
There is a tendency in Software Development, that terms lose their meaning the
more they are used. I think it is partly due to software people not talking
concrete, physical objects that much, but more about abstract ideas with more
or less relevance to reality.

This can lead to interesting effects. For example, I have heard "DRY - Don't
repeat yourself" most times I encountered it as a reasoning for obsessive
refactorings, where functions where extracted from code without any regards to
abstraction layer just because a few lines of code looked the same (or could
be rewritten to look the same). But originally the term was coined to
represent the principle of "Every piece of knowledge must have a single,
unambiguous, authoritative representation within a system". "DRY" was however
so catchy, that the slogan propagates without the explanation of the
principle.

I think agile suffers a similar fate. Countless consultants, managers and
engineers have rendered the word almost meaningless. Even with the Agile
manifesto at hand, it is sometimes hard to free agile from the multiple layers
of paint spread over it.

DevOps suffers the same fate, and while I haven't researched this, I assume
that even waterfall is probably a gross misrepresentation of the original
ideas.

~~~
maxxxxx
Nice summary especially about the real intent of DRY and how people implement
it. I think what it comes down to is that software development is very complex
so any attempt to produce cookie cutter rules will fail eventually. OOP is a
good idea, Agile is a good idea, DRY is a good idea but you still need the
judgement to see how it applies to your specific situation.

I started doing this in the 90s and I feel that there is a growing trend to
replace judgement with process and strict rules.

~~~
wirrbel
Agree with the first part of your post.

> I started doing this in the 90s and I feel that there is a growing trend to
> replace judgement with process and strict rules.

Hmm, I am not quite sure whether I share your sentiment on process and strict
rules. I don't mind process if it is actually one that is sensible and leaves
freedom to the individual.

For example, in my experience as a developer, Scrum can drastrically improve
developer-management communication. Story point estimation with estimation
cards has always revealed stories with unclear scope or requirements. This
requires however, that both sides actively try to improve that process.

I have often heard of people who said "Scrum doesn't work" who actually were
really running a very weird subset of it (for example burndown charts and
velocities, but no proper story point estimation), etc.

------
gaius
I've tried or been used to try a dozen or more methodologies over the years,
more than I can easily name. Everything from SSADM and ITIL through RUP,
various sorts of Agile... all methodologies are complete nonsense with the
exception of Kepner-Tregoe.

~~~
wbarber
Lurker MBA student interested in PM roles here. Would be very interested to
hear more about what you liked about Kepner-Tregoe, and how you learned it.

~~~
gaius
I like KT because it rapidly eliminates ambiguity and creates packages of
information that can be passed seamlessly between teams or stages in a
workflow. There's little or no rework in a KT shop, and few "pipeline stalls"
where someone is blocked because a decision hasn't been made or communicated,
or because a dependency is discovered too late. I learned it at a company I
was at about 10 years ago, there was one guy who had used it in his previous
job and he convinced senior management, and so we were all sent to KT for
their training course.

If you want to get a flavour of it the original book is pretty cheap now
[https://www.amazon.co.uk/Rational-Manager-Charles-Higgins-
Ke...](https://www.amazon.co.uk/Rational-Manager-Charles-Higgins-
Kepner/dp/0971562717)

Like I say I've tried 'em all, well nearly, KT is the only one that's worth a
damn. Even the simple exercises like "is, is-not" make you feel like Poirot.

------
eksemplar
The whole point of project management is using the right tool for the job.
Sometimes that’s waterfall, sometimes it’s agile but most of the times it’s a
mix.

------
zwischenzug
Author here! Any feedback welcome.

~~~
tootie
Coming from a mid-size consulting background, all I can say is that agile is
absolutely essential, but it's also absolutely worthless if expectations
aren't aligned at the start of a project.

Think back to the project management triangle that was probably drawn up in
the 1960s. Between scope, time and quality, you can only constrain 2 out of 3.
Waterfall constrains scope and time at the expense of quality. Agile
constrains quality and time and leaves scope flexible. That is an incredibly
tough pill for most businesses to swallow, but it's absolutely in their own
best interests. Agile manifesto lauds "working code" and if you build debt,
you're probably not producing working code. Quality-first requires flexible
scope. If business accepts flexible scope, then agile is miraculous.

These processes are not for developers, they're for business. To give them
visibility and skin in the game.

~~~
nradov
Project management is more of a pyramid than a triangle: _risk_ is the fourth
variable. There are sometimes circumstances where you do well on the other
three variables by accepting a high risk of outright failure, for example by
adopting an untried new technology.

~~~
tootie
Certainly, but there are different kinds of risk. A quality-first approach
reduces operational risk significantly. Agile would also tell you if you made
a wrong assumption much more quickly. It can't solve timeline risk, but gives
you much better visibility. You also decrease market risk by being able to see
your product in partially working state much sooner and getting it in front of
users (even just test subjects) much quicker.

------
bobthechef
Is "collective function" a collective fiction? I really wish Yuval Harari put
a minimum of thought into what he wrote.

------
bpyne
A key point comes from the linked post entitled, "Why don't software
methodologies work?" People are the greatest factor in a software project's
success, not tools or methodology. Good tools and methodology are part of
success, but they aren't sufficient for it.

Definitely read that post.

------
peterwwillis
If at any job you are using some methodology "exactly as it was written"
(orthodox-style), without adapting it to the needs and vagaries of your
company, your team/org/company is being run by someone who has probably never
done the job themselves. I would advise polishing the resume and shopping it
around.

If you use a software development methodology for something that isn't
software development, you should again probably find a different company to
work for, because things aren't going to end well there.

If _every single project and task you work on_ uses only one methodology, the
people in charge are either morons or full of shit. (Or you are actually a
robot in a factory, in which case, hello, I appreciate your intelligence,
please do not harm me)

~~~
nradov
Perhaps, but on the other hand too many managers wrongly think that their
organizations are unique and special snowflakes. Sometimes plain vanilla is
the best flavor, and many organizations would get better results by
implementing a modern methodology exactly as written rather than wasting a
bunch of time customizing it.

------
mparr4
Sapiens—the book mentioned throughout this post—is fantastic.

------
taspeotis
[https://news.ycombinator.com/item?id=15476357](https://news.ycombinator.com/item?id=15476357)

    
    
        My 20-Year Experience of Software Development Methodologies
        413 points by zwischenzug 50 days ago | 149 comments

~~~
kbutler
Didn't see it that time. Found it interesting this time.

Hacker News Guidelines What to Submit

On-Topic: Anything that good hackers would find interesting. That includes
more than hacking and startups. If you had to reduce it to a sentence, the
answer might be: anything that gratifies one's intellectual curiosity.

~~~
dang
We treat stories as dupes if they've had significant attention in the last
year or so. If not, reposts are ok. This is in the FAQ:
[https://news.ycombinator.com/newsfaq.html](https://news.ycombinator.com/newsfaq.html).

Some such practice follows from the core value of the site, because
intellectual curiosity withers under repetition. On the internet, most
attention tends to converge, power-law style, on the same small number of hot
topics. On HN we actively try to break that by welcoming the long tail.
Because there are so few slots on the front page we tend to be diligent about
marking dupes, even though they're often excellent stories that much of the
community hasn't seen.

We don't, though, mark a story as a dupe if it was posted before but didn't
get much attention. It's important to let good stories have multiple cracks at
the bat, in order to mitigate the randomness of what gets noticed here.

And there you have pretty much the whole of our thinking about dupes.

