
My 20-Year Experience of Software Development Methodologies - zwischenzug
https://zwischenzugs.wordpress.com/2017/10/15/my-20-year-experience-of-software-development-methodologies/
======
sytelus
There is an insider story about how these methodologies comes about. So there
are few groups of people whose sole job is to do consulting on
failed/late/over budget projects. Mind you, they _don 't_ write code but
rather they observe how things are going and then prescribe process/management
improvements (McKinsey style). Once in a while, these folks bump in to
terrible projects and whatever they prescribed sometime works like a charm. In
that case, they take that prescription on road and advertise the hell out in
conferences, magazines, blog posts. Unlike regular developers, they have all
the time in the world to do these activities. They write books and give
interviews and by the media power of distributing information suddenly they
pop out as process gods who knows how to fix any project. Eventually the new
things starts to fad, people realize what works in project X didn't worked in
Y, speaker engagements starts drying out and then these folks need new thing
to repeat the cycle.

The obvious problem is that these folks prescribing the development process
are not active developers. They are not even part of any real project over any
long duration. They are in job of inventing and selling processes and handing
out management advice as consultants. Whatever they prescribe, might have
worked only in specific context and for specific symptoms, usually with huge
dose of luck. Next time when you see new process fad, look up the history of
originator of this process, what company he is part of, how much code has he
written, how he makes money. You will know what I'm talking about.

~~~
mmcnl
This is not true for agile. The agile manifesto was in fact drafted by
software developers. Also, agile doesn't prescribe any process at all, in
fact, it does the opposite. Agile in essence is quite beautiful, unfortunately
it gets twisted and turned upside down until it's just another methodology
(which is exactly the opposite of its original meaning).

~~~
cc81
I actually like vanilla Scrum and think often you need a formalized
methodology for the team to follow. I've worked with a lot of different teams
and there are a few things I've noticed being pretty common among developers
(or sometimes being guilty of myself).

1\. Not working on what you are supposed to be working on. Sometimes some
developers feel that they should probably rewrite X or add Y and does so
without consulting anyone first. Something it is good and sometimes it just
takes time or adds regressions you did not need at the time.

2\. Senior developers owning a piece of the code base. Often grabbing a huge
chunk of work for themselves on it and then you don't hear from them for a
while when they hack away. Sometimes they do good work and sometimes not and
if they are sick or away and no one knows how their code works.

3\. Junior (or some senior) developers being stuck for a long while without
asking for help.

4\. Ignoring the customer and building what you think should be built.

None of those are automatically fixed with Scrum and I know some here will
just say "Yeah, we have a professional team that actually talks with each
other and does code reviews" so they don't need it. And I get that but for a
lot of those small teams building some CRUD application at a large enterprise
the formalized communication ways are a god send in my experience.

~~~
piva00
I agree with you that a formalized methodology is required for you to start
with agile, be it XP, Scrum, Kanban or whatever the flavor the team itself
feels more comfortable doing.

What no one tells these teams is that the methodology/process itself should be
flexible, should be allowed to change to accommodate the idiosyncrasies of
their work environment. What I've noticed helping some small companies is that
teams learn some framework and use that as their "agile" concept and then
proceed to be overzealous of the process itself, completely forgetting the
original manifesto.

Like always, it's a strange whisper game...

~~~
gregmac
Adjusting the process literally _is_ part of Scrum. At the retrospective, if
something isn't working, you call it out and change it. That can be something
as big as completely changing the methodology. I've never worked on a XP team
but my understanding is formally there is no retrospective process - I would
definitely want to put one in place if I did work with it.

Any development process that doesn't allow for changing itself is shortsighted
and ultimately self-defeating.

~~~
doseofreality
Except at most companies that use Scrum, there are very rigid rules sent down
from above regarding the implementation of Scrum. These rules apply to the
whole company and can't be questioned or changed via retros.

~~~
spost
This is definitely the case for me. I work at a reasonably large (~5-7000
total employees) company that had a top-down Agile Mandate imposed about a
year ago. It’s worked reasonably well, honestly, but there was a lot of
chafing initially, and there’s still some resistance. Problems with the
process are occasionally solved by abandoning it entirely, with mixed results.

~~~
PaulKeeble
The thing to be careful of is that a lot of developers also don't like certain
parts of agile, and those don't get used. They may have a preference to
continue to use long cycle approaches (code reviews) where a shorter cycle
approach is now necessary (pair programming). The end result is they pick and
choose the bits they want and end up failing to ever do the process even once,
not because its impossible but because it isn't their preference. They didn't
do it out of knowledge, infact they failed to see the interplay in the
processes working together and how the entire thing feeds together to produce
the whole.

I think most teams drop agile practices before they have knowledge to show
they aren't appropriate, they usually never get used at all. They adopt what
they have to because they don't really want to do it at all because they
consider a large chunk of it doesn't work.

Changing the process from a position of knowledge and experience on it is one
thing, doing it without that knowledge is the more usual thing and its
harmful.

------
richie5um
Really liked this article.

I liken software methodologies to cooking recipes.

On one end of the scale you have a highly skilled and experienced chef, who
can craft a great (requested) dish with a variety of ingredients and
equipment. Knowing how to adapt and what to add and when. This works well, but
requires a heavy cost (experience) and is, from the outside, difficult to
follow and control.

On the other end of the scale you have fast food. A strict process and fixed
ingredients that ‘just works’. Those involved in cooking have very little
knowledge of why, and are almost certainly not able to adapt. This works well
for scenarios where you exactly know and control both the input and output.

A skilled chef can ‘codify’ their recipes, but this results in a specific
description of what to do, and not why. Thus marginalising the adaptability.

In my experience, most software engineers are trying to attain the skilled
chef role. While most _managers_ want the fast-food style predictability. With
the ‘process’ being the battleground.

The best process is where you - a complete multi-disciplinary team - make as
quick iterations as possible, learning as you go. But, always knowing where
you are trying to get to (and why!) and constantly adjusting accordingly.

Good luck out there.

------
kenshi
Methodologies are a trap. The problem with any methodology is they tend to
lead people to stop thinking about the context they are operating in.

"The methodology says X, we should do X, why oh why aren't we doing X?" is a
familiar lament in the development world. I have been there and done that, to
be sure.

It really isn't going to matter what the Methodology says, if something or
someone in the context means it isn't being adhered to.

If the person ultimately paying for the project isn't convinced by the
Methodology, or wants to deviate from it badly enough, guess what? The
Methodology will be ignored.

You should always advocate for what the Right Thing To Do(tm) is, but you
should ask yourself first: what is the most important criteria that determines
what the Right Thing is?

 _Sometimes_ writing the quickest, hackiest, throwaway code can be the Right
Thing. It can be a horrifying truth for a software developer. Sometimes doing
the ballsy re-write is the Right Thing. Sometimes letting a project fail is
the Right Thing.

Sometimes last weeks context and decisions no longer apply.

More critical to project (and personal) success than any Methodology is
gaining a thorough understanding of the context in which you are working in,
as quickly as you can. And then doing all the Right Things that will make you
effective within those constraints.

This means there is no Single Truth for software development. No easy answers.
But there are a wide variety of principles you can draw from and apply and
_discard_ as needed.

Or as Bruce Lee put it: Be like water.

~~~
jrs95
What’s worked best for me is basically ‘agile’ but without a lot of what I
consider to be the bullshit. Project manager and/or team lead prioritize new
cards as they come in, and developers just keep working on them. No sprints,
just keep pulling work out of the backlog as needed. Stand ups and most
meetings can be replaced with quick updates via Slack, otherwise they can be
organized as needed (they usually aren’t). We had agreed that estimation was
mostly a pointless waste of time, so we stopped doing it.

~~~
mmcnl
This could work, but it relies on highly competent autonomous team members who
trust eachother. Often not the case.

~~~
jrs95
It _should_ be the case though. Competent, autonomous, and trustworthy is a
pretty low bar.

------
ozmbie
I've worked in a variety of environments, and the most important thing in my
experience is simply to ensure that your team is creating a working, tested,
and "shippable" update every ~2 weeks. It amazes me how many teams fail to do
this, despite calling themselves "agile" and "customer driven".

Use whatever methodology, technology, and processes you need to accomplish
that singular task. Orient your entire organisation around doing that. Your
customers don't care how you do it. They just want frequent updates and
progress. And if you're not giving them useful updates (you'll know, they'll
tell you) then the problem isn't methodology, it's probably your ideas. No
methodology will fix that.

~~~
maxxxxx
This only works in certain kinds of projects. I often work on projects where
you need to spend months figuring out some technologies and see what works
best. For a while we tried to deliver "shippable" updates every 3 weeks but
this just created a lot of work creating artificial prototypes without any
real benefit so we gave up.

If you work on some web site where the tech is well understood then shipping
something every 2 weeks probably works.

~~~
silvestrov
> figuring out some technologies

That is research, not software development.

For software development you know your tools and how to use them, and only
what the resulting product should be is slightly unknown.

~~~
maxxxxx
I call myself a developer but I often work on stuff where at the start we
don't know whether it will be desktop, web, mobile or whatever. We don't know
which database will be best so we try a few. I guess it's research but it
feeds straight into development which is done by the same people within the
same project.

~~~
zo1
>" _We don 't know which database will be best so we try a few._"

There is but a handful of valid choices for a relational database, and all of
them pretty much able to handle every single thing you can throw at it.

Or are you just trying out "flavors" of database? In which case you're wasting
the client or employer's time/money, in all honesty.

~~~
maxxxxx
I work in medical devices. We can store data directly on the device, on some
other device, the cloud or our data center. All have their advantages and
disadvantages regarding security, maintainability, battery consumption and a
lot of others. There is no clear answer up front so you have to try a few
approaches and see how they go.

------
dmytroi
Nice post! I worked in a few big and small companies, and from I can see
software companies with more then ~40-50 devs don't scale well (I only talk
about about single-ish product software companies here). The reason for this
is in my opinion is absence of financial liability of devs: if you're a dev in
a company with 500 other devs - most likely you have zero financial liability
to produce great code, on other hand you have all time in the world to create
job security code around you. Also if there is 500 devs on one project, most
likely one will see a lot of infrastructure just to get this devs busy with
something. This could be seen in companies were one department creates "the
framework", and others departments create "features" based on this framework.
In this case there is no "market" of frameworks - you basically stuck with it,
and "the framework" has zero market competition. Compare it to normal market
of middleware - if your framework is bad, I'll just go to your competitor.
Free market is much healthier situation in my opinion (could be a strawman
argument though). I am not wise enough to say that creating internal
competition inside one company is always good thing, though financial
contracts make a healthier political discussions.

~~~
emodendroket
I don't believe job security code really works. If they're in a place where
they're not paying any attention to you committing awful code why would they
care when developers tell them that only you can work on your code?

~~~
humanrebar
The job security comes when you write great code but awful designs and
architectures. The resulting objections are more abstract and harder for
project managers to really understand. Or they'll understand the objections
but not know how to weight their severity.

~~~
emodendroket
They'll just ignore them and do what they please, is where I'm going with
this.

~~~
dmytroi
Yes, but it's not as obvious as it sounds. For example let's say we have dev A
and dev B and we give them same task. Dev A solves the task in such way that
we don't need to come back to the solution at all, it "just works". While dev
B solves it in such way that now we need 1+ devs maintaining a solution full
time. Any solution that needs to be maintained full time is a form of job
security. Sometimes it is needed (like placing a small team on devops),
sometimes it is not (like in-house full time development of dashboard
frameworks).

------
shoover
I’m surprised that Don Reinertsen’s book The Principles of Product Development
Flow doesn’t come up in these discussions. Rich Hickey mentioned it once after
a conference years ago and that’s the only time I’ve heard it mentioned. I’ve
since found a few blog posts responding to conference talks (of which there
are several on YouTube from around 2015), but not many. I think people simply
don’t know about it. I wrote it down at the time and finally digested it over
the past several months. It’s an incredible book that I think would resonate
with many of the commenters here.

I think of Reinertsen’s approach as industrial agile, in the most honorable
sense of industrial. There is theory and some math. There is heavy emphasis on
project economics as a means of making complex trade-offs and a basis for
training, deputizing, and expecting employees at all levels of an organization
to make better decisions on the fly as part of doing their jobs. Lessons are
drawn from many fields, such as network communications, queueing theory, and
the Marines. Contrasts are drawn with manufacturing to explain exactly why
some of the techniques we adopt do not apply in product development but where
many techniques from kanban systems do apply.

The book is organized around hundreds of principles given as tools, to be
considered and applied as needed to your specific situation. Annoyingly, he
refuses to get into implementation. At first I thought he was saving pages,
but the more I think about I realize the seeming omission was necessary
genius. Digging into even one example implementation would result in the death
of the message and framework as people copied that and get dogmatic about a
methodology all over again.

The whole presentation is beautifully constructed and _tight_. It puts a
perspective and language on things in a way that makes me hopeful and
interested in project management, similar to how Rich Hickey’s talks raise my
understanding, vocabulary, and aspirations for software engineering.

I would welcome anyone involved in prioritizing commercial software projects
to read and discuss this book (and document their implementations).

------
mmjaa
My 30 years of Software Development Methodology:

* The user is more important than any of this. * The programmer is nothing. * The computer is doing all the work.

Thus: use what makes all the above, True.

~~~
icebraining
What does "the programmer is nothing" mean?

~~~
tluyben2
And "the computer does all the work"?

~~~
georgeecollins
I'm guessing: The machine does what the user (client and org) expects
effectively in a way that they easily understand.

------
baybal2
The biggest skill when interacting with a "big serious man" type of client is
to making him pay you for what you want him to pay you for.

In my circle, we have a lot of discussions on clients like that: "and I told
him, for what freaking thing do you need that freaking neural network when a
banal statistics does the job better?". Having such guy, "a Wharton grad with
stellar track record at the big 4" agreeing on the former requires
compromising his ego, self image, exploiting him wanting to not to look
incompetent in eyes of his superiors.

In the end, all such guys are made to pay 6 or even 7 digit sums for a fancy
UI over database querying tools and a stats package, and not a "big data,
machine learning blah blah blah," or otherwise being given "the exact thing
they wished for" that had no chance of even minimal functioning

~~~
jacquesm
But how will his company make it to 100M+ valuation if he can't claim it is
being done with AI?

------
z3t4
The fiction I like best right now is "continuous delivery": You release often,
push to production several times per day, one working branch, always ready to
deploy. Features being in development are behind flags. Regression tests help
a lot, eg, when you make an update or fix a bug you make an automatic test
that checks if the new thing works, or repeats the steps that caused the bug
and makes sure it's no longer there.

------
tluyben2
We use 'waterfall' (detailed, fleshed out specs in formal language) on the
hardware, firmware & backend and 'agile' on the frontend (web/app). So far
that served us well and it addresses the difference in developer skills _and_
needs as well. With the danger of generalizing; frontend people are different
from backend people who are different from hardware people (in my experience),
and this gives all what they enjoy working with.

~~~
_puk
Hardware (and firmware to a lesser degree) almost require waterfall. Moving
fast and breaking things doesn't really work well when you have to throw away
the physical product you broke as opposed to iterating it to a fixed state.

I can actually see the merit in waterfall for elements of the backend.. though
how do you avoid pre-optimisation or even complete wastage building out APIs
that end up never being needed?

------
zwischenzug
Author here: interested in any strong views on this - @ianmiell on twitter.

~~~
Jare
Me and my brother wrote our first commercial videogame in 1987 as a 16-17yo
pair of self taught programmer/artist/designers. We just did what seemed like
common sense.

\- We had periodic meetings with the client (publisher). we showed them
progress and they suggested directions to move towards

\- We didn't stick religiously to the initial design, and adapted as we saw
things work or not work to make the game fun

\- We stuck to the deadline and just cut what didn't make it

It felt quite agile to me.

~~~
georgeecollins
That sounds like heaven. What was the game?

~~~
Jare
Erikbye is correct, Stardust for Topo Soft on the Sinclar Spectrum and Amstrad
CPC 8-bit computers.

I attribute a lot of the merit to us being mentally organized, but mostly to
our producer Javier Cano (RIP), who had that magic touch for ensuring that
things around him just ran smooth, always with a smile and making it feel
natural. My god he was amazing.

------
GrumpyNl
All these methodologies are there just to get things done. After many years
involved in software development i have find out that it all comes down to
leadership. Of course you need people with skills, but no good leader, no good
product.

------
zyxzevn
I like the car analogy, and I had additions to it with software-components in
mind:

engine-choices representing data storage: very big engines. for big ships.
very small ones, for lawnmower. self-made ones, smoking.

gear-choices representing libraries: single gear gear from 1 to 1000. No
neutral and no reverse.

chassis choices (language choice): rusty old ones, hard to work with, but
lasts. new shiny ones, overpriced and falls apart.

locks = safety: one key fits all (all doors and all cars). different key for
each lock, can be bypassed by opening window. keypad with password, written on
the back-side of the car.

interior = user interface: millions of choices, all flashy and shiny, each
part looks totally different. Unusable steering.

break (software stability): no breaks. when used causes blue-screen

outside (program output): all parts have different sizes. holes covered with
duck-tape and rust covered with shiny paint.

Enjoy the ride!

------
hshhbd
I really enjoyed reading this and I strongly agree there isn't an easy task at
all. Depends on the business, people, existing processes, it's up to us to
find what really works. I had some thoughts based on my recent experience and
went into a mood to write something quite long on my LinkedIn
[https://www.linkedin.com/pulse/waterfall-scrum-kanban-
scumba...](https://www.linkedin.com/pulse/waterfall-scrum-kanban-scumban-
whats-next-richard-shenghua-he) if anyone interested.

~~~
hshhbd
Sorry, had a typo in the previous link, please use this one
[https://www.linkedin.com/pulse/waterfall-scrum-kanban-
scrumb...](https://www.linkedin.com/pulse/waterfall-scrum-kanban-scrumban-
whats-next-richard-shenghua-he/)

------
jeff6845
I've not found a methodology yet that can displace smart decisions.

Methodologies organize projects to help further smart decisions and avoid dumb
ones, but smart decisions and skill still dictate the success or failure of
projects.

In the end, the people are still more important than the methodology, but I've
encountered a mentality a few times where people believed a good methodology
will fix all.

I've seen companies keep changing methodologies looking for success with
largely the same staff, and key decision makers, but yet the results are the
same. Insanity, anyone?

------
emodendroket
The style he describes as RAD is really enjoyable to work in. No constant
scheduled meetings and unread documents; just use your common sense and write
something.

------
baybal2
There are no need for formal methodologies. A person with no technical
expertise should not be but on a position managing technical teams. That
requirement divides the industry on "tech companies" and everybody else.

The example of a tech company with gigantic dev teams and very little number
of MBA doctrinaires involved in development process is Intel.

Companies with external development clients that practice "we will agree on
any methodology you want for right amount of money, but we would not let you
anywhere close to our developers (who will be doing their own thing anyways)"
are Tatas, Wipros, Luxofts and others. Their biggest asset is their skill at
managing the client.

~~~
walshemj
Ah and how would say produce a management system for a core part of a
country's core data network (I was a team member developing a management
system for the UK's SMDS network) with out a process? - In this case we used a
hybrid waterfall /RAD system 12 weeks to develop the fist release

~~~
baybal2
The very very few cases where anything called a methodology works is when you
deal with a repeating tasks: making a customized versions of standard tools,
or what people call copy&paste development. SAP does it very well. Charges
people 7 digits for "customized solution" that in reality is not customized
beyond 20 lines of code from what the company sells to every sixth or seventh
client.

The goal there would be to not to dive too deep into the task that is already
solved in a way that "works well enough," and for what the client is already
ready to pay money for.

~~~
walshemj
Not everyone always works on me to crud applications leaving aside SAP and
other "miracle" tools :-)

------
whipoodle
I work for one of those fancy startups. Everything is chaos all the time, you
have all these extremely smart people in the small doing actually kinda dumb
things in the large. This worked better when the company was small and is
causing more and more problems as we grow. Anyway, the point being that
there’s lots of smart individuals and little process.

I mentioned this during a meeting with my team, and so a process was devised
for the next project we embarked upon. The process was waterfall, except
nobody referred to it that way (most of the people are like 25, and weren’t
working back when it was the norm). Everything old is new again.

It wasn’t important anyway; we didn’t follow the process.

