
Software Maintenance by Skeleton Crew Is an Anti-Pattern - 56quarters
https://18f.gsa.gov/2016/02/23/software-maintenance-is-an-anti-pattern/
======
steven777400
I work for the government (and perhaps more significantly, have never worked
as a developer outside of government), so it's difficult for me to understand
the ideas and concerns promoted here. Just as in private industry (according
to the article), our first versions are MVPs which have a roadmap for future
additions and improvements. Based on funding, those may or may not result in
an increase in team size.

At some point though, the team will decrease in size and the software will be
considered "complete for now". Generally, that software will continue to be
"on life support" (I guess you could say that?) for the next 5 - 20 years.
That isn't a bad thing. Some of the projects I maintain have code from the
mid-90s that I'm still making incremental updates to. Sure, I'd like to
rewrite it in something newer, but it isn't about me, it isn't about fancy
shiny. It's about the fact the system as it exists is proven and solid and
meets 90% of the customers stated need. "Whack-a-mole" changes are just a sign
of poor testing, poor architecture, or lazy developers. All of our teams here
make incremental changes on aging software and generally don't experience
major regressions except when it's time to do the big rewrites (which are
usually customer-initiated when they bring a raft of changes and some money).
When existing technologies are so obsolete as to be unsupportable, then we
(IT) will initiate a rewrite as well, but hopefully with incremental upgrades
we can keep a product running for as long as the customer likes without
spending any more money than they want to spend.

~~~
sarahallen
Steven -- I applaud your team's approach. From my two and a half years in
government I have rarely run into projects like you describe. We have lots of
legacy systems, typically built without automated tests. Often a vendor is
hired to build a "complete" piece of software, then given a meager budget to
"maintain" it.

~~~
steven777400
Thanks.

We have a lot of that issue of "hire contractors to build, then use FTEs to
maintain". I try to push back where I can because contractors that know up-
front that they won't have to support the system are often not inclined to
either follow standards (which are less exciting, I know, but make maintenance
more predictable) and/or not be concerned about avoiding excessive technical
debt up front.

------
dmlorenzetti
It may be worth noting that some government-sponsored software is intended to
get dropped as soon as version 1 gets released. Of course, this is a small
fraction of government-produced software.

This happens for a variety of reasons. Often funds are allocated specifically
to develop what industry calls a minimum viable product-- with the
understanding that if the project is deemed successful, the research group can
seek further funding to improve the tool.

There is also a model, in government research funding, that says the
government's role is mainly to point the way, or to open up markets. In this
model, if version 1 of the software proves itself useful, then an industry
partner can step in and it take over. However, if nobody shows interest, then
the government should _not_ sustain the effort, because the lack of interest
implies a lack of utility to the public.

For example, the OpenEIS project [0] was funded specifically to produce
"reference code" that a company can use to check the output of a professional
implementation. The reference code is not at all intended to be fast or
robust. It's intended to demonstrate the algorithms, and to document the
expected output for a variety of reasonable inputs. As a programmer on the
project, I knew from the start that my work would end with the first
deliverable.

[0] [http://eis.lbl.gov/openeis.html](http://eis.lbl.gov/openeis.html)

------
xigency
Without any concrete evidence or cases, the general description of a problem
in this article is not very convincing. Software maintenance itself is not an
anti-pattern. Instead, what is described is a situation of product/project
mismanagement.

    
    
        > ... software is not considered complete when it's first released.
    

I don't know where this assumption comes from.

    
    
        > Governments often use two anti-patterns... (2) moving to reduce 
        sustaining staff too early;
    

This seems like a complete description of the problem, which doesn't sound
like a pattern but an issue of conflating deadlines, releases, and project
completion.

    
    
        > To keep teams from constantly plugging leaks while missing 
        the reason the boat is sinking, we need to apply the
        principles of agile development and user centered design
    

Honestly, this sounds like the waterfall model is preferred to agile
development in this case. Agile development can more easily lead a team into
constantly plugging leaks than other, more traditional development models.

But I think this article is really addressing public facing web platforms
instead of all "software," which the US government has only really
experimented with during the last administration, so some of the sentiments of
this article, in the context of government, may be true even though it reads
false.

Beginning with the analogy to building construction: A skyscraper isn't
complete when just the floors and frame are put in place, and the building
team doesn't let the builders go at that point. When it is opened up, the
maintenance staff have their own job, but it's nothing like what renovators
might have to do years later.

~~~
tremon
_> ... software is not considered complete when it's first released.

I don't know where this assumption comes from._

My anecdata, doing IT support for about 1,000 small-business sites: it's
customary to only adopt new Microsoft releases after SP1 has been released.
Our customers sometimes even explicitly instruct us so.

~~~
xigency
While this is changing, in game development the finished product ships on
read-only media and without patches or online support, that is all the
customer is going to get.

Now there's a prevalence of day one patches but it's pretty lame to see
developers waiting till the weeks between pressing the disc and it appearing
on shelves to fix standing issues.

In the case of M$ software that's a security or support issue not the absence
of core features.

~~~
azernik
Already in 2013 92% of PC game sales involved no physical media
[[http://gamerant.com/pc-digital-physical-sales-
comparison-201...](http://gamerant.com/pc-digital-physical-sales-
comparison-2013/)]. This enables the day-1-habit, of course, but it also
mitigates the problems - there aren't any shelves on which unpatched versions
of the game keep sitting after launch.

~~~
acveilleux
Console is lagging behind for this.

------
mcculley
This seems to assume that all "private industry" does things in a way
different than how government does.

Different kinds of projects have different needs for upfront design. One can
quickly ship and iterate on an online dating app. One cannot launch a new
space shuttle every two weeks and keep trying until the crew survives the
mission.

~~~
WalterSear
>This seems to assume that all "private industry" does things in a way
different than how government does.

It does. A skeleton crew would be abandonware.

~~~
mcculley
I don't understand your reply. I'm familiar with private industry projects
producing high-risk projects requiring lots of long term up front investment
in capital and time and private industry projects working low-risk efforts
where using the customers as testers is perfectly fine. Likewise there are
small units of government entities that can iterate on low-risk projects in
addition to the big infrastructure projects that larger government
organizations work on.

I think the OP is painting with too broad a brush when talking about "private
industry".

------
jcadam
I've worked O&M on software in the defense industry. Never again.

On my current project, when development is done and we're officially
transitioned over to "O&M", my employer either gives me a new project to work
on or I'm leaving to do something else.

Oh, and I've come to the conclusion that civil servants are not capable of
grasping any development methodology other than 'waterfall.' Trying to do
'agile' is an exercise in frustration. The bureaucrats say they want agile
(buzzword!), but they really don't. Eventually you settle on doing 5-week
'mini-waterfalls' and calling it 'agile'. :/

~~~
enraged_camel
It's not just civil servants who can't understand agile. Most non-software
people can't. Which is totally normal and expected if you think about it: the
agile movement started in software. You can't build a house or a plane using
agile. It has to be waterfall.

~~~
dragonwriter
> It's not just civil servants who can't understand agile. Most non-software
> people can't. Which is totally normal and expected if you think about it:
> the agile movement started in software.

Agile is (to the extent it has concrete substance) very similar to Lean, which
(while it has come to software) actually started in manufacturing and
engineering of tangible products. The difference between the two is mainly
that Lean has a much stronger culture (and has thus also produced a lot more
tools) around validation of methods through measurement, and thus is a lot
more woo-resistant than Agile, which has proven to be decidedly prone to
devolving in exactly the same kind of one-size fits all, consultant-pushed,
top-down methodologies without good feedback on what works in the particular
environment that the Agile Manifesto was a response to.

> You can't build a house or a plane using agile. It has to be waterfall.

"Agile" and "waterfall" aren't opposed, agile is opposed to the idea that any
one methodology -- Scrum as much as Waterfall -- can be selected as right for
a team without reference to the particular team and context in which they are
working. To the extent that it works in a particular context, waterfall _can
be_ Agile.

~~~
enraged_camel
Agile is the idea that requirements are always in flux, and therefore it's
pointless to spend time upfront gathering requirements and writing specs and
following those specs since there's a good chance they will have become
irrelevant by the time the product ships. So agile says "just get something,
_anything_ , out the door as soon as possible, receive feedback, then keep
revising."

Waterfall is the opposite approach, where you have a formal requirements
gathering phase, then you commit everything into design docs, and follow those
docs with as few changes as possible. It's similar to how a house is built:
first you check your local zoning laws, then have an architect draw the plans,
then dig a hole and build a foundation, then build the skeleton, the walls,
etc. You don't tell the client, "hey, we just dug the hole for the foundation,
come live in it for a few weeks and give us feedback so we know what part to
build next."

------
manarth
This could be applied to any software that is developed as a product (in the
sense of a piece of software to sell to multiple players in a market).

The article talks about the demand/market for the software _increasing_ after
the release, until it reaches a peak, then tails off.

Agile uses the term 'Product Owner', but if you're developing an intranet for
Acme Inc, is it really a product? Once the new intranet site is released, is
Acme Inc suddenly going to hire a lot more people, increasing the "market"?
Will new features generate more "profit" for the intranet?

I wouldn't describe this as Gov vs Private Industry, but as software products
vs bespoke software development. And for bespoke software development, private
industry often has the same approach to maintenance: once it's done, it's time
to wind down the team.

~~~
boomlinde
_> Agile uses the term 'Product Owner', but if you're developing an intranet
for Acme Inc, is it really a product?_

Approaching this from a purely semantic perspective: yes, a product is
something that is produced as a result of a process. Not "in the sense of a
piece of software to sell to multiple players in a market", but that's a
completely arbitrary distinction that you drew yourself, so it isn't really an
argument against calling someone a "product owner" of a single customer
product.

 _> Once the new intranet site is released, is Acme Inc suddenly going to hire
a lot more people, increasing the "market"? Will new features generate more
"profit" for the intranet?_

Maybe Acme Inc find that the product that they have paid for have benefits to
productivity, but are willing to pay for continued development to refine it
further?

Having worked on a lot of "one-off" type projects I think that it is important
to maintain software gracefully. Existing customers will ask us to roll out
new features on short notice, and having built the product in a way that lets
us adjust and add to it easily enables us to sign deals at competitive prices.

That's not to say that this always happens, but I am less suspicious of this
as an ideal.

------
frozenport
Maybe the problem is that development is decentralized and done off-site? For
each each website, or component is built by separate, often lucrative
contract. On one hand developers re-invent the wheel knowing that they
financially benifit from the bloat, on the other hand they feel little
responsibility for their often weighty work.

Can we do something crazy like centralize our software development or have
these administrative positions electable? If you can run for town dog catcher,
maybe that asshole who made the WordPress site should also stand for office!

------
vinceguidry
I don't see the point of shoehorning technical concepts onto managerial
problems. All it's doing is confusing the point. It's like the author just
learned about the concept and now wants to re-imagine _everything_ in terms of
that concept.

------
hackaflocka
Peculiar that the author makes very strong statements without any proof of
research findings.

------
amelius
If users have put up with bugs for this long, why should the system be
improved?

Either fix the bugs quickly, or don't fix them at all, and move on.

------
peter303
Or the "continuous release" model of browser-based and app software. Tehy seem
to updated automatically nearly every week.

------
jxramos
This title is misleading, should be closer to "Government frequently adopts
anti-patterns when maintaining software" Something more constrained to what
the article is actually about rather that the global catch all "software
maintenance".

~~~
smarks
I agree that the title is misleading, but I'd clarify it to "Software
development _as typically practiced in industry_ is an anti-pattern." (Maybe
it also applies to government; I don't know.) It seems fairly typical in my
experience for the old thing to be continually patched over instead of fixed,
awaiting replacement by the new thing that is never quite ready.

~~~
xigency
In general, "the new thing" doesn't solve problems that patches fix. For a web
platform especially, it sounds like expecting a new engine from a new coat of
paint.

