
The World of Legacy IT Systems - niccl
https://spectrum.ieee.org/computing/it/inside-hidden-world-legacy-it-systems
======
johngalt
I've tackled legacy systems my entire career, and there is a certain art to
untying the knot of dependencies, procedures, and expectations. Not surprised
at all that most projects to modernize end up as smoldering craters rather
than functioning systems.

The legacy system trap is built on three pillars:

1\. Maintenance costs are always lower than upgrade costs. It's even cheaper
to pay for neither. Even if you do spend money aggressively to maintain it;
every feature added, operator hired, workaround published, becomes a growing
suit of armor protecting the system from replacement.

2\. Management incentives are rarely aligned with the planning and sacrifices
it takes to unwind technical debt. Everyone wants to cut the ribbon on the
shiny new big-data/AI/ML/buzzword project, but no one wants to take
responsibility for integrating it with the rusting hulks creaking along
actually running day to day operations.

3\. The software industry can shift paradigms enough that you end up looking
to buy a new/faster horse for your buggy. And nothing fits your use case, or
even offers a path out.

All three in combination mean you get businesses spending obscene amounts of
money on technology they can't use.

It is quite satisfying to successfully maneuver an organization out of the
trap. It's like solving a complex puzzle over months. Where hundreds or
thousands of real people will be better off because of your team's effort.

~~~
waheoo
> Not surprised at all that most projects to modernize end up as smoldering
> craters rather than functioning systems.

This is because people think rewriting is a solution. It very never is. It can
help, rarely, but it's never the solution, it just does a good job of hiding
what the solution was.

If you can't map out what is really broken in your legacy system and map out a
way to refactor it to what you want then a rewrite will always fail.

~~~
jffhn
>people think rewriting is a solution. It very never is.

I often see this idea formulated, in this forum or in articles.

>what is really broken in your legacy system.

Software can be arbitrarily broken (think of only copy-pasting code from
thedailywtf):
[http://www.cs.utexas.edu/users/dsb/CS373S/Images/softwaredes...](http://www.cs.utexas.edu/users/dsb/CS373S/Images/softwaredesign.jpg)

I knew a legacy system where everything was broken. There were no proper data
model and structures, so every computation involved multiple nested loops,
making everything much slower than needed. Everything was modifiable as if it
was public, so it was hard to keep things consistent. Allocation/GC and
latency were through the roof, they tried to add pooling but noone could
figure out where objects could be released. One of the core classes was called
Banana (changed name) but actually referred to bananas or to banana trees. And
the list goes on (threading, database usage, still plenty to tell).

Then someone, new to both the domain and the language, stepped up to rewrite
everything, which took a bit more than a month.

Ten years later the core of it didn't change much, they are still building on
it and it's still smooth.

~~~
waheoo
So sometimes the cost benefit of a rewrite outweighs the risk but I'd still
say all those things you pointed out were the problem that needed fixing,
rewrite really just fast tracked that no?

~~~
jffhn
It was rewrite "from scratch" (not refactoring), but with the previous team
available to answer any question (either about the legacy code or the domain,
which they knew well), which was an essential part of the success (I guess a
lot of rewrite fail because they are done by new people that didn't take care
to learn enough from the old system and the domain to thoroughly understand
what must be done).

The legacy code was too bad to make refactoring worth it, but surely not every
legacy system is in that bad of a shape.

------
jacquesm
Legacy is a special form of good: It means something actually works. Good
software outlives its original programmers, great software outlives its
original programmers and is still maintainable. You can only get to that level
if you make a commitment up front that this software is going to stand the
test of time, update your systems, don't get side tracked by fluff and eye
candy and concentrate on what matters: the core business logic, at the expense
of almost everything else. Do that and you stand a chance of avoiding the
legacy trap.

A year of complacency and you have a year of catching up to do and you'll
never get that year back unless you want to freeze development.

~~~
drieddust
Absolutely I have seen MGMT wearing "innovation" blinders only to realize
nothing fucking was wrong in the first place.

Before declaring something legacy, a fair evaluation of the future business
requirement and fitment of current solution is a must.

12 Years back I worked with an org who had an AS400 which they considered
legacy. System is still running and serves the business need perfectly well.
All plans to replace just wasted money.

Another company decided to implement a new centralized ERP without realizing
that company's highly federated businesses which can also be considered a
conglomerate of very small independent businesses doesn't need it and won't
use it. Million went down the gutter before higher ups realized the sad fact
and fired the CIO.

IT world is literred up with sorry tales like these.

~~~
alacombe
> 12 Years back I worked with an org who had an AS400 which they considered
> legacy

What happens when the hardware fails ?

~~~
p_l
You call IBM using your support contract to bring spares or use the chance to
replace with latest model (they are still updated)

~~~
collyw
I guess this is one of the reasons no one ever got fired for buying IBM.

~~~
p_l
No, it's just normal case of "the system is still in production and the only
reason you're on ancient hardware is because you're not upgrading", same as if
you ran business-critical component on random 486SX assembled by then-CEOs
nephew.

------
readingnews
As a systems admin of some 30 years, I have read this over and over, and even
seen it firsthand. I think the real root cause of (at least in my case, the
ones I have seen) of this problem is management. Leaders do not want to spend
now. Leaders do not want to make a progressive plan to keep up with technology
and move ahead. It costs money, and is very difficult to talk to upper-upper
management about. Telling the boss to tell his boss to spend money on
something "that works now" is very difficult. Few want to invest in the future
when it costs now and is working now (unless its the stock market).

~~~
tabtab
It should be weighed like any non-trivial investment. Present an estimated
yearly cost curve for keeping the system versus overhauling it: two curves.
Upper managers will probably relate to that better than techno-speak.

Typically a replacement system has a nearer-term spike in costs for
development, but is cheaper in the longer run than keeping the old one,
because the old one will need gradually more expensive duct-tape, chicken-
wire, and harder-to-find technicians willing to support something obsolete.

One advantage of the older system is that the costs are more predictable,
because it probably works "good enough" at the time and will likely just
gradually "degenerate". An overhauled system could be great or a grand
failure.

An exception is if a vendor stops supporting a key legacy component and there
is not enough demand for 3rd-party tweakers to take up the slack. Similar can
happen if a security hole is found in a key legacy component.

~~~
adrianN
Companies can't track costs that stem from systemic inefficiencies.

Say your programmers still use CVS for version control and you want to
convince management to switch to Git. The CVS server doesn't cost much, a git
server wouldn't cost much either. There would be a large cost for changing all
your tooling and processes that rely on the CVS model and in training your
programmers to use the new system. Why would you do that? How do you quantify
the benefits of atomic commits or simple branching?

~~~
collyw
> There would be a large cost for changing all your tooling and processes that
> rely on the CVS model and in training your programmers to use the new
> system.

When git seems to be by far the most used system on the market these days,
isn't training any new hires in CVS going to be at least as much of a problem?
I used it in my first job in 2003 / 2004, but I can't remember how to use it
now.

Do companies with legacy code tend to keep staff around for longer?

------
SCHiM
A thing that I found missing is that legacy systems' influence is much larger
than only the systems it directly touches.

Through particular requirements that some of these systems have they might
shape policy too!

I primarily deal in security assessments of networks. Guess what happens to
your password strength requirements if somewhere deep down in a dark and old
server room there is a mainframe that can only handle 8 character lowercase
passwords? Assume that this system has been the backbone of your, then hyper-
modern, completely integrated IAM system for decades?

...

That's right, security policy mandates 'simple' 8 character passwords... Not
ideal.

~~~
tonyedgecombe
That sounds like my last bank. I used a 16 character password but could login
with the first 8 characters and it wasn't case sensitive.

Every time I interacted with them there were hints that things weren't quite
right and sure enough they ended up in the news after a huge IT failure.

------
BXLE_1-1-BitIs1
Well there's this fashion of dumping 50+ IT guys. The systems they worked on
will then stagger on. The new guys will throw on patches, but there's no money
to really understand the system. The life support just gets more and more
expensive.

The smart way to migrate a legacy system is to keep your maintenance crew to
retirement age. Once they're approaching retirement bring in some kids to
migrate to the shiny new tech and use the old crew to make sure they don't
drop stuff.

~~~
phendrenad2
I wonder if in 100 or so years software development anthropologists (do those
exist yet?) will study the effects of ageism in tech using historical
hiring/layoff data.

------
thatguymike
I know that large business systems are complicated, and migrating old data is
much much harder. But a persistent part of me thinks... is this really so
hard? The IRS core system basically requires a decision tree, a calculator app
and several bindersful of business logic. They need to process about 350
million people each year, which is a big but hardly unmanageable Spark job.
Isn't this just a question of getting a team of good experienced Silicon
Valley engineers to build something that just works?

The Obamacare system is a great example of this. Almost half a billion dollars
spent on corporate contractors, then a team of like 10 experienced engineers
came in and fixed the damn thing.

~~~
jacquesm
Every year absolutely mind bogglingly large sums of money are spent on
redeveloping existing applications of that level of complexity. These all have
a very predictable ending: utter failure, and the legacy stuff ends up living
to fight another day.

It is a special kind of blindness: We think that just because our tools and
machines are so much more powerful that rebuilding a complex legacy
application is now childsplay.

From a UI perspective you are most likely right. But it's the couple of
million lines of business logic that will be your Waterloo. In a nutshell:
forget about it. A bit longer: it is possible but only by a _very_ experienced
crew who has nothing else to do under competent leadership and who can chart
an incremental path between where they are now and where they want to be. In 5
years or so. A big bang redevelopment is DOA. Every time.

~~~
protomyth
To add, what makes the business logic so painful is that you don't define the
rules yourself. They are the result of legislation, regulations, lawsuits,
contracts, and preferred work processes of very headstrong individuals. The
codes tries to reconcile all of that to get to a solution. It's not about a
great idea, it's about the laws of physics being written by a committee.

------
peterwwillis
Something I'm not sure is clear from the article is that not all legacy is
bad. The definition of the word 'legacy', essentially, is "something that is
handed down [through generations]". You can hand down good things and bad.

Banks often have incredibly old systems because their products don't change. A
savings account in 1940 is pretty much the same as a savings account today.
They basically have no need for DevOps, because they have no need to provide
fast-moving, competitive products, and silos are a natural fit in a world of
sensitive financial information. So some really old systems are pretty much
fine as-is, as long as the hardware can keep running, and a few over the hill
programmers are on call.

Another example of useful legacy is old physical infrastructure. The Frankford
Avenue Bridge in Philadelphia is a 73-foot long stone bridge that's 323 years
old, and carries a substantial amount of car traffic. But it's also not being
maintained like it should be, and the more-than-expected wear over time is
slowly eroding it. That's the bad part of legacy systems: when you know they
need maintenance and they don't get it, so they rot.

Modern infrastructure can suffer the same fate. Concrete bridges, wooden
houses, cars... they all need maintenance. So do modern computer systems,
where they need to get regular security patches. So the problem is really just
a maintenance problem, and we happen to let older systems go to hell more
often than we replace newer ones.

~~~
tonyedgecombe
_A savings account in 1940 is pretty much the same as a savings account
today._

The way customers interact with their bank has changed massively over that
period. In the UK traditional banks are losing customers to new upstarts that
don't have to bear the cost of all that legacy. They are nimble and much more
responsive to their customers.

------
bm98
Software lifetime is an interesting topic. See for example [1], a 1992 paper
that surveyed business applications and found an average lifetime of about 9
years, with a maximum of 20 years and a minimum of 2 years. 2 years versus 20
years!

Things don't seem much different today.

On the one hand, HTTP, HTML, and producing HTML server-side in a language like
Java or Python and with a SQL database isn't all that different now than it
was 20 years ago. There are Java web apps developed in 1999 that are kept up-
to-date and work just fine in 2020. On the other hand, there are applications
developed with popular technologies a just a few years ago that are hopelessly
out-of-date and in need of a rewrite today.

I see it as an end user too. I've been an ING Direct (now Capital One 360)
customer for 18 years. The web app that I use in 2020 seems to do the same
things as the one in 2002 (though it's slower today), but it seems like it's
been completely rewritten at least three times.

[1]
[https://scholar.google.com/scholar?cluster=23826996884004375...](https://scholar.google.com/scholar?cluster=238269968840043751)

EDIT: And HN! 13 years old and counting...

~~~
adrianN
I'd bet that some of the systems that were 20 years old in 1992 are still
running.

------
peterburkimsher
Photo from the article, colourised:
[https://imgbox.com/YzqSitam](https://imgbox.com/YzqSitam)

~~~
imperialdrive
Many thanks.

------
RickJWagner
It's almost always cheaper (and safer) to stay with what you've got. Right up
until unsupported software or hardware makes it untenable.

~~~
NomDePlum
Too often there is a push to replace what is not actually understood in the
first place.

~~~
alacombe
Which might be a good thing if the loss of knowledge came from employee
turnover. If the architect / author leaves, the visions leaves too. Next step
is monkey patching, fast forward a couple years, a decade or two, more
turnover, and you end up with a indescribable mess.

In this case the replacement is all about re-gaining the knowledge.

------
faebi
How much do legacy systems affect the success of a business? Are there clear
cases where companies with or without legacy systems are doing any better or
worse? I feel like both ways seem to work with the right amount of knowledge
and leadership.

------
7thaccount
The article was fine until it started randomly mentioning AI.

------
MattGaiser
A lot would be gained by treating software like a pet rather than as a
building. Regular brushing and care, occasional picking up of errant poop,
vaccines, checkups with the vet, and the knowledge that Fido will only live
about 10-15 years.

------
info781
A salesman will call anything that is not his product "legacy" so the word can
get confusing. Also there is an issue when commercial vendors start to
overcharge, and their software needs to be replaced with some thing cheaper.
This can become a higher priority than replacing COBAL.

