
The little legacy code that could: a fable of software ownership - icoe
https://circleci.com/blog/the-little-legacy-code-that-could-a-fable-of-software-ownership/
======
larrymyers
This is one of those interesting paradoxes that I haven't figured out yet.

Owning the legacy software that runs the business tends to provide job
security at your current job, but can hinder your professional growth at both
your current job as well as any future jobs.

Getting on to projects that are intended to replace legacy software tends to
get you a lot of positive visibility politically and the ability to learn new
technology and skills. If the new project fails there tends to be little
fallout from the failure, since so many people are attached to the project at
all levels. The goal posts will move so it can be reframed as a success.

If legacy software fails it usually means long hours and a lot of people
nervously asking you when it will be fixed.

Unfortunately there's just not a lot of value to being able to put on your
resume that you're an expert at an older language that nobody has heard of,
built on an in-house framework that will never be used outside of your current
company.

~~~
aeorgnoieang
> Unfortunately there's just not a lot of value to being able to put on your
> resume that you're an expert at an older language that nobody has heard of,
> built on an in-house framework that will never be used outside of your
> current company.

I think this is, fortunately, only _approximately_ true. Most of my career to-
date has involved exactly this kind of work and it's certainly not true that
there's no value in it, even on a resumé or mentioned in a job interview. And
certainly some non-zero amount of people derive considerable value from
exactly that kind of experience as some non-zero number of employers or
customers need someone with it.

And, in a lot of ways, working on legacy systems can be both immensely fun and
rewarding. Even just incremental improvement in or around such systems can
itself be immensely valuable, e.g. setting-up automated builds or deployments,
adding integration tests (to make future refactoring easier and safer), or
rewriting or replacing a key portion of the system using newer tools or with a
better design made possible due to the wisdom accumulated by a system
performing real work over a significant period of time.

~~~
noir_lord
Most of my career has been working on legacy projects (or fixing broken ones),
I like greenfield well enough but I’ve no preference for it over maintenance.

There is as much pleasure in making something broken work properly as creating
the thing (imo).

It’s a steady career as well, most programming is maintenance outside of the
fail fast world of startups.

~~~
on_and_off
Even at my current job in a startup, 90% of the code I own can be considered
legacy.

The startup is 7 yo, I joined almost 2 years ago.

The gap between old and new features is enormous .. and it will take years to
reach some cohesion.

~~~
oaw-bct-ar-bamf
I am relatively new to a project that is being sold and used in the order of
millions for approximately 25 to thirty years.

It was refreshing to find out that one of the approximately 7 languages that
contribute to the final executable that was abandoned in ~2007 has been
revived and is now (2016ish) being maintained by Eclipse

These kind of things make you realise that the Silicon Valley way of doing
things is not the silver bullet.

~~~
on_and_off
Yep .. it is always painful to hear stuff like "if you don't use [insert
whatever is hype right now]; then you are not a real engineer" coming from a
colleague.

People that have spent their careers in the silicon valley have a very
distorted view of tech.

------
Animats
The underlying technology doesn't have to be obsolete to have legacy code
problems. Linux, Windows, GCC, Microsoft Office - all have serious internal
problems from legacy code, and are tough to maintain.

It's not so bad for big projects with enough staff. It gets tough when the
legacy code does something hard, and the maintenance team doesn't really
understand why something was done in some way.

Second Life, the virtual world, has that problem. It's written in C++, and
some very good people wrote it about 10-15 years ago. They're all gone. The
people who maintain it today are struggling to fix serious bugs that have been
outstanding for 5-8 years. It's not just "legacy", it's "not web-like". It's a
distributed system with tens of thousands of servers in one data center. It's
a tightly coordinated soft real time system. There's a huge amount of in-
memory state, which is changeable in real time yet is constantly being backed
up. This is totally alien to people who only know transaction-type web-based
systems. So they can't hire anybody and have them be productive quickly.

~~~
bjt
Doesn't that describe a lot of MMOs too, though? The inventory/transaction
systems seem like they'd have similar needs (but with a LOT more items in the
Second Life database). And the in-world state is a lot more variable than in
an MMO. But compared to MMOs, those seem more like differences in degree than
differences in kind. Programmers with that background should be able to get
productive a lot quicker than web devs I'd imagine.

~~~
Animats
Most major games are built on one of a few game engines (Unity, UE4, etc.)
which have their own ecosystems. People become Unity or UE4 experts. They have
forums, conferences, tech support. SL is its own pocket universe.

Second Life is divided into "regions", 256m on a side, each managed by a
separate process constantly communicating with its neighbors. This
geographical distribution system is unique to Second Life. User avatars and
objects can move from one region to another. You can look across region
boundaries. Running Mono programs inside objects are stopped, frozen, copied
across the network, and restarted on a different machine.

Most big-world MMOs cheat somehow so that they don't have to really solve the
distribution problem. They're often sharded, so that the number of players
that can interact is limited. Or they're smaller. Second Life's world is 100x
the size of GTA V. Or they're portal based; you can only get somewhere via a
controlled portal. In Second Life, you can fly over the whole world. (Mostly.
Fast vehicles hit bugs the devs have been unable to fix for a _decade_.
Another "legacy" problem.)

This is what the machinery for a "Ready Player One" or "Snow Crash" world
looks like.

It's not parallel enough, and the servers keep running out of CPU time on the
main thread. Everything then gets sluggish in world. The system needs an
overhaul to be more parallel internally on the core functions, and that's
really hard, expensive, and needs a dev team the company lacks. Yet another
legacy problem.

The technology is all unique to this one system. The only thing that works
even vaguely like this is the new Spatial OS from Improbable, which took 150
people to develop, is proprietary, and hasn't been shown to really scale yet.
We'll know late this year, as Nostos, a new game from China, rolls out, how
well it really scales. That's the first AAA title to use Spatial OS. Spatial
OS has a deal with Google where it has to run on Google Cloud servers, which
is scaring off most of the big game development shops. That costs too much,
and betting your business on a lesser Google product usually ends badly.

Hence the recruiting problem Linden Lab faces. You want to tie your career to
this one-off strange system?

~~~
bjt
I'm familiar with Second Life's distinctive traits. I played there for years,
and did contract programming for the Linden Department of Public Works for a
couple of them (including coding around those janky region crossing issues you
mention).

They pay an incredibly high price for some design decisions that I think users
don't even get much value from. It turns out that most users would rather have
a private island than live on a continuous continent where neighbors are
always putting up eyesores. If you were to start with that fact, remove the
requirement that private regions even exist on the global map, and let them
spin down when no one's home, then you could give paying users a lot more
space for their money while also reducing the company's spend on servers.

While I take your point about work on Second Life not being super
transferrable to any of the AAA game engines, it seems like it would be very
transferrable to creating such engines themselves. And what engineer wouldn't
want to help build the world of Snow Crash or Ready Player One?

I think the deeper problem is that Linden Lab has stopped prioritizing
investment in the SL platform. They've set their sights on creating VR-focused
Sansar instead. But it's tough to convince people to move over to a new world
when it means leaving behind the bigger community, the bigger economy, and all
the clothes in their inventory.

~~~
Animats
_It turns out that most users would rather have a private island than live on
a continuous continent where neighbors are always putting up eyesores._

Linden Lab tried that. That's what Sansar is. It averages 13 concurrent users
on Steam. Maybe some more who signed up outside Steam, but under 100. Sansar
is a "VR game level loader", not a world like SL. Somebody creates a level
map, and others can visit, but not change much. Sansar has a Star Wars prop
museum, a Ready Player One prop museum, etc. They look great. You visit once,
and you're done.

Other VR game level loaders are SineSpace and High Fidelity. (High Fidelity
just gave up, and "pivoted to enterprise".) They also have user counts in the
2-digit range, but worse content than Sansar. The hook for that market segment
was supposed to be VR headsets, which turned out to be a niche product. Even
VRchat, after a surge in 2017, dropped to about half its initial peak and is
stuck at a few thousand concurrent users. Facebook Spaces? Whatever happened
to that?

Meanwhile, Second Life continues to plug along, with 30,000 to 50,000 users
connected. That's about where GTA V online is, and would be 11th place on
Steam if SL was on Steam. SL was maybe twice as big at peak, 7-10 years ago.

Hence the legacy code problem. It runs, it's profitable, it has a significant
user base, and it needs improvement.

------
dudul
"legacy code runs the business". While I do agree with this general comment,
I've also seen legacy code killing the business.

I've seen legacy code so buggy that it couldn't be fixed, that new features
required quarters to be added, that customer service team had to double size
every year to help customers with their bad experience.

When legacy code is not properly maintained, it can become this unescapable
hell. Yes, it sort of works but at what cost?

~~~
jrjarrett
I read this article and I have to disagree with one of the major premises that
"no one wants to own legacy code" \-- it's more that in a lot of
organizations, you're not ALLOWED to own legacy code.

What I mean is that you may want to take "ownership" in the sense of learning
it and improving it but because it's not new development, large barriers to
paying down technical debt and updating to newer development methodologies is
disallowed.

So it stays, untouched, bit-rotted, and inflexible.

------
avgDev
I am currently rewriting a 10 year old VB application, which is in web forms.
That application communicates with AS400 DB2 database. I get to use as400 here
and there.

My application is in .NET Core 2. There are some issues as anything IBM can be
a pain in the ass to work with. Also, the as400 dev and me have a tremendous
knowledge gap. I know everything new such as unit testing, proper source
control, dependency injection, linq and ORM tools. However, when it comes to
speed of queries he is much better at optimization. Since, he has worked with
some ancient languages he has a better understanding of low-level programming.

Honestly, it is pretty interesting at what speeds the current tools allow us
to develop applications. For example, he would need to set up the DB, then
security, then applications, stored procedures, map parameters in code and so
on. I can accomplish most of these things with an ORM like Entity Framework in
less than an hour and have a working project, obviously depending on the scale
of it. However, both approaches have pros and cons.

Having old legacy code is problematic. It is harder and harder to find
developers and rates are going up as the BIG fish still rely on them and can
pay more.

~~~
mattmanser
Having been someone who's had to "setup" the db, app, etc. before ORMs were
mainstream, you're wildly overestimating how complicated it was. It doesn't
actually take very long, you'd maybe save a couple of days or so on a month
project, having to write a a CREATE TABLE and then the corresponding class are
actually trivial. You make the design decisions when you write the create
statement, the class is simply a copy of it.

It's just a bit easier, not a lot. I certainly use ORMs now, but it's a
convenience rather than a revolution. The truly revolutionary thing for code
from that era of C# were lambda expressions.

In all honesty, he's an idiot for not updating his skills. If he simply learnt
a bit of modern EF and learnt how to use git, he'd run rings around you. I'd
note that "proper" source control practice has been around for well over a
decade now (SVN was adequate, just not great for distributed teams), so he
must be really adverse to change.

The unit testing and DI, on the other hand, is all pretty useless noise in a
statically typed language like C#, for all MS push it. The only benefit in DI
is keeping the same instance of the EF around, and it's pretty hard to
actually get that working properly through-out the entire stack when you want
to do anything even remotely creative.

As for unit testing, never seen the benefit. I took over a project with
something like 50% unit test coverage that I never bothered to keep up, in 3
years the existing tests caught 1 bug.

All that highlighted to me that it was an utter waste of time and money it was
writing all those tests.

~~~
avgDev
Interesting perspective. I do agree that it is not good that he did not keep
updating his skills but he has a few clients even bigger than us and is going
to retire soon, AS400 has served him very well.

~~~
mattmanser
I've met programmers who've done the same thing (for COBOL).

I personally would not enjoy it as all you end up doing is maintenance and
working with nasty old code bases.

------
backslash_16
For anyone who wants to learn about working with and on a "legacy" code-base,
check out Michael Feather's book "Working Effectively with Legacy Code"

In the beginning it goes over code smells and how to find refacing seams in
your language of choice (C, C++, or Java) and then each chapter is a group of
techniques you can use.

Working on a service that had both a very old monolith and some brand new
microservices, I found it invaluable. I think the first lesson I applied from
it was using pinning tests for safer refactoring.

~~~
touristtam
> I think the first lesson I applied from it was using pinning tests for safer
> refactoring.

Thanks for the book recommendation. Funny to realize I have done this in the
past without prior knowledge; it just made sense.

------
shyneeup
I think one of the reasons why it's such a conundrum is because software
development as a practice is just now getting to the level of complexity where
large groups of people have collaborated over a large codebase over a long
time.

One of the problems that the microservice architecture is trying to solve is
by making a a large software entity a collection of smaller easier to manage
entities. It's like single cell organisms evolving to multicell
organisms...each cell becomes smaller and simpler and more specialized and
replacement of cells becomes easier and allows the overall lifeform to "scale"
and become more complex.

Mature software companies have gone through this transformation multiple times
already in the form of, n-tier, SOA, microservices, and now serverless
architectures. This is all part of a natural progression of making individual
components simpler, the overall structure more granular, and as a result more
resilient. This resiliency opens up new capabilities to scale a complex
system.

My long winded point being that legacy software will always exist yes, but
each piece of legacy software is already getting smaller and more granular
where rewriting it will eventually be a more continuous operation of
refactoring small things--kind of like skin cells falling off your body or
hair falling out.

------
bitwize
This made me think of an enterprise-software version of _Wreck-it Ralph_. I'd
watch that, but I'd be part of a small audience (though arguably, _Tron_
qualifies).

~~~
aeorgnoieang
I'd watch a movie based on The Phoenix Project book.

~~~
garyrichardson
I wonder if someone has optioned The Phoenix Project?

------
supernomad
I think its very interesting to think about this article in the context on the
open source projects that relied upon by the vast majority of major
applications and companies. Specifically OpenSSL comes to mind here. That
library has been used/leveraged in so many ways to build empires. Yet it's
currently struggling to find dedicated engineers that will work on its core
logic. It's an unfortunate situation and while yes there is LibreSSL and
BoringSSL its non-trivial to move to their implementations (I have tried, its
wrought with peril). OpenSSL isn't on its own either, there are many open
source libraries/applications that are struggling to find "owners". None of
which can be "easily" replaced by any means.

I think the comments about "heroic" culture being fostered by the lack of
ownership is spot on, but it isn't a bad thing unless those heroic actors
aren't actively trying to get attention put on said pieces of legendary
software. Imo its better to have heroic actors than nothing, and especially in
the context of an opensource application/library that is used worldwide, I
hope we can find more to generate more interest in working on them.

~~~
phonon
[https://esr.gitlab.io/loadsharers/](https://esr.gitlab.io/loadsharers/)

[http://www.catb.org/esr/loadsharers/](http://www.catb.org/esr/loadsharers/)

------
galaxyLogic
"Why is there unclear ownership?"

Because Extreme Programming (and other agile practices?) make Shared Code-
Ownership a "value".

~~~
mikekchar
Well, I think the article not so great on this one. There is unclear ownership
because management thinks that software is finished at some point. They think
in terms of large, vague blocks of functionality and have difficulty
understanding things at a finer scale. When requests come in from users of the
software, they are often ignored by management because none of the requests
are large enough to fit into their large, vague ideas. However, all the
requests put together add up to a very large impact. The result is that the
legacy code gets neglected and new greenfield projects get the attention.
Because nobody works with the legacy code, it becomes strange and foreign. It
is built with older technology which is no longer sexy and is boring on your
CV, so the movers and shakers in the IT group tend to avoid it. It's not code
ownership that's a problem -- the whole company washes their hands of it.

In terms of code ownership, you can run your group in a number of different
ways. I've seen projects with code ownership work and I've seen projects
without code ownership work. I vastly prefer the latter, personally. As long
as you are churning the code regularly, a lack of code ownership means that
internal conflicts about how things should be designed are forced out in the
open. They don't fester for years and years, where groups of developers end up
saying, "I can't work with that person. They are crazy." You have the conflict
early when it doesn't have such a large impact and you sort it out early
(Note: some people are just inflexible -- if you find that kind of person,
knowing about it early is also good. You can deal with it).

When you avoid code ownership, you are also forced to have code that is clear
for your entire group. For example, if you have a single person and they work
in isolation, their code may be impenetrable to the others. But if each person
in the group has to work on the code, code that most people don't understand
has very little chance of surviving. Overall, I find that staying away from
code ownership results in considerably more maintainable code.

However, there are clearly advantages to code ownership as well. One shouldn't
snub their nose at the benefit of being able to see a piece of code and say,
"I wrote that". Having ownership makes it easier to have pride in your work.
It can be a very motivating factor. If you have people on your team who are
feeling disconnected and don't feel like they are personally making an impact
on the group, giving them a little piece of code to own can be very good for
them.

Similarly, sometimes you don't have a choice for your team. Sometimes you have
a team of people who just don't work well with each other and there is nothing
you can do about it. Partitioning the code and allowing these people to keep
their distance may be the only thing you can do. Ideals are great and if you
can achieve them, it's wonderful. But you have to be conscious of reality.
People are people.

I could go on and on, but I guess my point is that very often I see remarks
like the one you made that seem to take a very superficial view of things.
There seems to be no effort made to understand why other people have a
differing viewpoint. Of course, you also get fan-boi style postings of, "The-
new-hotness is the best thing because reasons" which are also unfortunate.
Usually the truth is somewhere in between.

------
gpvos
SRE = Site Reliability Engineering, see
[https://en.wikipedia.org/wiki/Site_Reliability_Engineering](https://en.wikipedia.org/wiki/Site_Reliability_Engineering)

MVP = (in this case, probably) Minimum Viable Product, see
[https://en.wikipedia.org/wiki/Minimum_viable_product](https://en.wikipedia.org/wiki/Minimum_viable_product)

------
mrfredward
>The initial development cost of software rounds to zero when compared to
operating costs in perpetuity.

Mostly a good article, but what is with this quote? Aside from being factually
wrong from a dollars and sense standpoint, doesn't it contradict their point
that legacy code doesn't get enough love?

~~~
stahnma
Author here.

I don't think it's wrong in most (nearly all) cases. The cost of running
software, keeping it updated, changing the HW it runs on, keeping it
compliant, etc over time costs much more than the dev time to build it.

Unless your service is completely replaced every couple years, I'd venture
operations cost more than development. (even if dev handles the ops).

The point of the statement is designing for operability helps with ownership
and combats the "leave it in the corner and hope for the best" mentality.

~~~
krisrm
This seems like moving the goalposts though. "Much more" is not "rounds to
zero". Software is just expensive, period - to build, and to maintain.

------
lecarore
I really like how they tell the story at the beginning, I felt sorry for an
hypothetical piece of legendary code

------
magashna
It's kinda dumb but I do like this children's book retelling of software
development

~~~
rocky1138
It's not dumb. Some of the most impactful stories are fables. You may also
like Java Koans.

------
phonebanshee
This would have been much improved if the cute legacy system has been speaking
a dialect of Latin found in ninth-century Bavaria, with a slang vocabulary
developed for its specific problem. And insisted that everyone else use that
dialect when speaking to it.

------
urthen
Cute story, but I detect a certain bitterness and refusal to accept the
"future" of software engineering is actually the "now."

People didn't think up microservices yesterday, they aren't some hot new fad
that'll be forgotten in a year, and they aren't being developed because people
just have nothing better to do. They're replacing legacy systems which nobody
wants to touch at a rapid clip. People don't like working on legacy systems
because they're at an unfortunate intersection of being business critical yet
flaky and hard to work on.

In the context of this story, well... your protagonist is actually on life
support. But not to worry! They'll be taken off it soon...

~~~
aeorgnoieang
If anything, it's a pretty pragmatic view about the high frequency with which
new systems intended to replace legacy ones fail to do so.

Certainly applications can be supplanted or replaced, and often are, at the
discretion of users and customers.

But 'living' (running) systems can often only be successfully replaced in the
manner of the Ship of Theseus.

------
reallydude
> Legacy means it worked.

No and shame on you for pushing this idea, is my first reaction. "worked"
doesn't mean anything.

Positing this redefinition of Legacy (and implying what "worked" means) rubs
me the wrong way, especially when tying it to the idea of code ownership.
Maybe I just have a warped perspective, except that I've run into these
scenarios.

"worked" might mean "it was easy to fix and it broke a lot", "it calculated
the numbers correctly but nobody knows how to change it without it breaking",
or "it costs $5k a run but it gives us the right count of a database column".
Legacy means it might be working, depending on your CURRENT needs. Your
current needs change, even when the code does not.

Code ownership isn't at a single level, which makes some of this analysis seem
idealistic.

~~~
stahnma
Author here.

Code doesn't become legacy if it didn't fulfill some purpose. If it didn't
work, it wouldn't be legacy, it'd be source code in somebody's home directory
that was never deployed.

All working software is not legacy software. All legacy software (still
running) works, for some definitions of works.

That's not to say your points aren't also really correct. Software is built
with current requirements and knowledge in mind. Best decisions with info we
have..etc.

~~~
davemp
I know of at least three projects (with huge budgets) right now that are
spawned from the corpses of old, failed projects that never worked. Those old
projects make up the legacy of the current effort and can’t be easily thrown
away.

