
The Lava Layer Anti-Pattern - rumcajz
http://mikehadlow.blogspot.com/2014/12/the-lava-layer-anti-pattern.html
======
rumcajz
Good write-up. I've been observing this phenomenon for years and I think that
one of the main causes is poor cost estimation by developers (or, often, no
cost estimation).

The new project lead asks management to allow him to re-write the project.
It's a 10 manyear task so the management declines.

At that point he decides to do the changes in gradual manner, completely
ignoring the fact that it's still a 10 manyear task, while, realistically, his
team can spend 2 manmonths a year on refactoring. Thus, the re-write will be
accomplished in 60 years.

The same pattern repeats with the next project lead and so on.

~~~
angersock
There's that wonderful point in a project where you can point-- _point
directly at!_ \--a fix to a fix to make legacy thing work. I love that point.
That's the point I start looking for new work and resenting my forerunners.

"second system" syndrome is very real, but nobody in management ever seems to
appreciate "Look, technology has _progressed_ in the last five years since
this was written...we can do more with less, pleeeeaaaase let us fix this.".

 _Nobody_ looks forward to working with legacy code. The business logic is
usually wrong, the business itself has changed, the original developers are
gone, the market talent composition has changed, etc.

I'd honestly rather see a business embrace the ablative nature of code and
instead build itself with the _assumption_ that, every 2-3 years, chunks will
be rewritten. This is an argument in favor of services and better
documentation and tests, by the way.

EDIT:

And even worse is when the manager/CTO/whatever _admits_ that the code is
going to need rewriting--but _not yet_. Because that means that a) you're
going to be wasting effort temporarily fixing something you know will be
thrown out and b) tacitly admitting that they are too scared of change to
embrace it and too short-sighted to figure out how to balance business
requirements with technical debt.

I've been burned by that multiple times.

 _A man-month of unpaid overtime for rewrites, not a nano-second for legacy
support!_

~~~
usea
_> I'd honestly rather see a business embrace the ablative nature of code and
instead build itself with the assumption that, every 2-3 years, chunks will be
rewritten

_>And even worse is [...] you're going to be wasting effort temporarily fixing
something you know will be thrown out

Can you reconcile these two points for me? I fear I may be misunderstanding
you. It sounds like you're arguing in favor of treating code as a transient
thing, but then lamenting having to write something that won't last.

~~~
angersock
Sure thing.

The subtle difference (not well articulated by me, I suppose) is that in one
case, you are given the go ahead to go and rewrite everything to conform to,
say, an interface spec or API or whatever. This year's implementation or
architecture may be scrapped, but then you get a cleanish slate.

In the latter case, what's happening is you are usually building an adapter
over something that exists, instead of throwing it away completely. What then
happens is that there is a lot of stress involved ("Is this wrapping
correctly?", "Does this still do what it used to?", "Have my tests covered all
the use cases?", "Wait, fuck, this original thing never even _had_ tests...so
aren't my tests are kind of pissing in the wind?", etc.) and then inevitably
the push is to fix the fixes, and then fix the fixes' fixes, and so on.

And sometimes that's unavoidable (maybe the legacy code is, oh I don't know,
numerical simulation of a heat-exchanger in Fortan77 that was cutting edge and
now the original author is quite dead and buried). But, the painful thing (as
mentioned) is when the person in charge _even admits_ that the whole thing
will be rewritten anyways.

In that case, why the fuck is the rewrite being put off? It signals a lack of
faith in the development team, and a lack of prior planning to deal with
downtime or other issues coming up.

------
Glyptodon
I think projects displaying the pattern often have a root problem that the
writeup didn't discuss: An initial application with requirements created by
non-technical folks who had it implemented by whatever the equivalent is of
outsourced workers/students/interns/guy they hired without knowing how to hire
a developer.

The initial developer doesn't understand, know, or follow basic CS knowledge
or design trade trade offs and constraints, but will try to create a
reasonable system using whatever has press at the time - Rails, Php + Zend,
etc.

Unfortunately they will follow weird aspects of tutorials they googled for
studiously while missing out on the design implication of their choices
leading to monstrosities like a product display page that requires hundreds of
recursively made SQL queries, an entire application where 90% of the code is
in one file, database tables that are composed in the most awful ways, and
more.

It's this first layer that causes everything that comes after. The root of the
problem isn't that trying something new is right or wrong. It's that the
original was done so badly nobody can reasonably continue down the same path
and everyone who comes after can't figure out how to cope.

Of course it's typically exacerbated by deadlines. When you can add a new
dependency and meet your deadline or try to do things the inscrutably bad way
they were done previously and miss it (let alone have any ability to test it),
it's hardly surprising that people choose to add what they know.

When confronted with something intractable of course you throw your favorite
spaghetti at the wall. And of course there's not any sort of order to what
sticks.

So I think the lesson is more about the criticalness of doing things remotely
right to begin with.

~~~
mreiland
stop blaming non-developers for the problems created by developers.

It's the result of developer who don't have the maturity to consider the long
term risk of their decisions. That has nothing to do with the business
leaders.

I freelance which means I see a lot of shit. In any given week I can
touch/work with 10+ tech stacks across both Windows and *nix. When you do that
on a regular basis it gives you a different perspective on things.

In particular, you write/design code in the same vein as the code around it.
It doesn't matter what your personal values as a developer are. Consistency is
more important than that. I have been known to pull people to the side and ask
them "I'm about to do X, how do you normally do X in codebase Y? I want to do
it exactly the way you do".

If you're going to change the way something like a DAL accesses its data you
better be able to either do all of it, or you do none of it for exactly the
reasons the article stated.

That is purely a technical decision, and the blame for that lies with every
single developer who chose to introduce a new tech into the stack for no other
reason than sensibilities.

~~~
yason
Honestly, we can't really blame the developers either. If you inherit a
compilation of different techniques, there isn't much you could do to it.
There isn't necessarily a code _base_ that you could build on as the same
thing is already been done in a number of different ways. So any way you're
going to make changes there isn't going to make the code much better and,
thus, the least worst choice would be to start gradually injecting some
consistency into the codebase even if that's the fifth layer on top of the
previous ones.

Maybe, just possibly, the second developer could at least theoretically
continue exactly from where the first developer left off (a single person or
team == a single codebase that's at least somewhat consistent) but it could
often make the mess even worse if the first developer really didn't have much
of a clue of what he was doing.

The managers are basically the customer who ordered the software and they
can't be trusted to know anything. It's a very common case that the customer
doesn't know what they need and it's the job of the developers to first figure
out the customer's _real_ problem before beginning prototyping.

~~~
chris_wot
Thats fine for the initial stab at the problem. But then someone who does know
what he is doing come along and recommends a module be rewritten and
management says no, because "it's working".

At this point, it's management's fault.

------
mkozlows
I 95% agree with this article, but think that 5% is an important disagreement.

Background: I spent most of the last decade writing, maintaining, and managing
a large enterprise app that had to evolve from its .NET 1.1 foundations to the
present day. Our code base looked almost precisely like the one talked about
in that article, and with the very real problems mentioned there.

But...

1\. Sometimes the productivity wins of a new approach are serious enough that
it's worth taking the consistency hit. I'd argue that ORMs, for instance,
really are worth it compared to the quasi-code-gen ADO.NET layer. Switching to
those midway through the project introduced inconsistency, but all future
development done with the ORM was much faster (and more reliable) than the
stuff done with the old approach.

(And yes, this is a dangerous argument, because it's easy to convince yourself
that it's true all the time, if you want an excuse to use a new technology.
But if applied skeptically, it really is true sometimes.)

2\. Developer morale is a thing. "You can only use technology invented in
2003, none of this fancy .NET 2.0 stuff, and haha LINQ, are you kidding me?"
is going to crush people's spirits, whereas "sure, let's try this new module
out using ASP.NET MVC" will introduce an inconsistency into the codebase, but
will keep developers more engaged and learning new things (and who knows,
maybe this experiment will prove to be one of those ones that gives you a big
productivity win).

You have to be careful with that, because you can't literally try out every
crazy new thing, but you also can't ignore everything new in the name of
consistency and expect anyone to want to work on the project.

~~~
rumcajz
C devs are OK with working with a 40 years old technology. Actually, they are
happy that it doesn't change every 6 months.

~~~
angersock
True, but the ones who are worth their salt know how to take advantage of new
libraries and approaches, and how to structure their code so that it is easy
to port and maintain.

Those devs are, naturally, in the minority.

------
vinceguidry
I think this is ultimately caused by developers not quite knowing where and
how to place their ambition. It's easy to look at the big legacy app and be
like, "I can fix that!" Then they roll up their shoulders and go to work,
management be damned. Management can't see or understand what he does, and
they're paying him for his expertise, so they give him the benefit of the
doubt and let him do what he thinks he needs to do.

I documented my own answer to this situation here. Basically, don't treat
organizational problems as technical ones. They're paying you to do a job,
everything else you do is basically an ambition. First rule of ambition is to
do no harm.

[https://news.ycombinator.com/item?id=8768524](https://news.ycombinator.com/item?id=8768524)

If you're going to refactor, don't refactor in the direction of new. Refactor
in the direction of coherency. Do things that you can finish, or, that at
least, don't matter that much if you don't. Like adding tests. Nothing wrong
with that.

If you want to fix the legacy app, you need to make less ambitious changes,
because you need to be able to finish them. Instead of changing the data
types, refactor the initialization code. The initialization code runs once,
data manipulation code is basically convention and you need to change it
everywhere at once or not at all. Refactor it to the point where you can
change it everywhere at once and then do that. Or get really good with a text
editor and macro the changes.

Giles Bowkett has a great book on it. Unfuck A Monorail For Great Justice.
It's Rails-specific but the ideas can work everywhere. Write code that
understands code.

[http://gilesbowkett.blogspot.com/2013/03/new-ebook-unfuck-
mo...](http://gilesbowkett.blogspot.com/2013/03/new-ebook-unfuck-monorail-for-
great.html)

But understand that there are probably better things to do with your time. You
can pick out pieces of the legacy app and try to understand them well enough
to rubber-duck debug them without having the source code in front of you. Once
you have that then really, you don't need to be a hero. You're already a hero
because you know the system better than anyone. You can focus on other things.
Maybe use your new-found credibility to build some consensus among your peers
and managers. Then when you make your case to replace the app, it won't just
be you.

~~~
algorithmsRcool
I feel like this is very sound advice but i'm not sure if i can really accept
it fully.

I'm a fairly green developer (3 years professionally) working on an C#
application that suffers from design schizophrenia as detailed in the article.
I'm more or less the sole developer assigned to it now.

Although I am new, I feel very strongly about my 'pride as an engineer' in my
application of sound design, knowledge and rigor to the best of my abilities.
So it discourages me greatly when i hear a lot of "Uhh, that was a long time
ago...", "We had tough deadlines..", "Well developer 'X' and me disagreed
about that..." from the former maintainers of the code.

I've taken to a rash practice of just tearing out tightly coupled, untestable,
duplicated and poorly designed code by the roots. I feel like i am somehow
passing a 'holier-than-thou' judgment on the work of my predecessors. But due
to it's tight coupling i can't have much confidence about even seemingly
simple changes since they always seem to come back with a wagon of defects.
The new code i write surely isn't perfect and has it's own defects that come
back to me, but i feel much better about the code in many ways and i am always
open to criticism and discussion over my design choices. If anyone would ever
give them serious study...

I have this gut feeling that if i leave the code i touch messy that i'm not
doing my job and the next poor soul to come behind me will lose weekends to
trying to clean up an even deeper mess. I feel judgmental even voicing these
thoughts.

~~~
vinceguidry
> I've taken to a rash practice of just tearing out tightly coupled,
> untestable, duplicated and poorly designed code by the roots.

That's not a terrible approach. Depending on the context the code is running
in, that might be a worthwhile way to maintain it, especially if there's not
all that much different between doing that and working on it as is. If it's
taking you the same amount of time, why not leave it better than you found it?
The important thing is that you finish the job, and it sounds like you're
doing that. You're not making a three-year plan, not telling anyone about it,
and doing a little bit every day. That way lies the lava layer.

Me personally, I have like four other projects I could be working on. Give me
a choice between working on old legacy and working on new hotness, I'll band-
aid the legacy every time and just get on with life. I didn't use to do it
this way. I wanted to make my mark on the legacy codebase, prove that I was
better than it. So I'd tell my boss I'd need the rest of the week to implement
this and then dive in. I'm happy I don't need to do that anymore.

~~~
algorithmsRcool
> I wanted to make my mark on the legacy codebase, prove that I was better
> than it. So I'd tell my boss I'd need the rest of the week to implement this
> and then dive in.

I think this statement sums up my attitude almost perfectly. I have a notion
that if I channel that line of thinking into positive structural changes in
the code then I should while I can. I just don't know whether or not it is a
good attitude to have.

~~~
acveilleux
My own experience is that it's easy for that to become either a source of a
lot of regressions or an infinite death march. It's key to thoroughly
understand the part to replace before getting involved in that kind of
wholesale replacements.

The amount of tests available on the previous implementation can go a long way
towards lighting the way and bounding the level of required efforts. Code that
is deeply embedded, untested and "ugly" is just a recipe for pain however.

~~~
algorithmsRcool
I need to work harder at mapping out our code to understand it as you say. I
keep telling myself every codebase is different and none are perfect, but it
is so nice to work in fresh or at least clean code.

My project is about 200KLOC C# 2.0; we have a single .cs file 13,000+ lines
long and no, it's not generated. We have a single method 900+ lines long in
our DAL containing 23 separate hand coded SQL queries and no, it is not a
report. Our codebase contains exactly 14 unit tests. 10 of which were written
by me. Your prayers are welcome.

~~~
chris_wot
You aren't working on a certain application that deals with ITSM by any chance
are you? With a "polling service"?

~~~
algorithmsRcool
Haha, nope. B2B software in the hospitality industry. It's both comforting and
sad that my codebase's ugly stats might be shared by other applications out
there.

------
function_seven
While the example shows five different people bringing in four different ways
of doing things, I was sad to recognize a lot of my own work in that story.
I've been the sole developer on a smallish web site for five years.

I've personally switched patterns over the years when adding features, always
telling myself that I'll go back and refactor the earlier work.

------
jamesu
I've seen this happen so many times, it feels like the norm. I guess it's
inevitable though, it's more interesting to work on something "new" and "cool"
than adjust your mindset to year old design decisions.

Would be interesting to know if there is a relationship between years of
experience in programming and how likely you are going to inject the latest
"cool" design pattern / framework into code. Personally I have become far more
cautious about this stuff. I'd rather adapt what is already present than
subject anyone else to an abstract monstrosity I just cooked up after reading
the latest hacker news post on the latest hype.

~~~
acveilleux
I think it's more a matter of as programmer mature/learn, they overly
disparage old code and idioms. It's not the new orthodoxy...

I've seen in with RPC calls (hand-coded HTTP -> Hessian -> Protobufs),
serialization languages (XML -> Json -> Protobufs/Bson), ad nauseam.

I think this kind of changes is the natural state of multi-year development
efforts. On a 10-15 years old code base, you'll see all sorts of things like
that cropping up and it's incredibly hard to steer clear of these kinds of
things.

What are the alternatives? Wholesale replacement of all uses? Wasteful in
resources! Not introducing new technologies? Offends the sensibilities of the
devs...

------
steven777400
The flipside to this is the older technologies start to become end-of-lifed or
unsupported by the vendor (like .NET Remoting or .NET 1.1). In that case,
there seems to be some level of continual need to push the application
forward, but of course we want to avoid the kind of inconsistent layering
described here. It's a tough problem.

The easy answer of "regular rewrites every 5-7 years" doesn't sit well with
paying customers or management.

------
lmm
The best project I've ever worked on was like this. I don't think it's an
anti-pattern; if anything I'd see it as a positive sign, as it indicates a
project that's under continued development and willing to investigate new
technology.

Just as it's vital to accept that business requirements will change during a
project, and structure your methodology to accommodate this, the same is true
of technical considerations. And the same solution applies, viz. Agile: break
up any "redesign" into small steps that can deliver concrete benefits inside a
two-week period. Don't treat the "transitional architecture" as a temporary
thing that doesn't need to be good, doesn't need documenting etc.; like with a
big database migration, you make sure that at every stage it's something that
you can develop under, and you know what you'd do if you had to abandon the
plan at any stage.

If you try to keep a codebase on 2003 tech forever, you're storing up trouble
for later; devs will become harder to find and more expensive to hire, and
eventually you'll be forced into a Big Rewrite, and we know how those turn
out. Like with deployment, better to embrace Continuous Re-Architecting. If it
hurts, you're not doing it often enough.

~~~
scribu
Could you elaborate a bit on why you consider it the best project you worked
on? Maybe it was interesting independently of how it was architected etc.

~~~
lmm
I think this kind of pattern was a big part of why I enjoyed it. Because it
was an 8-year-old, 500-kloc codebase, everyone knew you couldn't change it
instantly; you could see an archaeological history in the layering, and
understood that in all probability the style that was great today would become
the bad old code in three or four years' time. But the result was that it was
a living system, we knew we were working on it because it made us money, and
every day we were making the code a little bit better (even though that's a
never-ending process), which was hugely satisfying.

------
bglazer
From the linked article:

> Look at Stack-Overflow ... they also use only static methods for
> performance, we should do the same ... Gordy had dismissed Ina’s love of
> unit testing. ___It’s hard to unit test code written mostly with static
> methods._ __

I also happen to like static methods quite a bit, precisely because I thought
they were EASY to unit test. I thought that a static method, which is
essentially just a function in the enclosing object 's namespace, would be
EASY to unit test. Input -> output. No state.

Am I digging my own code grave here? What am I missing?

~~~
rverghes
He's assuming that the static method has a dependency on something. That it's
not purely functional. For example,

    
    
      public static doSomething() {
        Database.write("Did something")
      }
    

Makes testing hard because Database is hardcoded.

~~~
_asummers
Additionally, if you need to mock that doSomething() method's behavior at all,
you can't just subclass and override the method. You would need to bring in
byte code voodoo (using e.g. PowerMock in Java).

~~~
blktiger
And PowerMock is the bane of all existence!

It can be a useful tool, but it causes all kinds of weirdness which can
completely mitigate any advantages. On my current project I had to strip out
most of our uses of PowerMock. While the tests pass on my local machine just
fine, many of them fail for no good reason on the CI server.

~~~
lmm
If you can avoid it, do; PowerMock is basically only for tests that interact
with badly designed code that you can't change. But sometimes you need to
depend on a badly designed library, in which case having a somewhat cumbersome
way to mock it is better than nothing.

~~~
ZoFreX
"PowerMock is basically only for tests that interact with badly designed code
that you can't change"

Presumably where the definition of badly designed includes "uses static
methods"? That's something that I hold true, and teach to other people as
true, but I have started wondering about it. Whenever I ask why it's bad I get
told "it makes testing hard". Well, Powermock can make testing it possible -
"but powermock is only for badly designed code". What's badly designed code?
And so it goes round.

Is it really that conceptually bad? Or is it just that our testing tools are
insufficient, or that the language itself did not make provisions for testing
all of its features? I've seen a lot of code recently that uses dependency
injection purely for testing, and uses builders instead of constructors all
over the place, and it feels like a lot of effort and complexity to avoid
using PowerMock.

~~~
lmm
It's not just about testing, though that's where you tend to run into it
first. Statics and constructors are both second-class citizens in Java: they
can't conform to interfaces, so there's no way to separate interface and
implementation when using them, no way to hide information, no way to invert
control or any of the other principles of good design.

(Of course, Java was designed by people doing the best they could subject to
particular constraints, and is now constrained by backwards compatibility. But
other languages provide the same kind of features in better ways (or provide
syntax that means builders and dependency injection are much less effort,
depending on which way you look at it))

Pragmatically, good design is not the only concern; if you're not writing a
library with binary-compatibility concerns, then IMO the syntactic overhead of
builders is often a cost that outweighs the benefits. But the design costs of
statics and constructors in Java are real enough, and there are very direct
costs to using PowerMock (tests are harder to maintain and more brittle as
they have to specify which classes they're rewriting, and it can conflict with
other things that do bytecode manipulation e.g. test coverage tools).

------
raincom
This is seen in the infrastructure side at any non-startup company. Let me
give you an example of ACLs management.

1\. First they get devices from one vendor. They set up ACLs in certain way:
hardcoded IP addreses/ranges, hardcoded ports.

2\. Then come devices from another vendor. This addes another layer.

3\. Another vendor comes with an acl automation tool and tells team to create
objects/references for ip addresses/ranges/ports. Some guys follow it; some
thers don't.

4\. You can see a big mess of things created by many vendors, many software
automation tools, manual additions, etc.

One day, some guy changes the object/reference; all of sudden, production
outage: add another layer of manual edits on the top.

That's why they preach "if it aint broke, dont fix it, but add another layer
on top of it".

------
blakecaldwell
When working with a poorly-written or unstable legacy app, there _is_ a middle
ground approach. Rather than rewrite the whole thing (often impossible), or
introduce a new framework in the app, you can draw a circle around a core
subsystem and rewrite just that component as a microservice. This frees you
from the (often) monolithic app, gives you the freedom to pick an entirely
different language, if you want, and leaves you with a part of the app that
any new developer could completely wrap their head around.

------
oconnore
A more realistic way to fight this is to make Lava layer programs less harmful
by keeping documentation and testing efforts consistent throughout the project
(and back filling when previous developers didn't).

The constant narrative in the example is developers having no idea why and
where things are built a certain way. With more information and verification,
gradual changes and refactors can be more manageable.

Avoiding Lava layer applications is not feasible, so calling it an anti-
pattern is silly.

------
underwater
I've seen incremental changes really well. But in those cases there were some
important forces at play. Firstly, there was always at least one person
advocating for the change. They could make improvements over time, educate
other engineers, and react to feedback. Secondly, changes won out on merit.
Engineers used new abstractions because they were better, not because they
were new. Sometimes abstractions didn't work. And were expunged from the
codebase.

------
jonpress
I have experienced this personally in a couple of companies I worked for -
Just like mentioned in the article, they had high staff turnover. What happens
is that nobody truly understands the codebase and it requires many more
employees to maintain. To make matters worse, it's also a self-perpetuating
cycle. It's more a management issue than an engineering issue.

------
esro360
Hmm, this has been true for some projects I worked on. When there is a stable
dev lead for many many years. He needs to teach others his approach. Then it's
perfect. People tend to keep things that work well. Just don't hire those who
want changes:)

------
fsloth
This reads like a corollary to Naur's "Programming as theory building"
pages.cs.wisc.edu/~remzi/Naur.pdf

------
blakecaldwell
The names seem forced. I can't help but think they're anagrams of real people.

------
kuni-toko-tachi
There isn;t and will likely never be any strategy to prevent the conditions
that the author discusses. Software is biological in nature, and to take it
further, subject to the same laws as everything else, in particular entropy.
There is no "Maxwell's Demon" approach to writing software. There are best
practices and mitigation strategies, but those are incompatible with the
business of writing software.

Further, and I think this should be more and more clear to people, _any_
retained unencrypted data is _compromised_ data.

Unfortunately, our society is on the way to being completely dependent upon
centralized systems and institutions which are unaware of their limitations,
and in fact we live in an age where centralization is considered a virtue,
when it is instead a significant liability.

------
jrochkind1
yes. brilliant.

