
It's Not Sabotage, They're Drowning - jeffrey-sherman
https://shermanonsoftware.com/2019/11/15/its-not-sabotage-theyre-drowning/
======
stickfigure
I can perhaps understand where this one is coming from:

 _I refactored the code from untested and untestable, to testable with 40%
test coverage. The senior architect is refusing to merge because the test
coverage is to low._

You have a piece of software running in production; even if it's poorly
written and bereft of tests, your customers are satisfied. Customers
(unwittingly) have already vetted this version! A rewrite, even with brilliant
design and fantastic test coverage, is still more of a risk than fixing what
isn't broken.

This isn't to say that you shouldn't rewrite code, especially if it makes
other improvements possible. But don't discount how high a bar new code has to
reach to measure up with "it already works". Maybe 40% is still too risky.

~~~
arkh
You don't want to test code. Coverage is mostly useless.

You want to test functionalities: setup end-to-end tests. It will be hard, it
will take time and it often is a solo endeavor for the first months. But when
tested functionalities stop breaking for no reasons even when you refactor a
lot of code, when "the project for which a dev put tests" starts being "the
project we are confident to push to prod without breaking things" you should
get some traction.

And once you think all your functionalities are tested you can measure code
coverage: what is not covered should be dead code you can remove.

~~~
JohnBooty
I can't get behind this.

End to end tests are indeed important.

But let's say you have an end to end test that covers A, B, C, D, E, and F.

One day this test stops working. Well, what's the culprit? A? B? C? D? E? F?
If you had more granular tests in addition to the end-to-end tests, you'd
know.

Now, I agree there's a possible downside here. Maybe there comes a time when C
and D are no longer being used. It might not be obvious and C and D could
linger in the code base, with corresponding dead weight in the test suite.
This is not ideal. However, I would rather have this than a tough-to-debug
test failure.

~~~
andrewflnr
> Well, what's the culprit? A? B? C? D? E? F?

Probably, the one you just changed. How often is it really going to be that
hard to debug?

~~~
thfuran
I only changed L, which is nominally unrelated to all of those. You thought
this was a clean, well-modularized codebase?

~~~
JohnBooty

        You thought this was a clean, well-modularized codebase?
    

Well, even if it's nice and clean A, B, C, D, E are each going to have their
own dependencies!

------
CoolGuySteve
Nobody’s better at letting better be the enemy of good than software
engineers. For some reason our culture likes to argue and nitpick over the
most mundane stuff, usually missing the forest for the trees.

I’ve had jobs where the manpower spent just arguing outweighed the cost of the
change significantly. One dickhead manager I had spent weeks arguing over the
acquisition of a $500 SSD. It made me want to slap $500 down on the table just
so he’d shut the fuck up already.

The only way I know to counteract it is by saying “don’t let better be the
enemy of good”.

~~~
brain5ide
A few points:

1\. Computing and engineering in general is such a discipine where one
nitpicky detail can make all the difference. One bit flip, one keystone, one
badly selected material. However, the conduct of negotiation, could be
improved as always, but it's a general listening problem most of the time.

2\. From the dickheads point or view, everything can be solved by spending
more money on HW. Each unexpected spend like this is a risk to open the
floodgates and lose both financial and engineering discipline by setting a bad
precedent. Not sure you want easy handed decisions there as well.

~~~
biggestdecision
It makes no sense to cheap out on hardware though. It's such a tiny cost
compared to the overall cost of the employee, you are wasting money if
inadequate hardware is making your employees less productive.

~~~
retSava
In one sense, it doesn't make sense. The productivity boost would likely be
worth it alone.

From another perspective, this is a +500$ post in the cost column of that unit
(and perhaps n*500 if the whole unit then comes asking for the same thing).
The cost of spending time arguing is baked into the cost of salary, meaning it
doesn't show since it's the same regardless. It's a form of optimization, but
the optimized unit is not productivity.

A friend of mine was working as a recruiter where the whole company was
sitting all day long at their laptops. He was denied buying an external screen
to be more efficient since he then could have more information on the screen
(eg an application next to a browser), "because then everybody wants one",
even buying it himself.

edit: nb I'm not arguing for not buying the ssd/screen.

~~~
JohnBooty

        He was denied buying an external screen to be more
        efficient since he then could have more information on the
        screen (eg an application next to a browser), "because 
        then everybody wants one", even buying it himself.
    

My wife was denied permission _to bring her own secondary monitor from home_
for that reason. Unreal.

I also faced pushback at one job for bringing my own monitor from home. Same
reason given. I'm an entitled software developer though, so I basically
disregarded my manager on that one and he let it go.

~~~
CoolGuySteve
We set up a monitor for my wife at her large corporate job without asking
permission. While the rest of her office had tiny 1280x1024 screens from 10
years ago, she had a 1920x1200 24" monitor in addition to the corporate shit
monitor.

Sometimes people would ask her about it, she would say she bought it herself,
and they would say "oh" and move on about their day. It was a complete non-
issue.

You could argue people shouldn't be plugging in random components at work and
whatnot, but at the end of the day, IT is really powerless to do much about
it. Nobody's going to fire you for buying your own productivity enhancers, the
situation is just too embarrassing for management if they're reasonable
people.

And really, Excel is so focused on visual space that having a larger display
for her is a quality of life issue that only costs $120.

~~~
JohnBooty
I completely agree, but there really are managers petty or shortsighted enough
to block this stuff.

I think one mistake my wife made, though, was mentioning it to her boss before
she did it.

What she should have done was _done it._ Then her boss would be more inclined
to let her keep it.

------
marcus_holmes
I think it's kinda hilarious that the article talks about a psychological
effect that hurts software teams, and HN gets into a full-on debate about
software testing strategies ;)

I've seen this stuff before. Developers making obviously-bad decisions is
usually a symptom of a dysfunctional work culture and a code base that's both
vital and fragile. It's not fun to work in such places, but it is possible to
fix them.

~~~
hvidgaard
The main issue is for management to accept that velocity will be significantly
reduced and that many issues will pop up along the way and require a
substantial QA effort to keep the quality from dipping too low.

It's still the best way forward as a complete rewrite is rarely a good idea.

~~~
marcus_holmes
yeah, I had to explain to a CEO that we weren't going to be shipping any new
features for a few months while we upgraded core systems and eliminated tech
debt. It was a tough conversation. But about 6 months later we were able to
start developing new features again, and everything was much quicker and
smoother. The lead dev had regained his composure, too, and wasn't being
actively obstructive to everyone who tried to do anything to "his" code base.

------
dredmorbius
A possible way to sell instrumentation / metering is to build your own
scaffolding, find a problem using it, and sell _both_ the scaffolding _and_
the solution it's provided.

As the article notes, metering _by itself_ simply illuminates problems. For
the drowning team (or individual), that simply delivers bad news.

If you can show that the effort actually delivers results, you've got a way to
sell the instrumentation as well.

In more pathological cases, you don't reveal the scaffolding, but start
cranking out code fixes based on it. You may have the opportunity to sell the
metering later, when you're asked how you're finding and fixing the problems.

Rescuing projets from antipatterns is painful at best. Enlightenment is a
painful process. It peels away comforting lies to reveal unpleasant truths.

------
steve_gh
So many years ago I was involved in an issue with a BTS (a cellular radio base
station) for a telecoms equipment manufacurer. The software was written in C,
and had a message parsing architecture between processes, with fixed length
queues, which discarded messages when they overloaded.

There were various problems (the subject of a number of war-stories, which I
won't repeat now) - but basically anything that wasn't using a properly
designed state machine could die horribly when messages started getting
discarded.

So we did some work around the fundamental design of the queue handling
mechanism. We came up with a software modification which would improve the
efficiency of the queues handling, which would stop the queues overloading
(which we saw happening occasionally at peak loads on the biggest BTSs).

We could measure various aspects of the queue handling performance, and use
queuing theory to prove that the performance would be better at very high
loads.

We had a test rig and we could test the relative performance of the new
software vs the old software at normal loads.

What we didn't have, was a test rig which could run at the highest loads, so
we couldn't test the behaviour of the new algorithms at very high loads.

So the improvements were binned. The existing software was known to fail
(badly) at extreme loads. The new software could be tested at reasonable
loads, and mathematically be shown to improve matters at very high loads, but
because it couldn't be tested, the process said that it must not be deployed!

~~~
Aeolun
How was the original version deployed? The definition of ‘very high load’ was
different at the time.

------
mirimir
When I was trained in lifesaving, one strategy for panicked victims, who might
drown the rescuer, was just to give them some space to drown enough to lose
consciousness. Then they were much easier to rescue.

~~~
wheelerwj
it doesn’t usually apply to drowning victims, but more broadly its applied
when someone is choking in general.

if someone is refusing help, you wait until they pass out, and it becomes
implied consent. I assume in this example, that would be firing or otherwise
being told by your supervisor to take the help.

~~~
kevingadd
Consent isn't necessarily the issue in drowning individuals, since the
instinctive drowning response can make it dangerous to save them even if
they're not about to tell you 'let me drown'. Maybe actually a good comparison
for software teams that refuse to make incremental improvements out of fear or
panic... it's not that they don't want things to be better or refusing help,
necessarily - they're just terrified of the consequences if the improvements
go wrong.

------
inertiatic
>I refactored the code from untested and untestable, to testable with 40% test
coverage.

This tells me you think "coverage" is a good metric, which also tells me
you're you're not covering a whole lot of cases (edge or not) where the
"untested code" functioned as it should.

You're asking to replace something which has potentially some known issues,
with something that has a potentially unlimited amount of unknown issues.

And the more experienced person at your job refused? What a tyrant!

~~~
poutrathor
I strongly disagree.

The said refactoring was probably not a huge task since he is reporting it
done. Larger tasks are most often planned and monitored.

Splitting some class and decoupling some resources + adding the testing
framework and implementing a first batch of tests. Of course it might have
introduced few regressions.

But

> has potentially some known issues

Read again that. Exactly, none knows how many bugs there are, only how many
were found so far. Bug tracking is often lacking, behind schedule, wont fix,
etc.

On the contrary modern code analyse tools are frightening good at spotting
untested conditions every where (sonarcube comes to my mind). Much better and
much reliable than human developer and human tester. Sonarcube does not care
if the case can not happen because user interface does not allow the input.
The bug is here, and the bug is spotted.

The more experienced person is plain wrong : more tests is better. It's not
panacea yet, but it's better.

~~~
inertiatic
function int untested_sum(int a, int b) { return a + b; }

function int tested_sum(int a, int b) { return a * b; }

function void test_providing_total_coverage() { assertEqual(tested_sum(2, 2),
4, 'Expected sum to be 4!'); }

~~~
Doxin
Which is why I'm a big fan of testing tools like hypothesis:

    
    
        from hypothesis import given, assume
        from hypothesis.strategies import floats
        from math import isnan, isfinite
        
        
        def tested_sum(a, b):
            return a * b
        
        
        @given(a=floats(), b=floats())
        def test_sum(a, b):
            assume(not isnan(a))
            assume(not isnan(b))
            assume(isfinite(a) or isfinite(b))
            assert tested_sum(a, b) == a + b
    

Running that test gives

    
    
        Falsifying example: test_sum(a=0.0, b=1.0)
    

Note how it also forces me to put a bunch of assumptions in there explicitly,
hypothesis is _very_ good at finding weird edge cases. For this example I
don't really care that it goes wrong when NaN gets involved, but if I did I
discover through testing that that case even exists.

------
Ensorceled
Reminds me of this scene from Frank Herbert's Dune:

“Because of an observation made by my father at the time. He said the drowning
man who climbs on your shoulders to save himself is understandable– except
when you see it happen in the drawing room.” Paul hesitated just long enough
for the banker to see the point coming, then: “And, I should add, except when
you see it at the dinner table.”

------
lowercased
> I refactored the code from untested and untestable, to testable with 40%
> test coverage. The senior architect is refusing to merge because the test
> coverage is to low.

what's likely even more frustrating is that other merges went in for years
with no tests or testing in place at all.

~~~
umvi
Seems like unit tests are only really valuable for library code.
Application/glue code... not so much, especially if it is something being
actively developed and you are constantly throwing away tons of unit tests.

~~~
AstralStorm
Totally untrue. Unit tests in strict sense, perhaps, but nobody writes true
unit tests - there's a high chance these are actually short range functional
or integration tests. Such tests are of high value.

~~~
umvi
Maybe it just depends on the context. I write embedded linux applications. One
application I wrote talks to a Qualcomm modem using their QMI framework and
then passes on messages to an antenna subsystem.

I wrote some library functions, which were easily unit testable, then I wrote
the glue code, which I found wasn't very testable at all. I found myself
having to stub out the entire QMI Framework/modem and the entire Antenna
subsystem API since obviously I don't have access to those things at build
time. So basically now my application/glue code is being testing in a fake
vacuum that may not even reflect the real behaviors of the modem or antenna.
So I waste tons of time trying to predict how the modem and antenna might
behave, and test my application code accordingly.

Lastly, I end up scrapping a bunch of the tests because the actual behavior of
the modem/antenna is different than I predicted and changes slightly with each
update.

------
jacquesm
The way to establish change is to do it gradually, with buy-in from all
parties _prior_ to introducing the change and to make small steps in the
beginning. If you showed up with a huge chunk of code that has been refactored
I'd probably not merge your change either. The test cases have been written by
the same person that did the refactoring, there is a _very_ good chance that
any wrong assumptions you made about the code will be present in both.

------
koz_
This allergy to reality is surely the biggest trap not just for startups but
people in general. It's surprising but I feel like the norm is for people to
care far more about feeling that everything's okay, rather than everything
actually being okay.

That said, one problem I see with the anecdotes provided is that the problem
solver is doing two things: discovering a problem and solving it. Conflating
these two things is a great way to encourage pushback: every change comes with
an associated risk, so the general rule is to say no (using whatever flimsy
reason) if it's not well understood and agreed that the problem being
addressed is a real problem worth solving.

~~~
conradp
> ... the problem solver is doing two things: discovering a problem and
> solving it. Conflating these two things is a great way to encourage pushback

This squares with my experience strongly. At a prior job, I remember two or
three times where a set of changes was almost rejected for production because
despite solving real bugs. The bugs hadn't affected anything in production
yet, but the fixes carried the risk of _maybe_ breaking something.

This resulted in some fun times, like a particular client running something
like three major versions behind on our software, precisely because management
didn't want to accept any disruption risks there, and the old version grumbled
along well enough.

------
taneq
> When you add visibility to a system, the numbers are always bad. That’s why
> you’re putting in the effort to add visibility to an existing system.

This is something everyone needs to internalize. Don't be upset if the numbers
are bad when you first instrument. That's WHY you instrumented.
Instrumentation is a ratchet to help you improve your product.

------
ltbarcly3
I think saying 'they're drowning' is still giving them too much credit. People
who built terrible broken systems don't know they are bad at building things,
and they probably don't have a clear idea of what would be better or worse to
begin with. They don't know it's better to have a system that can be debugged,
they don't even know what the difference is. If you give them examples of good
designs and bad designs, they can't tell you which is which. When they say
'this is good' or 'this is bad' they will astound you with the reasoning.

Software is an industry with no standards for employment. All it takes to be a
Software Engineer is for an incompetent manager to hire you. More than half
the people I interview barely have a basic familiarity with writing software.
These are people who have been employed as software engineers for years and
will certainly find another job. They struggle to write a nested for loop.
They write functions with thousands of lines. They just don't know what they
are doing.

The author here claims they don't want the visibility of the situation to
increase, but I think they don't know it's a problem to begin with. Their
stupid excuses for not moving forward are just their stupid thoughts said out
loud.

~~~
vanniv
My team has built garbage software at times. It isn't because we don't know
how to build good software, and it isn't because we don't care, and it isn't
because we don't want to build good software.

It is usually because of a continuous sequence of emergencies to which we are
forced to react.

An emergency occurs, and we have requirements to do something in 1/3 the
necessary time. We get 80% of the way though that and, before launch, there's
a new emergency, and we have to rush to that.

So we end up with crap software that looks like shit and only almost works.

But it almost works enough to keep our very large business humming along.
Sure, we hate it, and we're making more technical debt, and over time, we get
less effective.

But sometimes we go through cycles of a year or two when the business either
can't afford to do it right or isn't willing to pay the price to do it right.

Sometimes, they really are willing to accept paying 10x as much so long as
they can pay in a year or two.

(I hate it, but it is true)

~~~
AstralStorm
Make it 100x as much with 5 times as many people in 10 times as long.
Efficiency is never important when there is money to burn and it's not yours,
and you won't be blamed when it breaks.

It does not even take any emergency - often it is conscious or subconscious
project management strategy.

It's related to observations about Peter principle, Dilbert principle, Gervais
principle and office politics and dynamics.

Avoiding this tendency is essentially only possible with small teams with full
ownership of the problem they're working on and direct stakes.

------
Aeolun
So being the person that’s manning the lifeboat, what do you do? Kick them
down?

If you do nothing everyone will surely drown.

------
peter_retief
Could it be a competence barrier to protect incompetence? I have experienced
so many similar experiences, normally with middle managers who are terrified
they will get caught out for not knowing what they are doing. I noticed they
always stick with brands like Microsoft or SAP are deeply suspicious of open
source. Oh and they dont drown, they get promoted. Could this be an
opportunity for a startup to identify this risk "as a service"

------
stakhanov
Between sabotage and people wanting to sink the lifeboat, I have a third
option: These people have a warped perception of what's happening and
priorities around what's happening. Warped by their own irrational beliefs
which have come about as a result of cognitive dissonance reduction.
Initially, they can see how messy & bad things are but since they, themselves,
are responsible for things having gotten to that point, they start bending
their own beliefs in a way that will make themselves and their past actions
look good. In the final phase of this psychological process, some kind of
Stockholm syndrome kicks in in relation to the poor standard of engineering
that is making their lives miserable: They start to actively seek out poor
engineering because it's the only mode of existence that their psyche will
allow them to fathom.

Warstory time.

A tech company I worked at was being encumbered by a codebase with poor code
quality. Bugs were being introduced all the time, with no chance of preventing
it because nothing is testable/tested. Bugs were introduced at a faster rate
than they were fixed. The whole system was a timebomb because it was built in
a dying ecosystem. For certain kinds of code changes, their effect couldn't
even be made visible at development-time at all. You had to release them into
the production system and look at metrics to see their effect, hoping that
nothing would blow up, so the system was layers and layers of code that was
trying to be minimally-invasive with respect to existing code.

Whenever an engineer would be newly hired, they would, within their first week
of working there, suggest "Let's take a month to refactor this/that part of
the system". When an oldtimer would hear the new guy say "refactor", their
warped perception would hear "accomplish nothing" or "threaten our mode of
existence". They would give the following speech: "Hey, get it in your head,
noob: We don't do that here. This company is not focused on code quality, it's
focused on a mission. Every week we come to work and we accomplish a part of a
mission. I _KNOW_ the code is probably the worst piece of shit you've ever
seen in your 20 years. [This company only hires extremely experienced people].
The engineers who work here are all on board with our mission-driven culture
and pride themselves about their skill at dealing with the shitty codebase.
Your suggestion tells me that you can't deal with it, and you are unwilling to
absorb our mission-focused culture. You will not get a career out of working
here, unless you properly absorb that element of our culture".

So, in their mind, they'd taken "shitty codebase" which is a negative, and
turned it around and made it into a religion about a "mission-driven" culture
which is a positive. [Cognitive dissonance reduction at work, right there].
The reality was, of course, the poor code quality was a huge drag on this
company's ability to accomplish a mission of any kind: Because changes that
took their competitors a day, took them a week. Changes that their competitors
did in a month, would have been a suicide mission for them and they therefore
rationalized their way out of even wanting to do them. ...but I wouldn't
recommend to the newly hired engineer in the story to mention that particular
elephant that was in the room.

Now, here comes the really sick part, where the picture of the drowning man
falls apart: From time to time, due to a rare alignment of circumstances, this
company might be _forced_ into a situation of starting an implementation of a
new element of the system with a clean slate, i.e. completely unencumbered by
the current codebase. They would _intentionally_ implement the new codebase to
be just as shitty as the legacy code. They would implement it in the dying
ecosystem. They would generously insert unreachable code. They would pursue
the fastest path towards making it untestable.

Now, under normal circumstances, if I hear someone telling that story, I would
suppose there might be a motive for this. For example, I've seen developers in
hedge funds intentionally rig a system to explode if they should ever be
forced out of the company and be as hard as possible for anyone else to fix,
for obvious reasons having to do with their incentivization structure. But
that's not what was happening here. They had a culture of openly sharing
knowledge of the system, were hiring new people all the time, and working hard
to bring them up to speed about the system's working etc, so it wasn't that.

My guess is that it must be the "Stockholm syndrome" element I mentioned: They
started to develop a psychological attachment to a poor standard of
engineering. It was making their lives miserable. But it was also the only
existence they knew and they felt was available to them, so cognitive-
dissonance reduction was maneuvering their psyches into actually favoring that
mode of existence.

Some might have said things like: "One day, after we've grown enough, we will
do a complete rewrite." Or "Any day now, we might be acquired by another
company and will leave all of this behind". But it was more in the vein as
when a religious persons speaks of a glorious afterlife.

There is a lot of really sick shit that the human psyche is capable of, and
sometimes it surfaces in engineering. I think that's my takeaway from all
this.

~~~
palisade
I have so many war stories there's not enough space on HN to contain them.
I'll just state my overall experience. I write elegant well designed code, but
generally speaking you won't find that anywhere especially if you're working
in teams. Unless the code is all written by one person who had a chance to
design and implement it well solely by themselves with adequate time, it's
going to be a mishmash of they just didn't have time and the pressure was on.

I'm of the mindset that if it works, it's good code regardless if it isn't.
This I think is the healthiest mindset to have in any team. I will refactor
things slowly to clean up issues and of course my own code is well thought out
and integrated into the codebase regardless how reckless anyone else is being.
Any other way of interacting with a team would be antisocial and not
beneficial ultimately. The way I see it is they wouldn't be paying me if there
wasn't more work to do. The most important thing is that it works and the
customer is happy. Your disappointment with elegance or coverage comes second
to that.

The thing that actually annoys me is when, having exceeded my own delivery
deadlines and everyone else's if I provide value added like a new feature that
puts us far ahead of our competition I get pushback. Management doesn't want
to have to maintain it. Even though my code is really solid compared to the
rest of the garbage going in there. They just see me as a plumber fixing a
leaky pipe and don't want anything further. I get that there's ROI and whatnot
but it still bothers me when I show them some next level shit and I'm told to
mothball it. Especially when I'm the reason we're usually way ahead of
schedule. </rant>

~~~
stakhanov
I've had that same thing happen to me numerous times too. I wrote code that
demonstrably and undisputedly added to the value of the product in a way that
went far beyond anybody's expectation. I took no longer to do it than it would
have taken to do it the "normal" way. People get mad at me making the argument
"Well if you leave the company, somebody else is going to have to understand
your next-level shit, and we just simply don't have people here who can do
that. So then what?"

~~~
Discombulator
I think it depends on the details. If your code is simply cleaner, well
documented etc, then fully agreed.

If you introduce new dependencies, or the latest free monad transforming
framework when there was none, then I’d be wary as well.

~~~
stakhanov
I don't do "framework" ;-) In fact I'm usually the first to protest when
somebody tries to introduce frameworks or other bloatware. And getting excited
about algebraic structures is something I got out of my system while doing my
PhD. I'm strictly talking: Value-add to the product, albeit at the cost of
higher complexity to the extent that it is unavoidable.

------
deboflo
This is one of those things you don’t truly understand until you’ve
experienced it first hand. I have. Very appropriate metaphors.

------
merrill_g
In my world (marketing, not dev) the saying is "you cannot optimise something
you do not measure".

------
pvaldes
"I think that is not sabotage, but I don't have any real data that support or
dismiss my statement" would be a better title.

Your mileage may vary.

