
Consultants Ate My Unit Tests - dragthor
http://metroize.com/consultants-ate-my-unit-tests/
======
jasonkester
Ah, the sadness of seeing your code again after somebody else has had it for a
couple years. I feel the author's pain.

I had a company license the source code from one of my old projects to use as
the basis for a new site they were building. I built out the bits they needed
on a consulting basis and got them launched. Then they started thinking about
how expensive I was, and how much they would save if they moved the
development overseas.

Fair enough. My philosophy on such matters is that it's best to let a client
learn this lesson through painful experience. All the best, guys.

They made it about a year before the site stopped working completely. Reading
this article reminded me of the first time I opened the source that came back.
You just wouldn't believe that you could do that much damage in such a short
time.

I quoted them double my old bill rate and got them back up and running, with
all the new functionality they had been wanting, and re-launched. They brought
in a new guy to run things, and he started making comments about expenses, and
how surely there must be a way to do this cheaper. He even found a local kid
who knew some C# and would be willing to help out for a lot less than I was
billing.

No problem. All the best.

I took a quick look at the codebase when they got back in touch a few months
later and politely declined to come back on board.

~~~
chrisabrams
It's sad how often this scenario comes up :/ get what you pay for, plus
another, larger bill later. The technical debt collector will find you.

~~~
jxramos
yes he will find you and he holds a mighty ransom!

------
taylodl
I've seen this with my own two eyes. When asked why they removed the unit test
they said after they made some rather extensive code changes the tests were
failing so they deleted them. Nor was there any trace of irony in their
response. When asked didn't they realize that was the whole point of having
unit tests they replied that they knew, but they had deadlines to meet and
there wasn't enough time. Since controls had been put in place to not ship
with failing unit tests but not to check for test coverage they took the easy
way out and deleted the tests. I informed management what they had done and
thankfully they were relieved of their contract and their consulting company
was informed what they had done. I don't know what happened to them after
that.

Thing is, once the PM was informed what they had done we did a couple of
sprints where we restored the unit tests (which we still had in our history)
and got them passing. They had no idea the consultants were doing such shoddy
work in order to meet deadlines. The upside of all this is to this day that PM
now actively checks the Jenkins dashboard to see the current state of tests
and that no tests have been deleted - and that's for all his projects not just
that one particular projects. Now that the story is part of our IT lore the
new consultants get the message loud and clear we're not afraid to send them
packing.

In the end this turned out to be a good story for us - but wow.

~~~
Spooky23
Always assume that consultants will never perform any task that isn't required
for milestone sign off.

~~~
daigoba66
I'm a consultant; this is good advise.

It's not laziness or apathy. It's just that we're only paid to do a task.

~~~
catmanjan
Good luck explaining the point of unit tests to technology illiterate managers
- they don't understand and really they shouldn't have to.

~~~
zeisss
I always use double accounting from finances to explain tests. You don't do
logic changes in one places, but in a second to double check it actually works
out.

~~~
rhizome31
Sounds good, if only I understood double accounting :-)

~~~
andrewflnr
The principle is simple. Basically your entire accounting log is always a
balanced equation: sum(assets) = sum(liabilities) + sum(owner's equity). Every
transaction gets recorded on both sides of this equation, so you can check for
(certain kinds of) mistakes by adding up everything on both sides and seeing
if it matches. This is a vast oversimplification, and possibly slightly wrong,
but hopefully illuminates the connection with unit testing. :)

------
joeld42
I don't agree with the implicit premise that if the cheaper consultants had
been required to maintain the tests and provide coverage for what they wrote,
the code quality would have stayed high. In my experience, it would have
turned into just as much of a mess, except would have taken three times as
much work to maintain all the tests in the process.

Maybe the tests would have had some benefits, but would they have enough to
justify all the extra time to write/maintain them? Certainly not.

In my experience, good programmers write good code, tests or not, and bad
programmers write bad code, tests or not. I'm not against testing altogether,
it makes sense in some contexts (like very widely used libraries). It's also
useful for beginners who aren't sure "where to start" with a project. But even
in cases where people swear that their tests are invaluable because it's
tricky code that breaks often, that's a sign to me that that code needs to be
made less fragile, not more well tested. TDD is a cargo cult.

~~~
joesmo
I couldn't agree more. Almost every other form of testing, automated or
manual, is more effective at catching bugs than unit testing in my experience.
I rarely find unit tests catch any actual bugs (two I can remember in two
years in two major codebases) and they are actually a liability rather than an
asset because of that in many cases due to the time it takes to create and
maintain them.

Often, tests are so far removed from their actual run context with mocks and
stubs that they're only testing the functionality of the mock/stub framework
itself. OOP code is hardly ever unit test friendly by definition and a lot of
bugs are at integration points and due to app state that won't show up in unit
tests. There are few places, in other words, where they are useful outside
known algorithms with expected outputs in OOP. For one bug prevented per
application per year, it's easy to see why spending weeks a year on preventing
said bug might not be worth it. I'd rather use them sparingly and test in
other ways that are more effective at catching bugs.

~~~
twistedpair
> I rarely find unit tests catch any actual bugs

What kind of code are you writing? Sure, a Spring service that just integrates
a bunch of other services is hard to test due to mocks galore, but any dense
logic blocks should be moved to standalone, testable classes.

For my math libraries, Fibonacci compression algos, etc, I find the hundreds
of unit tests invaluable. Add to that, in tools like SBT, "sbt ~testOnly
com.MyTest" will rerun tests on every change, giving me feedback in under a
second, rather than the whole dev/test cycle of
save/restart_server/click_on_UI. Write unit tests for the tricky bits that
will benefit from them.

~~~
joesmo
These are known algorithms you're talking about, the one place where unit
tests actually consistently make sense. I'm not writing a math library,
however. I'm writing APIs and web GUIs. We have an immense amount of business
logic and we do unit test a lot of it, but as I mentioned above, those unit
tests hardly ever catch any bugs and they take a lot of time to maintain.
Every time there's a change in the logic, the tests have to be updated.
999/1000 times, when a unit test fails after a chance it's because it needs to
be updated, not because there is a bug. Keeping up with those updates is
simply not worth the time to catch one bug per year. Neither is writing new
unit tests in these areas. Writing them for well-defined algorithms with a
limited (or at least definable) set of known inputs and outputs, as I
mentioned in my original reply, is still useful. Other types of testing that
our developers could do to make sure the code is high quality are much more
useful of course and therefore preferred because they actually help make the
codebase better rather than creating the illusion of doing so as unit tests
often do.

~~~
githulhu
I don't use tests, and I find that when I need to make a change to some code
that has, say, four or five different paths through the logic, a lot of times
one of the paths will break. Testing all of the paths manually each time there
is a change is incredibly tedious and time-consuming. I feel like if I had
unit tests I would be able to just see instantly if anything was broken. But,
I never can seem to find the time to write them...

------
knappe
I've been on the other side of this as a consultant. I once was consulting for
a place where I asked if I could run the existing tests to confirm I hadn't
broken anything when adding in some new functionality. The owner of the
project yelled (actually, really yelled) at me to never run the tests, ever.
Apparently they deleted production data.

I didn't spend much time consulting for them.

~~~
AnimalMuppet
I would have given them a bid for fixing that problem...

~~~
troels
The yelling problem?

~~~
AnimalMuppet
The "can't run tests without deleting production data" problem.

The yelling problem, I'd be tempted to fix for free...

~~~
joshmn
> The yelling problem, I'd be tempted to fix for free...

So, run the tests?

~~~
AnimalMuppet
My temptation was more toward a knuckle sandwich. Not a productive response,
and probably death for a consultant's business, but I would be tempted...

------
chvid
Some of the reasons why I delete unit tests from time to time:

\- Often the quality of unit test code is a lot worse than the production
code.

\- Some times people test too much in a single test (ie. more than one unit)
making it very hard to change the design of the application.

\- Some times a unit test misunderstands how the application is supposed to
work or the requirements change.

\- Some times a unit test tests something too little or too trivial.

\- Some other test is already covering the case.

I personally don't weight 100% test coverage as critical. Often other aspects
of QA in application development are more important (i.e. integration test or
documentation).

~~~
msie
I don't know why you were down-voted. TDD fanatics?

------
alexc05
I, very recently, began writing a suite of tests to go along-side a complex
mobile app. During a stand-up meeting I (somewhat) proudly announced that for
a particularly onerous piece of business logic, I had 100% coverage with tests
that run in less than a second.

The project manager in the room was worried about the time it was going to
take. All the developers in the room piped up to say "oh, no, this actually
saves time"

The next day, I found myself in a private meeting room with the project
manager and my line manager.

Both were trying to tell me that unit tests are not part of the scope of the
project, not were they approved and, that I was to stop writing them.

Manual QA was the way to go at the end of the project.

Yesterday, I was in the office from 10am to 2am on a national holiday
addressing issues arising from QA.

It was unproductive to spend the day wondering how many of those defects I'd
have been able to catch months ago if I'd been writing tests all along.

When you're the only one on your team who respects the tests, the world feels
like a very bleak place.

~~~
douche
It does suck how much of the time you wind up in the situation that, to make
any progress, you have to work in secret behind some smokescreen of other less
useful work, to keep the micromanaging and the hemming and hawing from
consuming any improvements.

It's not uncommon that orders of magnitude more time wasted in debate and
FUD'ing round on things than it takes to actually do them.

------
cognivore
I'm going to be a contrarian here and say that if unit tests, in most cases,
made economic sense they'd be a given in almost all software development.

But they're not. Do we assume business owners are dumb? Do we assume managers
are dumb? That's a popular position for technical people. "Oh, you just don't
understand how valuable those unit tests are!"

But the cost of maintaining unit test vs. the cost of fixing bugs later might
favor just fixing the bugs later. Maybe it doesn't sometimes - but one is an
upfront cost you know about, and one is a nebulous cost that you might or
might not have to pay later.

Given that, I know what most people paying the bills are going to choose.

~~~
bo1024
It's a question of incentive alignment. Managers are not dumb, their
incentives are aligned to short-term success rather than long term. Consider
the difference in scenarios between producing ok code very quickly and good
code more slowly. The manager will immediately feel the negative consequences
of taking the second decision, if their incentives are badly designed.
Meanwhile they may never feel the negative consequences of the first decision.
When bad code breaks down, how likely is the manager of three years prior the
one to take the blame or heat for it, let alone face real consequences? Often
they won't.

------
Gratsby
I had a different experience with tests. 18 years ago at one company I laid
out a test plan. Some of it was automated, the majority of it was manual. It
applied specifically to a large custom built application.

2 years ago, I was at an entirely different company. They handed me their test
plan documentation. Lo and behold, it was my document, now laden with another
consulting company's logo. It was missing a few tests, but there was nothing
new.

I was angry for a solid 4 hours. Then I had to laugh about it. I would be
flattered that my 18 year old work was still being used, but it was quite
simply dated. If a big company was being cheap and getting things built by
offshore resources who had no better way to accomplish it than by stealing my
work, then they got what they deserved.

------
astockwell
I have this same underlying feeling every place I work. Maybe it's because I'm
not in the Valley, but in the last several jobs I've had, I was the only
person advocating for things like unit tests, release pipelines, or even
VCS(!). And each time I get a new job, team members and bosses from the old
one inevitably call with similar self-inflicted crisis like this.

~~~
dietrichepp
I don't think it's because you're not in the valley. I think it's because only
a few percent of developers in the world care enough to become leads and make
this stuff happen across the team.

------
vinceguidry
Man, this makes me sad. Not about the atrophied codebase, but about the
inevitably extreme difficulty of keeping one properly maintained.

Expertise in software maintenance isn't common and it isn't cheap. Every
greenfield dev is going to eventually see his project handed off to people who
won't understand or respect his conventions and methods. That mostly destroys
it, but business requirements can put it on chemo and a food tube and keep it
alive against its will for a long, long time, and it's not a pretty sight.

It's practically impossible to. I've been maintaining an ancient Rails
codebase for the last two and a half years. I'm perhaps the fourth or fifth
such guy to do so since it was built. The project was never updated past a
certain point release and it's more work than it would be economical to do to
do that now. I'm experienced enough now to take it on, but both me and the
company have better things to do with our time and resources.

It is so ridiculously easy to make a bad decision that seems like a good
decision that instead ultimately dooms the project. Software is like this
weird life-form from another part of the galaxy whose motives are so alien to
us that we can't even begin to come to grips with it. We know it's not
actively plotting against us, well, pretty sure, but more than that? All you
can do is shrug your shoulders and hope it doesn't consider you a threat
today.

------
bunderbunder
I don't think this should be surprising to anyone.

Outside developers don't have any real ownership of their work product. Even
if they know better in principle (and many do), they don't have any incentive
to practice good stewardship. And bargain-basement outside developers aren't
magical productivity pixies; the reason they are able to charge lower prices
because they don't need to charge you for all the things they aren't taking
the time to do.

------
ErikAugust
Moral of the story: Great developers are profit centers, bad developers are
cost centers.

Unfortunately, many execs without development experience neglect to see it
this way.

They see "expensive cost center" that can easily replaced with cheaper labor.
Cool thing is, as others (like the author) have said, they tend to get burned
and end up re-hiring you on even better-than-original terms.

Kudos to the author for doing good work!

------
daxfohl
FWIW I've been eating my own unit tests these days. I find they slow me down
and distract me from what I'm doing.

You've got the "large scope" unit tests that test functions that take lots of
dependencies, via mocking. These often end up being so mocked, that you're
testing the mocks more than the function itself. You feel smart when you write
it, but a week afterwards you can't even remember what it's testing.

You've got the "small scope" tests that test small pure logic functions.
Great, but these hardly ever change. It doesn't seem worth the overhead to
maintain a test for unchanging code.

There's a "fuzzy middle" that _could_ be reasonably worth unit testing, but
these are the functions that are most likely to get completely ripped up
during refactoring and new feature dev.

To me, a sound architecture (no global variables, a minimum of mutation[1],
etc) and a sound type system means things generally work the first time
anyway, without the need to maintain a big honking test suite and everything
required to run it.

I'm finding it's much quicker to get things out the door without this
overhead, the architecture ends up being simpler ( _far_ simpler in some
cases) because I don't have to put test hooks in for everything, I can
refactor without the worry of how many unit tests will break, and I've
experienced no downturn in quality.

Of course there are cases (public core libraries, large distributed dev teams,
really complex pure logic) where unit tests are absolutely necessary, but for
_most_ of my projects these days I'm finding them to be redundant.

[1] In fact, one could argue that the greatest benefit from the TDD phase was
that TDD is harder with globals and mutation, so it reduced incidences
thereof.

------
stcredzero
_maybe the more disorganized the code, the more “less expensive” consultant
work there is to do (charge)?_

I know of one consulting shop that had a dependable stream of income from a
major European bank, because that bank had a project that didn't use source
control, so they got to be the periodic hero and get paid for it.

------
scotty79
That's normal. Any non-essential complexity gets dropped. Accidental
complexity accumulates.

In one project next developer ate my build script. Why bother with compiling
less to css if you can just edit css and forget the less files and build
script. Same went for r.js bundler.

In other project I participated angular got dropped because people couldn't be
bothered with it.

------
rsyring
The bulk of the first comment of the article was as good as the article IMO.
Quoting here:

>> But maybe the more disorganized the code, the more “less expensive”
consultant work there is to do (charge)?

> This is implying maybe they intentionally disorganize the code so they'll
> get more billable hours?

> I seriously seriously doubt it.

> Rather, it's as simple as: If you want to add features or make changes
> paying the cheapest possible amount, you simply aren't paying for well-
> organized code.

> It's not just less experienced programmers, it's less experienced
> programmers trying to get you the feature in the least possible hours. They
> aren't doing artificial things to bill you more hours, it's in fact quite
> the reverse -- they really are trying to get the feature done in as few
> hours as possible, to bill you as few hours as possible.

> And when you do that, you simply don't have time to organize the code well,
> or keep the tests working. You're shoving it out the door as quick as you
> can. Of course it takes more time to keep the code well-organized than to
> just hack it until the feature works as quickly as possible. Of course it
> takes more time to maintain the tests -- over the short term, and all you
> ever have is the short term when you're paying and getting paid little by
> little feature by feature.

> They have other customers, they don't need to bill you artificial hours.
> They really are billing you as few hours as they possibly can -- because
> that's how they get customers, being the cheapest. That's exactly what the
> client asked for, it to be done as cheaply as possible.

> You get what you pay for.

I have only recently started working for a consultant that bills hourly. It's
an eye opener. We are a small shop of experienced developers, we work for
people that do want us to produce quality, and we do, we do good work, and we
do write tests and maintain our tests. But it is a still a constant struggle
inside my head between doing it as right as I really want to, and not charging
the client more hours than the feature seems 'worth'.

> If you as a client prioritizes price even higher over quality (and you have
> no way to judge internal quality anyway, not being a coder, so why wouldn't
> you? It looks nice and works, what else can you judge?), if you're hiring
> less expensive developers to save money... they might be skilled devs in
> fact, but if you are trying to get it done as cheaply as possible, you are
> not going to get well-organized code. And you as a client don't care, what
> do you care about well-organized code? Until it reaches a breaking point
> where your technical debt is so high you can't get any more features at all,
> and you realize, oops... and probably still don't understand what you did
> wrong or could have done differently.

> Good software is expensive. Too expensive. More expensive than most people
> who need good software can afford. Which is why we have so much shitty
> software.

------
raulk
Cheap labor != consultant. In fact, real consultants tend to be expensive. A
real consultant is an expert in a field, not an asterisk or wildcard for
anything.

------
revelation
Of course, then when the OP was asked back for one particular thing, he didn't
go and setup the unit tests again, either. He did the same exact thing as the
consultants.

~~~
tedmiston
He was contracted to fix one thing, not to rewrite the unit tests. At the end
of the day, he can't change what the company does or doesn't value.

~~~
revelation
Yes, as are the consultants. Why put them down when they are doing exactly
what was asked of them?

Of course they don't bother with unit tests when asked to change thing A that
breaks a bunch of them.

------
x3n0ph3n3
The format of this article was downright annoying.

~~~
chillaxtian
yes. i don't understand the trend of duplicating quotes from the article
inline in a larger font.

~~~
bunderbunder
It's mimicking something many print magazines do. But without any apparent
understanding of why or how print magazines do it.

~~~
delecti
> But without any apparent understanding of why or how print magazines do it

I have honestly never had any idea why print magazines do that. Maybe I've
never seen a print magazine do it "right", but I think it seems just as
pointless and annoying in print or online.

~~~
plorkyeran
A common way of consuming magazines is to flip through them looking for
something interesting, and a good pull quote will intrigue you and make you
want to read the surrounding article.

