
Why Testing Is No Longer Sufficient for Today’s Software Delivery Pipelines - caution
https://blog.overops.com/why-testing-is-no-longer-sufficient-for-todays-software-delivery-pipelines/
======
eesmith
This article goes on about "why traditional testing is not sufficient". I
think their view of 'traditional testing' is a strawman.

Traditional testing _never_ said that 100% code testing was sufficient.

Even 100% branch testing isn't sufficient. SQLite gets bug reports (eg,
through fuzzing) even though they have some of the highest covered code
_ever_.

Viktor Fracic's comment “Code coverage means nothing" _is_ the traditional
view on coverage. Code coverage only means something when you are testing so
that code coverage can help. Otherwise, it's easy to get high coverage with
meaningness tests.

And that's been the view of code coverage for a very long time.

Ditto for "High quality testing takes time and doesn’t guarantee success." Has
anyone ever said that high-quality testing guarantees success? Perhaps a few
odd-balls, but it's never been the majority view.

"I’d rather see people move fast and be comfortable with failing" depends very
much on your market. (Boeing avionics, _ahem._ )

In other words, testing has _never_ been sufficient. Only necessary.

------
xg15
Seems like they are essentially arguing for the ol' "move fast and break
things" \- though I guess in a slightly more responsible manner: You can't
catch all problems with thests, so it's important to be able to react quickly
to problems - and in fact, being able to react quickly is already important
from a business perspective anyway.

Sounds reasonable, I just wonder what kind of practical differences this
advice does make.

I do hope this isn't a lot of talk to go back to the Facebook style of "moving
fast and breaking things" in the end. The "let your users be the QA" part
somewhat sounds like that...

------
pezo1919
Software engineering shall rely more and more on mathematical proofs. Go
Idris! :)

------
richardjennings
there is no alternative to testing;

that coverage as a metric is insufficient to quantify the quality of testing
is not a surprise to anyone.

------
mikestew
_" Viktor Fracic: “Code coverage means nothing."_

I guess I don't have any need to listen to that Viktor guy, then. If Viktor
runs his tests, gets 50%, and wants to try and convince me it doesn't mean
anything, he's going to find it falling on deaf ears. I don't care how
"meaningful" ones tests are, if they only cover half the code someone is
queueing for a bad day somewhere down the line.

OTOH, I have heard with my own ear holes a test manager say that they had hit
100% coverage and therefore testing is complete (paraphrasing). A test manager
at Microsoft said this in a meeting with other people present. So I can see
where Viktor might be coming from, but he has still thrown out the baby with
the bathwater.

~~~
Consultant32452
I don't know everyone else's experience, but I found unit testing encouraged
me to write better quality code. Testable code is better code. Methods are
smaller and have more focused purpose, leading to more reusability. Writing
unit tests with >98% coverage forces me to think more deeply about and have a
greater understanding of the code I'm writing and how it all interacts. I am
more likely to consider edge cases where things might go awry and deal with
them. So even if someone believes that tests don't buy them anything directly,
I can only speak for myself but the second and third order effects have been
great for my career.

~~~
eesmith
I found that unit testing with small methods meant I was writing my tests to
those methods.

But the methods were specific to one implementation approach.

If I wanted to rewrite the algorithm, eg, for clarity or performance, I ended
up replacing many of the tests, which felt like test churn.

I prefer trying to figure out where the semi-permanent/API-like interface are
in the code, and test through that. The result is closer to integration
testing than unit testing, although I use a unit test testing framework for
it.

~~~
Consultant32452
I had the opposite experience. One of the many good practices unit testing
encourages is encapsulation. With encapsulation and mocking during testing,
there is very little chance of a cascade affect when changing a method or
doing some refactoring.

~~~
eesmith
The question is, what's the encapsulation level?

If you only change the internals of a method, then it's clear that the method
call is the API.

But the specific method calls are rarely the deliverable (outside of library
packages). Those are usually much higher, eg, web API or command-line options.
Instead, the methods can be rather fluid.

My experience is that the "unit testing with small methods" approach usually
emphasizes testing each method.

The analogy I use is that of building a bridge, where you don't care what sort
of bridge is made, only that it can handle a specific amount of traffic.

So I build a truss bridge, with tests that each of the struts is correct.

But this is programming, and I decide it's better to replace the truss with a
suspension bridge, so I do that.

Which means I have to throw away all the strut tests, since I don't have them.

While if I had only used load tests from the start - integration tests for
what the customer actually wants - then I could do wholesale rewrites without
changing much of the tests.

~~~
Consultant32452
I want to be clear I'm not disagreeing with you here. But I would like to
point out if you were building a real bridge every component would be
inspected individually. They wouldn't just let you build a whole bridge and do
a final load test. Whether that level of robustness/ confidence is important
will be highly contextual.

~~~
eesmith
Yes, and part of the mythos of unit testing - especially around small methods
- is based on that real-world analogy, when "this is programming", and many of
the rules are quite different.

As another analogy, if you breadboard a device you can test each individual
component directly. Once it's on a chip, most of those individual tests can no
longer be used. That's not to say they can't be tested, only that the testing
must be done on a higher interface level.

So, do you use those higher-level tests, which are more like intergration
tests, or do you write the low-level unit tests and expect to throw the away
at any sort of non-trivial code changes?

