

How test coverage is meaningful (or isn't) - llambda
http://martinfowler.com/bliki/TestCoverage.html

======
reiz
I think testing and code coverage is important, but over evaluated. 1 year ago
I worked on a Java Project with over 2500 test cases and over 82% code
coverage. The portal was still pretty buggy.

82% code coverage means nothing. To get such a high code coverage we tested
really everything. We even wrote unit tests for DTOs, Entities and so on. To
have a bunch of tests for simple getters and setters is pretty use less in my
opinion :-)

I agree with the part, that to write good tests is not easy. That is not a job
for students or for bad payed employes with less experience/skills.

In my current project I have much less tests. I am just testing the important
parts. But that I am testing very carefully. And I have now much less bugs
then in my old project.

~~~
jeffskj
The only problem with not writing tests for getters/setters is that is makes
it harder to identify actual code that is untested. If code coverage tools
were smarter about not counting getters/setters towards the coverage number it
wouldn't be a problem but if they aren't tested your coverage report might say
you have 25% coverage in a package and you don't know that's because it has a
bunch of DTOs in it or if its because there's bunch of actual untested logic
in that package. The way we solved that problem on my team was to write a
little utility that used reflection to automatically test the basic
getter/setter pairs which made writing the tests for DTOs basically one line
of code.

~~~
astral303
Many coverage tools have ways to ignore getters/setters and things like "if
(debug logging enabled) log" types of statements.

~~~
jeffskj
I'm not aware of any open source ones for Java that can do that, Atlassian
Clover (not open source) might do it but I'm not sure on that. If you know of
one I'd love to take a look at it.

~~~
reiz
I think the same. And I think getters and setters are useless! Just wasted
lines of code. One more reason to switch to a dynamic language like Ruby or
Python. The applications I saw in the last 14 months in the Ruby community are
much more object oriented than the stuff I saw the last 10 years in the Java
World. And because they are more object oriented their are much more easy to
test. It is so easy to write test cases for a Ruby on Rails App, but it is a
nightmare to do the same for JSF & EJB.

------
pacaro
Code coverage can provide two things

1) It can be a signal, a low number raises a red flag, but a high number
doesn't raise a green flag.

2) It can be a great analysis tool. When you have partial coverage, digging
through the coverage reports and seeing where you _don't_ have coverage can
tell you a lot about what isn't being tested, what the test writers
assumptions were, etc.

------
yummyfajitas
High test coverage is useful in one important regard.

If I've got a module and 25 untested __unicode__(self) methods, I might not
notice the untested big_complicated_method(self, _args,_ *kwargs) in the
coverage report. That's less of a problem if I have 100% test coverage (down
to 95% after writing big_complicated_method).

~~~
pbiggar
I think the important thing here is the change: 100 -> 95\. A 5% decrease
should be noticed, in some fashion. You don't need to be at 100% coverage to
do that.

~~~
yummyfajitas
The change is important, but so is the absolute number of untested lines.

A) Untested lines of code: 10-25

B) Untested lines of code: 2-4, 10-25, 54-68, 59, 63, 68, 71-77, 85

I know that in case A, I'm far more likely to look at 10-25 and come up with a
good test case.

In case B, I'm not sure I want to skim 2-4 (skip, it's ok), 68 (just a
__unicode__), 59 (oh, that should never be executed), ah, 10-25 is what I
should look at.

It's sort of the broken windows theory applied to testing.

That said, I don't have 100% test coverage in most of my projects.

~~~
pbiggar
A tool which told you there was an absolute change of 5%, but didnt tell you
where that change was, would not be all that useful.

I'm sympathetic to the "all-or-none" broken windows philosophy that many
developers use, but I disagree with it. If you need to get to 100% of some
metric in order to get some visibility on its changes, you're just not going
to get that metric.

At Circle, we make hosted CI, and I frequently talk to customers who use
jenkins or something, and whose tests are broken. Inevitably, they broke the
tests one day, their priorities dictated that they couldn't fix it right away,
so they left their test suite red. Because it was red, they didn't notice the
suite getting more and more broken over time.

If their test suite hadn't been binary, they would have learned immediately
when they broke more tests, and they would have been in a much better place.

TL;DR I blame the tools.

------
stcredzero
_The reason, of course, why people focus on coverage numbers is because they
want to know if they are testing enough._

Bug rates and what they are doing should be the final arbiters. How often do
Unit Tests find regressions? How often do bugs escape into the wild? As with
many things, what is the return on investment?

