
Measuring the Cost of Regression Testing in Practice - luu
https://dl.acm.org/doi/pdf/10.1145/3106237.3106288
======
joshuanapoli
I think that there is a wide spread in overall software development process
that can lead to good product quality. Some teams will automate testing to a
greater degree. Some rely on consistent manual testing. Others seem to hire
and raise fastidious developers that produce fewer defects to begin with. Each
strategy takes a while to build into the culture, and during that build-up
seems to be a heavy cost. So I think that it’s hard to judge the benefits of
test automation separately from the software development organization and its
product.

------
blamestross
The nasty part is assigning "monitary costs" to the bugs CI discovered.
Without that you can't compare it to the cost of writing, maintaining and
running the tests.

------
marcosdumay
Why do people ever write flaky tests? What kind of information one can hope to
get from them?

~~~
Ididntdothis
Stable and meaningful tests are very hard to write. Probably harder than
writing the software that gets tested. It’s even harder to maintain them and
keep meaningful. Tests that made sense a while ago may become flaky or
obsolete when the system changes.

~~~
2rsf
Statistics is simply working against you, system tests are a long chain of of
(let's say) independent events with (let's hope) a small chance for failure.
But your test multiply those probabilities creating a much bigger probability
for failure.

Now you run multiple tests, each with a more complex environment than your
real system (you need to control your tests, test logs etc.) usually on a
lesser environment than your production one. Add all this and you'll get an
uncomfortable probability of failure

~~~
abjKT26nO8
_> But your test multiply those probabilities creating a much bigger
probability for failure._

Minor nitpick: when probabilities multiply, they get smaller. Such is the
nature of numbers in the range [0, 1].

Multiplication would happen when counting the probability of two (independent)
events coinciding. What you're thinking about here is the probability of any
one of several events occuring. That will be a (rather convoluted) sum, not a
product.

~~~
2rsf
We engineers can do convoluted sums, the assumption is that the events are
independent and you multiply the probability of success (or 1-probability of
failure).

