
QA = Time and Money. How much should you invest? - ThatMightBePaul
https://blog.rainforestqa.com/2015-05-18-qa-is-a-hedge-against-risk/
======
calinet6
QA is interesting, sure. I'm more interested in the Q part, and you don't get
that just by having a role or department checking work before it hits the
customer.

This article does a great job of describing the minimum level of quality
allowable to scrape past certain stages of growth.

The question you should be asking is: what level of quality, built in from the
beginning, intentionally and with full support, will create a product so
valuable to the customer as to generate a wild feedback loop of success—not
just scraping by at a minimum acceptable level. What level of quality will
generate returns, not just reduce your debt to an acceptable level? And more
importantly, how do you achieve it?

Quality _is_ your value to the customer. If you think of it systemically, you
won't need a QA. As Dr. Deming said, "Inspection does not improve the quality,
nor guarantee quality. Inspection is too late. The quality, good or bad, is
already in the product." And Harold F. Dodge: "You can not inspect quality
into a product." Deming advocated instead for a holistic understanding of the
factors driving quality, including management and leadership, the processes
used, and continuous improvement of systems.

This is the type of discussion I always see lacking from discussions of
software QA. On the manufacturing timeline, it's like we're in the 1920's.
It's all very realistic, but we could be so much further along.

~~~
ThatMightBePaul
You mentioned 'level of quality'. That can be pretty difficult to quantify. In
the absence of well defined metrics, a lot of thinking about QA is
qualitative.

"I _feel_ like we have a high level of quality here".

That makes conversations _about_ QA pretty challenging. Metrics is the one
area I'd LOVE to see move forward.

~~~
zachrose
Heh, maybe that's because the nature of quality is qualitative rather than
quantitative.

There are things that can be measured that might be correlated to quality
("bugs", sales, support, etc) but ultimately, classical quality seems to be to
be a marriage of what was expected or desired with what was actually there. As
a developer, my take is that quality usually stems from "developing" those
expectations as much as making the relatively concrete thing to compare them
too. When anyone can quantify the first half of this equation, please let me
know at once.

------
ffn
Can someone please explain why one prefers unit testing to integration /
acceptance testing when you're a seed stage company? I mean, your primary
objective as an early stage company should be getting an operational product
out to the market, so it makes sense that you test that your users can operate
your product - aka integration / acceptance test with crapybara or phantomjs.
On the other hand, if you're just unit testing you're bound to miss the bigger
picture (usually this shows up in the form of misspellings in your html views,
some CSRP bullshit, external apis misbehaving, etc.), and will wind up with
all tests that pass but a product that still doesn't work.

~~~
geoelectric
Unit testing is easier to do with the developer resources on hand since they
probably already know how to write them, probably necessary for quick pace in
dev anyway, and gives you a large portion of the risk mitigation testing in
general will give you. You also check in unit tests _with_ the code if you do
it right, whereas integration or acceptance tests require additional passes
once all the code covered is checked in, so it's more painful to the schedule.

Integration tests also require more development to allow integration while
still being hermetic (isolating out any external factors, since you might have
to architect to share injected dependencies between modules) -or- more
maintenance if you don't make them hermetic and they have to track external
dependencies. They require more time to triage failures as well if they're not
hermetic, as external failure is a possibility; you can't just trust the test
results. And final acceptance tests should never be hermetic--they should
accept based on realistic environments--so they're almost always a little
high-maintenance.

So while the value is there, the cost is much higher and it'd be more of a
distraction from primary development. Cost _and_ benefit have to be
considered.

So in this model the integration/acceptance part is handled by dogfooding,
which is the lowest cost way to get a decent chunk of it, even though it only
walks happy-path for the most part.

But that's why I said I'd add exploratory to B2C seed, is to get back some of
the integration testing on non-happy paths and obscure paths.

Edit: find docs on the test pyramid and consider that the shrinking number of
tests as you go up also implies somewhat lower technical debt as you skip
going up. The shrinkage is a function of the cost vs. benefit--you're just
taking it to its logical conclusion of omission.

~~~
ThatMightBePaul
This. Unit testing is better for a lot of reasons. Dev familiarity is
definitely a big reason. I find they're a bit more robust, too.

Integration tests are more brittle. Which, can be good ("Hey, there's a bug
here..."). For an early product, your app is probably simple enough that you
get a good enough feel while dogfooding. 100% correct that also includes some
informal exploratory work :D

RE: Complexity & the relative need for testing, I really like the perspective
'Out of the Tar Pits' takes.
[http://shaffner.us/cs/papers/tarpit.pdf](http://shaffner.us/cs/papers/tarpit.pdf)

------
geoelectric
This is probably the most honest article I've seen, ever, regarding how much
QA/testing companies at various stages need.

I'm impressed, because there's a real tendency within QA culture to say "the
ideal is all the things all the time, we're the gatekeepers," which has forced
quality into a bad all-or-nothing situation. We all know intuitively that it's
not all needed a lot of the time and the success stories are clear, so the
trust in any of it is undermined. So then you get a dev culture saying "QA is
unnecessary, we can handle all of it," which is generally true at the
beginning but can send your company off a cliff if you don't manage
transition.

The sad part is that even once quality processes get around that, the all-or-
nothing attitude leads to a pyrrhic victory. Quality teams still end up only
doing what can be afforded (because math) but everyone _thinks_ they should be
doing more. That leads to the pervasive opinion of ineffective quality teams.
And, unfortunately, the scramble of _trying_ to do everything at once (and
therefore mastering none of it) often makes that a self-fulfilling prophecy.

And I love that it stresses that QA is about hedging bets. Bugs get out,
period. The gatekeeper attitude is what leads to the do all the things
attitude in the first place. "At any cost" is generally a bad way to
strategize.

As for this chart, I'd have added exploratory/informal to seed B2C--
exploratory is biggest bang for buck for finding new bugs, and incubating
products are -all- new bugs--but I suspect they're lumping what I'd have
recommended under dogfooding.

So yeah, very nice. Of course, the best thing to do in any situation is
consider the context of your company, product, and market. Think about how
maintainable it really needs to be and what would cause the most damage:
losing customer money, leaking their data, embarrassing you in the market,
eroding their trust. Are there only a few customers to lose or are you mass
market and can afford a round of sufficiently obscure failure?

Those are the things to prioritize when you start picking your battles. But
this is such a great set of guidelines for where to start that conversation.

(Edits for typos only)

~~~
ThatMightBePaul
Thanks for the good word (I'm the author). I 100% agree about setting the
right culture / the difficulty of managing the transition.

It takes a ton of work to establish the right infrastructure + culture.
Specifically for the reason you mentioned: a lot of folks see QA in black and
white.

It's doubly hard when you're growing fast.

------
joshdotsmith
Wow, crazy timing. We interviewed Fred at Rainforest on the Talking Code
podcast – and that interview launches tonight at midnight (PDT).

I'm going back through and picking out my favorite quotes right this moment in
preparation for the episode launch. Some of my favorites:

"How did we get to the place where the generally accepted best practice is not
very good?" (referring to automated testing)

"The visual representation is only somewhat semi linked to the actual
underlying code that represents it."

"A really good QA person can actually start to QA the requirements
themselves."

"Good QA people are the customer before the customer."

"Shipping a bug at Facebook scale is pretty catastrophic."

"For tiny company X, a YC company, shipping a bug – while painful – is not
going to be the end of the world."

"For 95% of people, QA is not a competitive advantage, so it doesn’t make
sense to have it in house."

This has honestly been one of my favorite interviews ever, and I'm just really
lucky it happened on my own podcast.

[http://talkingcode.com/podcast/episode-6-fred-stevens-
smith/](http://talkingcode.com/podcast/episode-6-fred-stevens-smith/)

~~~
MichaelCrawford
"QA the requirements".

Safety-critical software does not commonly has bugs. The problem it faces, is
incorrect requirements.

------
ecopoesis
Zero QA, or at least zero QA by dedicated QA people.

Developers end up using QA as a crutch. The end up not testing their code
completely, and sometimes they don't test it at all. Too many times I've seen
code checked in, after going through code review, that just doesn't work. I'm
not talking edge cases, it does not function at all.

Get rid of the crutch, and force developers to be responsible for their code.
Developers should do all the QA on their code.

If you must have QA, don't waste them on checking if features are I plmented
correctly, repeating a job developers should do. Instead, have them actually
check quality: have them look at the whole app or site holistically. Have them
try to find the bugs that developers won't be able to find.

------
EliRivers
This article, as is typical around here, addresses a very small part of the
industry. Here's a view from a different part; the part where a project has a
lifecycle of a decade or so.

I used to work for a software consultancy that had been acquired by a US
defence company (second tier). Over the previous 15 years, the software
consultancy had elevated their QA to the point that bugs in the delivered
product were literally unknown. Nobody ever saw any in the delivered product,
over a decade. Every requirement could be traced to the design that would meet
it, to the code that implemented it, to the test that verified it. The test
documentation was beautiful, and anyone could (and did) execute a complete set
of tests, writing their own personal signature on every confirmation of test
success. The signed test specs were scanned before being sealed in an envelope
against the customer ever choosing to inspect. QA had (and did) veto over
requirements, design, and tests.

We were working with two other major, globally known defence names, and the
customer had learned through experience that when something went wrong, come
to us last. Sometimes, the customer came to us to ask us to work out which of
the other companies involved had screwed up (because sometimes, the other
companies were themselves unable to work it out). Once, they sued one of the
other names for incompetency and gave us their work because we knew how to fix
it better than they did, because our interface tests highlighted everything
that their software didn't do in accordance with the spec.

We could replay any test, at any version, at any time, and compare the results
now with the signed results then from the sealed envelope. The time savings we
had because we got things right the first time, by doing it properly, was
immense.

Where this is going is that this was priceless. We had the customer's absolute
trust, based on experience. Nobody ever found a bug in our delivered software
(I'm not saying there weren't any; I'm saying that our QA was thorough enough
that the customer was never going to do anything with it that we hadn't tested
already). Code coverage, all warnings on, static analysis, valgrind, cross-
platform compilation and testing, we did everything. Customers outright told
us that we were worth paying extra because of the peace of mind and the
reduced risk, so they paid us more and we spent less doing it.

QA is time and money. The better your QA, the more time you will have and the
more money you will make.

~~~
geoelectric
So, this I agree with too, in contrast with my other comment. There are
particular industries (engineering, banking, aerospace, defense, for some
examples) where it -has- to be correct. These are also the industries where at
least the final product should probably be waterfall or spiral, and generally
you need formalism all the way down.

The problem comes when people bring lessons learned from that to companies
that don't need that level of assurance. Documentation that's never read again
is a waste. Spending time to make tests replayable if they'll never be
replayed again is a waste. And so forth.

As for bugs being literally unknown, I'm jealous--it'd be nice to claim I ever
guided a project to that state. But keep in mind that's different than no bugs
period, just means no bugs in what was tested or used.

If you're delivering a product with very specific use cases, flows and a tight
scope with strong bumpers around anything unintended, it's pretty
straightforward to guarantee all that. But if you're delivering a sandbox
product on the internet to a bunch of randoms, much less so.

After all, ten checkboxes on a dialog are 1024 testcases. That's not even
accounting for free-entry fields. You'll never cover all combinations of all
uses of the whole app interface, ever, period, and equivalences and fuzz
testing only get you so far. That's a simple (if slightly fantastic) example
for UI testing, but the same basic thing applies to most other kinds of
testing. At the end of the day you often have to pick your battles.

So I really disagree on the better is more. There is absolutely a diminishing
return. It just so happens that for what -you- were doing, that point of
diminishing return is pretty high.

~~~
enraged_camel
>>The problem comes when people bring lessons learned from that to companies
that don't need that level of assurance. Documentation that's never read again
is a waste. Spending time to make tests replayable if they'll never be
replayed again is a waste. And so forth.

To me, writing thorough documentation and solid tests from the beginning
serves a very important purpose that is often overlooked: it establishes the
right type of culture. I think of it as laying the foundation of a building.
If you skimp out on materials, the whole thing will be unstable and prone to
collapse. And going back later to fix it is often times a lot more costly than
doing it right in the first place.

You would never make the argument that a solid foundation is unnecessary if an
earthquake never hits, right?

~~~
geoelectric
After doing quality for 15 years after another 5 as an application developer,
I'm prepared to say that it's an awfully expensive and ineffective way to try
to set tone. All it mostly does it get the rest of the company to assume
you've got quality all handled and start chucking stuff over the wall at you.
It doesn't inspire them to a higher standard.

If QA groups spent half as much time actually testing or helping stabilize
development processes as they did maintaining brittle docs they'd probably be
considerably more effective. At the end of the day most verbose test docs
should have been one set of canonical product docs and a list of
inputs/strategies. Instead we repeat and fragment the same info across a bunch
of disconnected test docs and then let them rot under the weight of
maintenance.

The problem is everyone thinks they have a "one true way." We know that
concept is bullshit in software dev and that approach should be tailored to
the problem at hand. We understand how important YAGNI and SPOT is, and why
sometimes insisting on a perfect architecture is a bad thing in the real
world.

Why people think it's any different for QA is beyond me. The same principles
largely apply.

------
codelion
I am not sure you want to delay penetration testing until you are post IPO.

~~~
ThatMightBePaul
Agreed.

Caveat: do you need to do pen testing _every_ release? I'm guessing, not.
YMMV.

If you're a defense contractor (as some other commenters mentioned), your
priorities are probably quite different.

~~~
infosectosser
I head infosec for a “Series A - C” B2B company and a fairly standard request
from a potential customer is to see not only our own penetration test reports
but third party penetration test results, as well. As a result, we run
automated pen tests on weekends and before major releases. We also work with
an application security firm every 6-12 months. For what it’s worth, we don’t
do anything nearly as intense as defense contracting or handling financial
info.

That said, I liked the article - thanks for sharing.

~~~
ThatMightBePaul
Np! Thanks for sharing.

Glad to hear security is taking the front seat some places. Anecdotes like
this help me expand my world view. <3

------
mistercheese
thoughtful article

------
MichaelCrawford
As a consultant, I used to ask my clients to QA my deliveries. Few of them
were willing to, to the extent they did they could not or would not provide
meaningful bug reports.

This led to my adoption of test-driven development - mostly but not all
automated testing - with the eventual result that I advertised that my final
deliverables would be ready to ship to end-users, without any requirement for
QA by my clients.

~~~
ThatMightBePaul
TDD is great. It should be a given on any project that isn't just a spike.

I'm curious what kind of products you were building? Simple websites, or full
fledged applications?

100% unit test coverage can still miss a host of gnarly bugs. A lot of the
ugliness of QA comes from how A interacts with B. Or how A + B + C + D work
together. Or at least, that's my 2 cents :D

~~~
geoelectric
You also have to qualify 100% coverage: SUT lines, branches, or paths? Those
are all very different levels of rigor.

The two most useful code coverage stats I've found are 0 and "less than
yesterday." The former tells me something pretty important about the culture
of the team who owns that code; and having rules against the latter is about
the only way to ensure a test-on-checkin policy.

~~~
ThatMightBePaul
I love this metric, and am totally stealing it.

~~~
geoelectric
All yours!

