
Want Better Quality? Fire Your QA Team - DanielRibeiro
http://blogs.forrester.com/mike_gualtieri/11-02-17-want_better_quality_fire_your_qa_team
======
bmelton
My wife has a freelance QA consultancy, and while I understand that there is
often the sense that QA is a 'nag', I can honestly say that whenever she is
involved in my work, the quality of my output is much better, and takes less
time to get it there.

A good QA analyst / team does much more than point at a problem and say 'there
it is'. A good QA analyst can determine why you made an error, and can
anticipate other errors you might have made in the same way. A good QA analyst
will learn your formulae, learn your equations, understand your workflow and
try to find root cause, even if they're not developers.

Having a good QA person with a trained eye for knowing what I'm likely to have
overlooked is also FAR more cost effective than your high-paid developers
doing the same task. If they know that 'heads will roll' when they get errors,
I'm also betting that makes your releases even slower.

Off-topic completely, but I generally prefer non-developer QA analysts, as
they generally prove themselves to 'think too much like me' to be effective.

~~~
DanielRibeiro
People who know how to test are extremely valuable. Specially when doing peer
review/pair programming with (solving the blind's eye thing)

Testing as a whole is not an easy task as it involves testing some really hard
things: memory leaks, performance, UX, correctness of parallel and distributed
algorithms/programs/systems.

However, Wealthfront engineers probably put this position more into
perspective [1]:

 _We do not have QA team and do not want to have one, the reasoning is that if
a human is involved in testing then there is a higher chance of missing things
and you simply can't test all the site dozens of times a day._

Granted, Wealthfront uses continuous deployment[2], while still being
regulated by the SEC. I really enjoy their lean startup approach to minimize
inventory (ie: code not in production), while still keeping an immune system.
And investors seem to agree it is really paying off [3].

[1] [http://eng.wealthfront.com/2010/04/findbugs-husdon-and-
pizza...](http://eng.wealthfront.com/2010/04/findbugs-husdon-and-pizza-
driven.html)

[2] [http://www.eishay.com/2010/07/continuous-deployment-at-
kachi...](http://www.eishay.com/2010/07/continuous-deployment-at-kaching.html)

[3] [http://techcrunch.com/2010/10/19/kaching-gets-100m-under-
man...](http://techcrunch.com/2010/10/19/kaching-gets-100m-under-management-
and-gets-all-wall-street-with-a-new-name/)

~~~
bmelton
At least in my wife's case, she generally does pre-release QA, and doesn't
necessarily do continuous QA. Still though, even if you need support for
continuous deployment, you probably want a QA person to set up your automated
test cases to test the output of a site in a way that, in my opinion, unit
tests don't seem to catch.

~~~
arosien
Having worked in Silicon Valley for 12 years, I have yet to find QA people who
could successfully automate testing in a non-fragile way or without using an
expensive, proprietary and crappy testing harness. Those who know how to
program are invariably better at writing automated tests. One recent CS
graduate should be able to put a team of QA out of a job. (Disclosure: I work
for Wealthfront)

------
prodigal_erik
> This app can't go down, and it can't be wrong.

Shops who set the bar this high _and mean it_ are exceptionally rare (I look
for them when I'm on the market). Where I am now, an hour of downtime costs us
my salary for weeks, and still we cut corners that don't always sit right with
me.

But having generalist engineers create functional and integration test plans?
Not only do we tend to cost more, but we have less experience at it than QA
specialists. And a dev shouldn't test their own work, because they won't
notice any unwarranted assumptions they made in design and coding.

------
PaulHoule
ridiculous.

a large fraction of errors that turn up in QA are conceptual errors made by
the development team; like the blind spot in the eye, you don't know what you
don't know, and there are certain problems that you'll never find yourself.

Phil Crosby would say that management is responsible for quality; management
writes the specifications and has the responsibility to hire and fire
developers and QA people. The buck stops with management.

Efficient quality requires practices to be work correctly across the
organization, but it starts in the specification process. If you don't know
where you're going, you're sure as hell never going to get there.

~~~
petervandijck
"If you don't know where you're going, you're sure as hell never going to get
there." -> Also not always true. Think Twitter. Think Christopher Columbus.

~~~
msbarnett
Columbus knew where he was going. And would have died trying to get there, if
he hadn't serendipitously run into something else, first.

~~~
l0nwlf
But he misjudged that something else (America) with his original destination
(India).

Quoting from wiki (<http://en.wikipedia.org/wiki/Christopher_Columbus>):

"Amerigo Vespucci's travel journals, published 1502-4, convinced Martin
Waldseemüller that the discovered place was not India, as Columbus always
believed, but a new continent"

------
daxelrod
If a developer's reaction to a client-facing defect is "QA didn't catch it",
you have cultural problems that firing a QA team probably won't fix. If you
don't have a culture of personal responsibility, a finger can always be
pointed elsewhere.

Developers should absolutely write and use unit tests. Where QA really shines
in the larger scale: validating actual business requirements.

~~~
awj
Exactly. For me, every client-facing defect is an embarrassing personal
failure. Having them caught in QA simply lessens the shame, and allows me to
dedicate all the time I would spend freaking out about the newfound emergency
to figuring out improvements that eliminate that entire class of errors.

------
chrisbuchino
I 100% agree with this article. Quality is everyone's job, not just the QA
teams' and I believe that, in most cases, developers can and should be
responsible for developing, testing, and releasing their code. When developers
can't "throw code over the wall to QA" because there is no QA, they will do a
better job testing it themselves.

I don't buy the popular opinion that developers can't test their own code like
there is some sort of magic that goes on in the mind of QA engineers that
causes them to be able to find bugs that developers couldn't find themselves.
In fact, I think the opposite is true - developers are better suited to test
the features they are writing because they know the code and know the risk
areas, where things need to be regression tested, where more attention should
be placed, etc.

I do think that it can be helpful for another set of eyes to look at a
developed feature and this is where pair programming can be useful. It does
not need to be QA doing this.

One caveat is that this approach does not work for all teams. This works for
small, agile teams where roles and responsibilities are less defined. If you
fire your QA team but your developers can't test their own code because they
are set in their ways of writing crap because they expect QA to find the bugs,
you may end up firing your dev team too....

~~~
datasink
Having been in a high pressure position in which my team and I were writing
mission critical code which could "have no bugs whatsoever" first without QA,
and then later with QA, I can say confidently the quality of the code and
occurrence of bugs dropped dramatically after we introduced our QA engineers.

Our prior condition was very similar to telling a newspaper writer "DONT WRITE
TYPOS". You can scan a page you've written several times carefully and still
miss your mistakes, even if you are a conscientious writer. And the newspaper
industry isn't the only industry that prevents mistakes through multi-party
review processes, either.

------
kenjackson
QA and dev are two different skills. It's like saying that we save on design
by having the devs do it, or having your quarterback also play noseguard. Sure
it works in high school, but not when you're a pro.

I'd much rather hire someone who knows testing and QA well and let my devs
focus on writing bug-free product code.

~~~
arosien
I disagree. Developers who don't know how to test are bad developers, they
should go. And conversely, if you have QA, they should know how to develop,
otherwise you're paying them to hunt, peck, and click. Why not have QA write
the code to have computer do that and save like 1000x the time? That's
efficiency AND quality.

~~~
kenjackson
Of course QA knows how to program. But their skillset is about breaking code.
It's about writing this crazy C++ function that shows how your API has a
security bug in this case. As developers their like dev tools developers as
their customers are generally devs.

Skillset maybe isn't the right word... mindset maybe better word. Both should
be good programmers -- in fact you'll often have more test code than product
code. But you likely won't be able to switch your dev team to QA and vice-
versa and get the same results.

~~~
arosien
You're lucky to have QA people who know C++, I've never met one. Maybe I've
been in too many web-oriented startups.

------
rst
By the time anything gets to the QA person at my dayjob, the devs have usually
tried every damn thing that they could think of to break their own code.

The usual reaction to the bugs she finds is "what made you think of trying
_that_?"

------
pnathan
I work in QA in a high-reliability industry. Correct operation of some our
products saves millions and saves lives. We need _super_ high quality. How we
do QA is pretty traditional. In particular, we have functional test teams. The
perspective we take is that the testers take the specs and develop tests to
exercise the functionality, and theoretically, we (QA) cover any spots the
developers missed, as well as providing assurance that the developers aren't
talking out of the wrong hole when they say they're done. Seems to work most
of the time except when mutual brain-farts develop.

I don't know that I would go so far as to recommend firing myself to increase
quality. I would have to visit with many teams and really gain experience in
the way they work. I do think that we could take a step forward by having a
standard across the company to have fully automated test suites running
continuous builds and tests. That, however, runs facefirst into the problem of
Legacy Stuff and ROI. Automation just plain takes time when you are
interfacing against external programs. It's not always worth it in the
short/medium term.

There's no silver bullet to the software quality problem. But me, I think that
continuous integration provides a copper bullet.

------
vacri
This works as long as your developers are talented in QA and are motivated to
do it. That sounds like it requires a lucky constellation of events.

------
ibejoeb
Don't be so quick to brush this off. The argument isn't so different from
saying that we don't want to distinguish "front-end" and "back-end"
programmers. It's probably a special circumstance that you need a team that is
_entirely_ dedicated to verification.

Feynman talks about this arrangement in his essay on the shuttle program
(<http://www.fotuva.org/feynman/challenger-appendix.html>). In a nutshell, the
core software team is responsible for checking the code. Then there is a
totally removed group that is adversarial, i.e., it is measured on its ability
to trip up the software team. This is the same a "setting consequences" as
this author describes.

Now, the shuttle program is truly life critical. I think a good software team
can produce quality results internally.

~~~
joe_the_user
_The argument isn't so different from saying that we don't want to distinguish
"front-end" and "back-end" programmers._

I don't want to distinguish front-end and back-end programmers. I would like
to distinguish front and back-end _development_.

I do everything currently but I still need put on a conceptual "testing hat",
"GUI hat" and "back-end hat". That requires mental effort and costs time. A
real Q&A would save that even if I was exactly as skilled as the Q&A person.

~~~
ibejoeb
Yeah, that's true. I think the point of the source is that the first line of
defense is with the originator of the code. Sure, the context switch is
expensive, but that's the price of higher quality.

It's not really about firing the QA team. Instead, it's about having a QA
function that does not entirely relieve the original author of his duty to
ensure quality.

------
bugsy
Ah it's a single anecdotal story and comes with no specifics about who it is
or in what way things supposedly improved, just some broad calls for firing
QA.

~~~
jacques_chester
Forrester will ring a few more people they picked out of the phone at random,
write it up as a report for their library complete with beautiful graphs, and
then begin to charge thousands of dollars per copy to tell other consultants
what the "Industry Best Practice" is at the moment.

------
techiferous
> They better get it right, or heads will roll. As British author Samuel
> Johnson famously put it, "The prospect of being hanged focuses the mind
> wonderfully."

Gee, sounds like a _wonderful_ place to work. When can I sign up?

------
varaon
My head is more often buried in code than in the product's UI. Our QA team has
a better nose for when interaction or UI details are off than I do. In
general, they spend more time _using_ the application, and have a better
instinct for when a given behaviour is wrong.

Some of the bugs they report would fall under the automatically testable
category, but many don't.

I suspect that dogfooding is more effective at companies where the product
itself is useful to developers or is a consumer app. e.g. Basecamp, FogBugz,
Facebook, Netflix, Gmail, etc.

------
jacques_chester
Ah yes, Forrester.

I hope all of you reading the blog post realise that this brainfart will cost
your organisation $5,500 per copy downloaded.

------
lhnz
I don't think this would be a good idea. When developing I don't always have
the time to be as thorough as a QA team is in testing my own code. It's very
helpful to have somebody else with a non-technical background to help spot
problems that I am blind to.

~~~
petervandijck
That you can afford to be blind to because you have someone doing QA?

~~~
awj
No, that you're blind to because it's in the nature of software development.
Spotting flaws in something you've spent the last $X hours _convincing_
yourself is correct is hard to do. It's easy to make invalid assumptions and
internalize them in your thinking to the point you don't notice them. This
problem exists, and largely is solved by dragging in _someone else_ to help
with it. That someone else, unless you'd like to squabble over semantics, is
doing QA.

------
dgabriel
Considering I work with one of the most incompetent QA engineers I've ever had
the displeasure of encountering, I couldn't agree more. This person wastes my
time, and the organization's time, and has not actually increased the quality
of software (as far as I can tell). It is a waste of a developer's time to
decipher a bug that states "it's broken," without clearly defining "broken,"
or "it," or how to reproduce the behavior. Often, reproduction involves user
error (like forgetting to sign into the vpn). If it weren't for this bozo, our
team could get more work done in less time.

I assume that others have had better experiences.

~~~
zeemonkee
So because one QA guy is bad, the whole profession is bad ? You could say that
about any profession, including developers.

~~~
dgabriel
Bad QA is worse than no QA. In my 15 year career, I've seen two decent QA
analysts, and they both had extensive technical backgrounds.

------
sh1mmer
Peopleware has a chapter about "the black team" that was at IBM. The chapter
is about team work but they describe how in the process of gelling the QA team
delighted in finding programmer bugs. They describe how creation of a
reputation and ethos in the black team not only made them a little better than
other QA teams but significantly better.

While it's easy to say that QA isn't important many of the QA teams I've
worked with weren't respected or valued, and often saw QA as a stepping stone.
If you want to increase the quality of the code, don't fire the QA team,
motivate them.

------
jamesgeck0
What is the metric for quality? Is the article saying that having fewer people
looking for bugs resulted in more bugs being found? Or fewer bugs? Or just
fewer bugs found?

------
mak120
Most of the other comments already voice my reaction to this post. I will only
add that a "heads will roll" attitude towards motivating devs on the part of
management would prompt me to switch to some other place very fast. Simply
saying "you have to do X as well as Y or heads will roll" just sounds like
trying to save money on QA and squeezing more work out of devs - who could be
putting that time and effort to much better use improving the product.

