
Ask HN: Small sample size user testing fallacy - homami
I had a debate with a colleague of mine about testing some Web UI code.<p>My friend&#x27;s argument goes like this: If we cannot test our code with big enough representative sample of Android phones, then let&#x27;s just release the code and collect usage statistics from real users without user testing it at all.<p>Basically since testing our code with few phones does not prove that our code will work on 99% of the devices so it is as good as not testing it at all.<p>I think there must be a logical fallacy in this argument, what is it?<p>I can think of a counter argument that goes: let&#x27;s assume we get 99% confidence that our code runs on every phone by testing it with 30 devices:<p>(1&#x2F;x)^30 = 1 - 99% then x = 1.166<p>confidence(N) = 1 - (1&#x2F;1.166)^N<p>Now with N = 5, I am already more than 50% confident that the code works on every device. But then the question is where does my assumption of confidence(30) = 99% come from in the first place.
======
imauld
Depends on the range of Android versions you support and what your app does.
I'm not an Android dev but I'm fairly certain if you all you want to do is
display some text it'll probably work fine across most versions of Android. If
you are using some bleeding edge API you may have some issues.

The "lets just release it" strategy isn't terrible. It's a trade off between
your dev time of finding bugs pre-release and having users find them and what
the impact is. If you take money you should make sure that process work on any
version of your app that supports it. If the payment screen has some weird
visual bug but functionally it works fine then it's probably okay for users to
find.

Cover your core cases on all versions you support, things outside of that can
probably be passed on to users safely.

------
itamarst
If your code fails on 100% of devices, testing with 1 device will catch it.

If your code fails with 99% of devices, testing with 5 different devices will
almost certainly catch it.

Etc.

Which is to say, you won't catch obscure bugs, but you'll catch wide-spread
bugs. And wide-spread bugs are exactly the ones you want to catch before
general release.

------
rajacombinator
Depends what kind of testing you’re talking about. Tech / does it work testing
should be pretty easy to accomplish regardless of users/phones available. UX
testing is a lot harder and probably better to do a combo of “just release”
with recruiting testers.

