
Ask HN: startups and A/B testing - ecesena
tl;dr: do you have any example of scientifically sound A&#x2F;B test that works for an early stage startup? Although I&#x27;m a fan of A&#x2F;B testing, I&#x27;ve found myself confused&#x2F;concerned about timings.<p>I was reading Google analytics docs on multi-armed bandit experiments for A&#x2F;B testing.
https:&#x2F;&#x2F;support.google.com&#x2F;analytics&#x2F;answer&#x2F;2844870?hl=en&amp;ref_topic=2844866<p>In summary, assuming 100 visits&#x2F;day and 4% conversion rate (CvR) with page A, a scientifically significant experiment for page B to say that the new version actually generates CvR of 5% will require 66 days with the multi-armed approach, or 223 days with the naif 50%-50% approach.<p>To me, even 66 days seem an enormous amount of time for an early stage startup (say, 1k-to-100k users, 10% active users, growing at 5%&#x2F;week), where decisions need to be made in a few days. Assuming you can pay for ads, you can probably increase the visits&#x2F;day, but then the CvR drops down. 
So, unless you&#x27;re testing a really bad A page vs a really good B page, I can&#x27;t figure out an example where the time required is acceptable (say, max 1 week).<p>I was wonder if anyone has real numbers that also satisfy a more theoretical appetite ;)
======
ig1
If you're a typical early stage startup with 100 visits a day then you
shouldn't be caring about the difference between 4% and 5% - it's not big
enough to matter.

At that stage you should be testing big changes not little ones and increasing
traffic.

~~~
ecesena
I agree on focusing on big changes, but these hardly need to be _A /B_ tested.
A/B should be more for optimization.

So this seems to confirm my idea that (statistically sound) A/B testing is not
for startups...

~~~
ig1
You should definitely be A/B testing big changes on things like landing pages.

------
karolisd
Measure first. Setup analytics correctly so as you get traffic, you can
analyze your funnel and decide where to focus on testing.

Multi-armed bandit testing is cool, but at this point is it worth consuming
brain cycles on that vs getting more traffic?

~~~
ecesena
I totally agree with you guys, traffic first.

But if you look around everybody is just talking about A/B testing. My point
is that that doesn't look to me statistically meaningful... it's kind like
you're taking the A/B testing approach, but applying it the wrong way...

What I was asking for was a real example of A/B testing for a startup, but
you're basically confirming me that this doesn't exist. :)

~~~
karolisd
How many steps are there in your sign-up process? What's the progression rate
for each step? What area needs the biggest improvement? Focus on that.

