

The Importance of Sequential Testing - MidsizeBlowfish
http://www.austinrochford.com/posts/2014-01-01-intro-to-sequential-testing.html

======
noelwelsh
[I'm assuming the author posted this. MidsizeBlowfish has no profile
information.]

Good stuff, and just one of the many reasons that most A/B testing practice is
flawed. I haven't seen the SPRT used much in practice. I'm not sure why. A
quick Googling indicates non-parametric sequential tests have been developed.
What about the case with more than two alternatives?

My field is bandit algorithms, which are a generalisation of hypothesis tests.
Here there are algorithms like Empirical Bernstein Stopping
([http://icml2008.cs.helsinki.fi/papers/523.pdf](http://icml2008.cs.helsinki.fi/papers/523.pdf))
which achieve the same early stopping from a different approach. If this is of
interest sign up to the course I'm running
([http://noelwelsh.com/data/2013/08/16/free-bandit-course-
draf...](http://noelwelsh.com/data/2013/08/16/free-bandit-course-draft-
outline/)) -- I plan to cover this algorithm soon.

BTW, you have a very formal writing style. I think if you're trying for a more
general audience you should loosen up a bit. This ain't a journal paper. :-)

~~~
MidsizeBlowfish
Haha, yes, I am the author. I do have a very formal style, an unfortunate
product of too many years in grad school.

The SPRT is a nice toy test, but only useful for point hypotheses. There are
generalizations to more realistic composite hypotheses, though.

~~~
noelwelsh
It would be interesting to me if you covered the generalizations. This field
seems to have evolved entirely separately from the banditry I'm accustomed to.

~~~
MidsizeBlowfish
I'll put a follow up on some other sequential tests into my list of blog-
posts-to-do and let you know when it happens.

------
nitrogen
A heads up to the author: with narrower browser window sizes (e.g. less than
~1024px), the left side menu overlaps the text.

~~~
MidsizeBlowfish
Just noticed that on my iPad, will look into it.

