
The Kolmogorov-Smirnov Test - steveklabnik
http://daithiocrualaoich.github.io/kolmogorov_smirnov/
======
btn
This is a very nice review, but in practice I've found the K-S test to be much
less useful than it initially appears:

1\. Failing to reject the null hypothesis is not the same as accepting the
null hypothesis. That is, concluding "these data _are_ from some distribution
X" is spurious.

2\. There's a 'sweet-spot' for the amount of data. If you have too few
samples, it's very easy to fail to reject; and if you have too many, it's very
easy to reject (the chart at the bottom of the "Two Sample Test" section
illustrates this).

3\. The question "are these data from some distribution X?" is usually too
strong. It's usually more informative to ask "can these data be modelled with
some distribution X?"

~~~
murbard2
Agree with you on all three, but specifically for 1., can you think of
pathological pairs of distinct distribution that the test would often fail to
reject?

~~~
tgb
The article says it's poor at detecting differences in the tails and much
better at differences in the medians. So that's where I'd start to find
problems.

~~~
murbard2
Playing with the tails make all kind of mistakes possible, but that seems like
a criticism that would apply to any attempt to identify a distribution based
on sample.

------
losvedir
Nice review! I tried using the K-S test once for some of the old Matasano
crypto challenges, to determine if the letter frequency of the text after
running it through some deciphering algorithm was from the same distribution
as the letter frequency of a sample of the English language. Couldn't ever get
it to work, though... maybe that's an inappropriate use of the test, or maybe
my sample (Pride and Prejudice, IIRC) was unrepresentative. In the end, simply
computing the distance of the two letter frequency vectors (sum of squares)
worked.

I did it all in ruby at the time, but it looks like rust may have some stats
libraries now? Should give it a whirl again this time using rust, or maybe do
tpatcek's new Stockfighter game instead.

~~~
capnrefsmmat
The K-S test assumes the data comes from a continuous distribution, so count
data would mess up the test's false positive rate. You could avoid that by
doing a permutation test, or one of the variations of the K-S test designed to
account for data with ties.

------
regularfry
What a wonderful article. I've learnt several useful things just by skimming
it, even if I never end up using Kolmogorov-Smirnov in anger.

~~~
golergka
Why anger?

~~~
pliny
Used in anger is an expression that roughly means 'used to achieve something.'

I think it comes from 'shots fired in practice' vs. 'shots fired in anger'
(i.e. in combat, for the purpose of killing).

------
princeb
the KS test is quite sensitive.

if you are specifically testing against the normal dist the jarque bera test
might be better, although also rather sensitive (prone to false negatives).

for two samples if you have enough data to bin, the chisq test is also
available to you.

------
navi54
This is quite nice! But does anyone else knows more useful websites such as
this with down-to-earth explanations of statistical tools?

~~~
graycat
That the K-S test works is some cute math by the _father of modern
probability_ A. N. Kolmogorov.

But that test and many more are part of _non-parametric_ , that is,
_distribution-free_ hypothesis testing. That statistics has long been popular
in the social sciences. A major theme in such statistics is permutations.
Another major theme, and more recent, is _resampling_.

I first learned about such tests from a book that was sitting around the
office, Sidney Siegel, _Nonparametric Statistics for the Behavioral Sciences._

These tests are all _one dimensional_. Once I published a paper on a
distribution-free test that is multi-dimensional -- my paper may remain the
only such.

These days, see also the work of B. Efron and P. Diaconis.

More can be done.

------
irremediable
I've used the KS test (and several other nonparametric stats) in research a
bit. It's a cool thing to learn about, though it's worth noting that a _lot_
of applied research can make do with parametric approaches. Thank you, Central
Limit Theorem.

------
fizixer
I've heard Bayesian data analysis is the most powerful compared to all
statistical tests/techniques. Can anyone clarify/expand?

