
Nelson Rules - misterdata
https://en.wikipedia.org/wiki/Nelson_rules
======
repsilat
Out of curiosity, with a regular normal distribution I wonder what the
probability is that the most recent point finds a problem. I guess you could
calculate these things separately for an approximation, but I'd probably just
want to simulate it...

For a first cut:

\- Rule 1: 0.3% of samples are more than 3 standard deviations of the mean.

\- Rule 2: 1/2^8 = 0.4% chance the previous 8 points were on the same side of
the mean as the most recent one.

\- Rule 5: 2.5% chance of being above 2sd on either side, 3 choose 2 is 3, 2
sides, so 0.375% of exactly 2. "2 or 3" is not much higher.

\- Rule 6: More than 0.55%, if I've done my maths right.

\- Rule 7: 0.3%

I guess you're going to get a lot of false positives if you're sampling
reasonably frequently -- maybe one in 50?

~~~
Bartweiss
Thanks for this. Rule 5 immediately stood out to me as a high error source,
and I came to the comments hoping someone had done the actual math.

My guess is that positives from these rules would be logged rather than
immediately reacted to. If over 1000 detections you break each rule at about
the expected rate, all is well. If you break a few rules well outside of
expectation, it raises some questions.

~~~
hebdo
By error source you mean false positive source? Because according to the
analysis doc the rules 1 and 6 have way higher false positive rates.

------
jameshart
Harry Nyquist would probably have something to say about the validity of rule
four. Fourteen points in a row of alternating increasing and decreasing isn't
just indicative of an oscillation, but an oscillation that's close enough to
your measurement frequency that you aren't actually able to measure it
accurately. It could be a real oscillation, or it could be an artifact of much
higher frequency behavior.

~~~
gilgoomesh
I think Rule 4 is valid because it is intended to catch both situations you
describe: oscillations at approximately the Nyquist frequency and also much
higher frequency behavior (Rule 8 is intended to catch oscillations below the
Nyquist frequency).

In any of these cases, the Nelson Rules state that the signal is unlikely to
be a stochastic random variable around the given mean. It is instead likely to
have some underlying shape that is not being described.

~~~
jameshart
The thing is, something going on above the Nyquist frequency could create ANY
of the Nelson rule patterns - or none of them. Expecting it to create a nice
alternating up/down pattern is wishful thinking.

~~~
euyyn
Yeah but the rule isn't "if this doesn't happen everything is fine", rather
"if this happens, something's probably wrong".

------
jacquesm
Related:
[https://en.wikipedia.org/wiki/Moscow_Rules](https://en.wikipedia.org/wiki/Moscow_Rules)

Which includes my favorite: "Once is an accident. Twice is coincidence. Three
times is an enemy action."

~~~
creshal
> 3\. Everyone is potentially under opposition control.

Even the author of the rules? Hmm…

~~~
pjlegato
"Being under opposition control" does not necessarily imply "is not telling
the truth."

------
learnstats2
Thanks for this reference!

GCSE Statistics (UK school exams at 16 years old) teach a simpler system of
process control rules, closer to Western Electric
[https://en.wikipedia.org/wiki/Western_Electric_rules](https://en.wikipedia.org/wiki/Western_Electric_rules):
and that is the only place I have ever come across them.

Is this in current practical use?

------
calinet6
Oh yes. Yes yes. Learn this and understand how it applies to your systems,
your processes, and especially (surprise) your people.

This is one quarter of how W. Edwards Deming promoted organizational quality
control—understanding how variation works, period. (The other three being
understanding psychology, understanding systems, and understanding the theory
of knowledge or scientific method).

This applies directly to understanding whether observed variation has a common
cause (is a natural pattern of the system), or is special cause (something
unexpected):
[https://en.wikipedia.org/wiki/Common_cause_and_special_cause...](https://en.wikipedia.org/wiki/Common_cause_and_special_cause_\(statistics\))
and this impacts how you handle the variation.

For those criticizing validity, I'll say this is a way to mentally model how
to understand variation, and is not meant to be 100% accurate. You're trading
intuitive modeling for perfect math. But it will allow you to get close in a
back-of-the-napkin quick way so you can identify patterns to study in more
depth. Also, think of this in the context of many types of systems, not just a
tight electrical signal pattern (which are easy systems to understand).
Systems of people doing software development, machines in manufacturing
processes, complex network error patterns, etc etc.

People don't often have a good idea of what's important and what's noise,
especially when you don't even have a control chart but are just using
intuition and a few data points. We see outliers and variations all the time
in processes, especially in human processes like those we encounter in most
software companies. Estimation and delays, developer performance, load
failures; all kinds of complex systems that exhibit variation that people are
usually "winging it" to understand.

Instead of understanding the variation and the data, people often handle every
large variation in the same way, trying to "fix" it or peg it on some obvious
correlation they think they observe. This says: hold on, understand what
you're looking at intuitively first. Then gather more data. Don't act without
understanding. Deming was fond of saying, "don't just do something, stand
there!" Lots to be learned from that, and much to be gained from the simple
intuitive understanding of patterns in variation.

------
IshKebab
This seems terribly fragile and ad-hoc. It doesn't even take into account
sampling rate and it clearly depends on it.

I guarantee there are better methods.

~~~
vajrabum
That's exactly right. They are heuristic. Control charts were designed to be
used on a manufacturing floor by workers who updated their charts manually on
paper and called an engineer to aid in further investigation when a
manufacturing process might to be out of control. Making out of spec parts or
having an expensive machine break because it's operated outside it's limits
cost real $$ so some false positives on a low frequency signal can be worth
it. Also manufacturing engineers or QC would be responsible for the frequency
of testing and sampling procedures so it's unlikely to be as naive as you
might assume from the outside.

------
thanatropism
Since I was preempted for the comment "this is typical quality-control ad-
hockery BS", I'll play devil's advocate and argue that the point of quality
control is to identify two components of a mixture distribution[0]: a bounded
distribution of uncertainty, which can be modeled as a beta ("PERT", in some
patois), and an unbounded, "error" term that functions more like a Poisson or
even a Pareto.

This is already an adhockish simplification of something like Mandelbrot's
seven regimes of randomness [1], which is itself, well, an oversimplification
of his own work. But it formalizes the insight that quality-control is trying
to impart -- the identification of inconsistencies among consistent variation.

So let's run some simulations in Matlab. We'll generate M numbers distributed
like Beta and N like a Pareto (a "long tail", "black swan" distribution) with
identical mean and standard deviation, and shuffle them before we interpret
them as a time-series. Then we'll check Nelson's rules. Since we know how many
ordinaries there are, we have a target.

In 10^4 repeated simulations each with samples of 180 ordinary betas and 20
Paretos, we expect to identify 10% of abnormals. Now, my samples are shuffled,
and Nelson rules rely on time-structure (but this is precisely their weak
spot); my code [2] also has visible bugs I didn't bother to fix because they'd
involve thinking too hard and didn't seem so serious in large samples. Still,
here are identification rates:

\- Rule 1: 0.5%. This will be counter-intuitive to students of the normal
distribution, but recall that the ordinary observations have a bounded
distribution and we're really catching only the abnormals. Now, we've missed a
lot of the 10% by this rule.

\- Rule 2: 6.32535%

\- Rule 3: 3.4421%

\- Rule 4: 4.484%

\- Rule 5: 27.83735%. This is the "medium tendency for samples to be mediumly
out of control".

\- Rule 6: 8.49465%

\- Rule 7: 4.38775%

\- Rule 8: 2.1294%

(Edited after some bug fixes that, comfortingly, didn't change the results by
much!)

[0]
[https://en.wikipedia.org/wiki/Mixture_distribution](https://en.wikipedia.org/wiki/Mixture_distribution)
[1]
[https://en.wikipedia.org/wiki/Seven_states_of_randomness](https://en.wikipedia.org/wiki/Seven_states_of_randomness)
[2] [http://lpaste.net/137664](http://lpaste.net/137664)

~~~
huac
a confusion matrix would help a lot here for a better idea of classification
performance

------
darkhorn
I have B.S. in Statistics and I have never heard the term "Nelson Rules".
However all of this information was taught under other names when we were
dealing with "Normality". Also, don't forget to convert your data into
Standard Normal Distribution (and it is not that simple, you have check some
tests also!!!). And of course you will always make mistakes because even 4th
year Statistics students make mistakes...

------
anonu
I wonder what the application, if any, there is to finance/stock charting?

~~~
haddr
Stock prices are more complex than that. In general they don't follow normal
distribution.

Then they have some sort of trend.

Then they might be autocorrelated.

Then sometimes they have some seasonality.

Also different time series are correlated between them to some degree..

------
niels_olson
We live by these things in the clinical lab, though we know them as the
copyrighted version, the "Westgard rules". Glad to have the original, thanks!

------
zwerdlds
There is also a MILSPEC for general quality control processes, though I can't
find the particular document at the moment.

If you're one of the people in the thread describing this as too subjective or
strict, the MILSPEC is probably more appropriate for your process.

