
Type I and Type II Errors: The Inevitable Errors in Optimization Experiments - ashfromconvert
https://utm.io/uuIl
======
jhayward
I'm reminded of this apt tweet [1]

> _If a Type I error is a false positive, a Type II error is a false negative,
> and a Type III error is getting the right answer to the wrong question, is a
> Type IV error GIVING IMPORTANT CONCEPTS NUMBERS INSTEAD OF NAMES FOR NO
> GODDAMN REASON AND CONFUSING GENERATIONS OF STUDENTS_

Also, I find the Confusion Matrix[2] to be a helpful reference, especially the
multi-colored table with formulae for each condition.

[1]
[https://twitter.com/mjskay/status/1201380151356989440](https://twitter.com/mjskay/status/1201380151356989440)

[2]
[https://en.wikipedia.org/wiki/Confusion_matrix](https://en.wikipedia.org/wiki/Confusion_matrix)

~~~
avz
To highlight the persistence of the confusion compare the confusion matrix in
two wikipedia articles: [1] and [2].

[1]
[https://en.wikipedia.org/wiki/Confusion_matrix](https://en.wikipedia.org/wiki/Confusion_matrix)

[2] [https://en.wikipedia.org/wiki/Pre-_and_post-
test_probability...](https://en.wikipedia.org/wiki/Pre-_and_post-
test_probability#By_predictive_values)

(In case edits are made one way or the other: both articles display similar
confusion matrices but disagree on the names of the off-diagonal elements. In
[1] false positives are called "type I errors" and false negatives are called
"type II errors". In [2] false positives are called "type II errors" and false
negatives are called "type I errors".)

(Edit: layout)

~~~
ImaCake
I _hate_ the number naming system. I have no idea why anyone would find it
preferable to use them for literally any reason. Specificity and Sensitivity
already almost mean the right things in common language already, they are
perfect words for this. The only use for these is to confuse people so they
don't realise you have no idea what you are talking about.

I think some people struggle to grasp that these are actually strict
mathematical concepts. But personally, that makes me feel comfortable with
them. It is a lot easier to get a math concept consitently right than some of
the fuzzier concepts in epidemiology.

------
Scea91
It is funny that although I have published peer-reviewed papers about
evaluation of ML models and could probably discuss with you about many nuances
of the process if you woke me at 3 AM, I always find it difficult to remember
which of [Type I, Type II] is false positive and which is a false negative. If
I have this problem I suppose almost everyone has it.

I wish people stopped using [Type I, Type II] when we can use clearly superior
terminology. This feels to me very similar to amateur software engineers using
non-descriptive variable names.

~~~
pfortuny
This is exactly my experience. I have never known what type the error I was
talking about was...

Imagine calling (for instance) type I groups those which are abelian and type
II those which are not... And then not being a professional mathematician. So
is xy=yx here? Mmmmhhh type I says yes or was it no?

~~~
laichzeit0
I have the same problem with specificity and sensitivity. Always have to look
up the damned terms even though I’ve used them countless of times.

~~~
ImaCake
They mean what they would mean in common english.

Specificity because "how specific is the test? Does it measure the true
positives correctly without getting a whole heap of false positives as well?"
You might ask someone to be "more specific" about something so they aren't
including irrelevant things in their discussion.

Sensitivity because "how sensitive is the test? Does it detect the needle in
the haystack you need to find? How many does it miss." In common english you
might complain that your car brakes are too sensitive - they are too quick to
register the pressure from your foot.

Don't forget these are actually strict mathematical concepts. Hope this helps
clarify :)

~~~
parekhnish
I think the issue people have with these names is that their English meanings
(as you described) make sense when the positive class is not as prevalent as
the negative class. If they are equally probable (or worse, if it's the
opposite), then the English meanings quickly become out-of-context

------
yters
If I accidentally swap "Type I" and "Type II" in my mind, is that a Type I or
Type II error?

~~~
ashfromconvert
You've just made a Type I error because you've lacked 'specificity' in your
mind by swapping the two! :O

------
selimthegrim
Just to mix things up:

[https://statmodeling.stat.columbia.edu/2019/03/12/r-package-...](https://statmodeling.stat.columbia.edu/2019/03/12/r-package-
for-type-m-and-type-s-errors/)

------
Volt
What's with the shortened URL?

