
Don't Cite the No Free Lunch Theorem - another
https://peekaboo-vision.blogspot.com/2019/07/dont-cite-no-free-lunch-theorem.html
======
jmmcd
Good! This article is about the NFL for machine learning (Wolpert), and also
mentions NFL for optimisation (Wolpert and Macready). My recent paper [1] is
about NFL for optimisation and mentions NFL for ML. The overall message is the
same: most people misunderstand it, and you probably shouldn't cite it. And
the anthropic principle is a sufficient assumption for this.

[1] [https://arxiv.org/abs/1906.03280](https://arxiv.org/abs/1906.03280)

~~~
yters
If the NFL doesn't apply in the real world, then where is the one algorithm to
rule them all?

~~~
jmmcd
There probably isn't one! But as argued in the paper, our reasons for thinking
so are not based on NFL.

------
meerita
The first thing that came to my mind was that Milton Friedman quote "there's
no such free meal, someone has to pay it".

~~~
segfaultbuserr
> _Milton Friedman quote_

This meme was actually popularized by Robert Heinlein in _The Moon Is a Harsh
Mistress_ , ten years earlier than Milton Friedman started using it.

~~~
meerita
I learnt it from Milton, but it's nice to still discover other people figured
out this.

------
yters
Even though the NFL only holds absolutely when problems are closed under
permutation, that doesn't imply that most problems are a good fit for a
particular algorithm:
[https://arxiv.org/pdf/1609.08913.pdf](https://arxiv.org/pdf/1609.08913.pdf)

------
eli_gottlieb
>What it actually (vaguely) says is “You can’t learn from data without making
assumptions”.

That's what they taught us it means in our graduate machine learning class.
What were people citing it to mean?

~~~
contravariant
This is pretty much equivalent to 'you can't learn from random noise', isn't
it?

~~~
eli_gottlieb
It's a little more like, "If you assume nothing is more complicated or costly
than anything else, then any set of examples you see could just as easily be
generated adversarially to trick you as not."

------
bradknowles
TANSTAAFL?

~~~
amingilani
I think you're asking if this is about "there ain't no such thing as a free
launch." It isn't, I thought it was at first myself, coming from an Economics
background. It's about a machine learning theorm.

