

Ockham's Razor is Dull - neilc
http://apperceptual.wordpress.com/2008/06/14/ockhams-razor-is-dull/

======
boredguy8
"simplicity is a guide to truth" is not a summary of Occam's Razor. It says
that _all things being equal_ the simplest explanation is generally the best.
If you have an elegant solution that doesn't account for everything, there's a
problem[1]. That problem is either in the theory or in the data, but until
that problem is resolved, you're not helping advance much of anything.

So, for instance, we can still (I am told by credible astrophysicists) posit a
geocentric cosmology if we're willing to adapt a rather complex system. But
rather than think this is true, we have an alternative hypothesis that
accounts for the same data without introducing a plethora of adjustments.

But of course, gravity _alone_ doesn't count for the data, hence Einstein.

Gravity is a "simpler" explanation, but it doesn't _account_ for our
observations, so it get axed.

Scientists doing philosophy without training is almost always a very scary
thing to witness.

[1] Re: we don't make curves that fit every data point: those curves are not
-predictive- (i.e. you can perfectly account for -known- data, but you blow
when you try to predict -future- data) and hence don't provide a reliable
account of phenomenon. So they're rejected on that basis.

~~~
neilc
So given an infinite set of possible theories to explain a data set, which
theory should be preferred? Simple accuracy over the known data is frequently
not sufficient, as you point out, because of overfitting. Given noisy data
(which is almost _always_ a given), you can't just use maximal accuracy as the
sole criteria for which theory to prefer. One way forward is to combine
accuracy with some measure of the complexity of the theory (e.g. minimum
description length), which has much of the flavor of Occam's Razor.

------
dhbradshaw
Interesting article and perhaps more interesting comments after it.

I wanted to add one point. The focus of the article is on the utility using
Ockham's Razor for finding correct (or predictive) models of the world. A
second reason to use Ockham's Razor is that simpler theories can have more
utility because they are easier to use and understand. Often a simpler theory
can be more useful even if it is less correct.

------
sah
The reason to prefer simplicity is that fewer bits of information are required
to store a simple theory. This makes simple theories easier to use correctly
(since there are fewer bits which can be subject to transcription errors, for
example). So among equally predictive theories, the simplest one has important
advantages.

If simplicity is also aligned with some bias in the natural world, that's
purely a bonus. You might expect that to be the case if simple mechanisms were
more probable than complex ones, and simple mechanisms were more likely to
lead to simple behavior. So maybe there's something to that idea, but maybe
not -- this author seems to know a lot more about it than I do.

------
xlnt
The content of Ockham's Razor is "don't make stuff up for no reason".

If it's dull then you're doing it wrong.

~~~
Herring
I always thought this was completely obvious, but most people tend to miss it.
It's not about accuracy at all. It basically says unnecessary assumptions are
unnecessary.

