
The basic concept of least-squares modelling - gballan
http://blabr.io/?63a3263c90c7af9cf8a7
======
shoo
This is a nice interactive example, but it doesn't explain _why_ one would
prefer to use a least-squares estimator over any other arbitrary method. For
example, one could measure the error in any other L_p norm - e.g. L_1 or
L_{\infty} instead of L_2. What's so special about this one?

To answer that, we have the Gauss-Markov theorem [1] -- provided certain
assumptions regarding the data are met, least squares is the best linear
unbiased estimator.

Note, however, that no-one is forcing us to choose an estimator that is either
linear or unbiased. What if by dropping one of those two somewhat arbitrary
constraints we could get a better estimator?

One example of this is the James-Stein estimator [2], which is a biased
estimator that has the amusing property of dominating (i.e., always performing
no worse than, and performing strictly better than in some case) the bog
standard least-squares estimate when one is dealing with data in three or more
dimensions (and a few more common assumptions).

Crudely speaking, the James-Stein estimator produces estimates that are biased
toward the origin, when compared to estimates produced from ordinary least
squares. It is also possible to define a James-Stein estimator that biases
toward an arbitrary fixed non-origin vector -- these variants also dominate
the bog-standard ordinary least squares estimator! Personally I find this
result both intriguing and also reasonably offensive (which means my
mathematical intuition needs adjustment).

[1] -
[https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem)

[2] -
[https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator](https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator)

------
rmohanx
@shoo: Pretty interesting, Stein. Thx. Taleb advocates abs. deviation, mean
abs. dev. and the like.

OP named reason to pick least squares: "easy to calc" IIRC, the rest of the
rationale didn't exist back when it was invented and picked....

------
dvh
When explaining the least-squares, why not drawing the damn squares in the
chart then?

~~~
gballan
Author (and Blabr dev) here. I have no answer--other than it seems like a
really good idea, now that you've said it.

The plotting widget I used is based on flot [0]. It looks like flot can draw
simple shapes ([1], see "markings"). I'll look into improving the widget--
thanks for the idea.

[0] [http://www.flotcharts.org/](http://www.flotcharts.org/)

[1]
[https://github.com/flot/flot/blob/master/API.md](https://github.com/flot/flot/blob/master/API.md)

