

Everyday Statistics for Programmers: Nonlinear Regression - koblenski
http://sam-koblenski.blogspot.com/2014/10/everyday-statistics-for-programmers_21.html

======
dietrichepp
The terminology is wrong: the polynomial curve fit is still linear regression.
The "linear" in "linear regression" does not mean that the resulting function
is linear, it means that the resulting function is a linear combination of
basis functions. For a polynomial, the basis functions are 1, x, x^2, ...,
x^n.

The exponential fit, however, is non-linear.

From Wikipedia
([http://en.wikipedia.org/wiki/Polynomial_regression](http://en.wikipedia.org/wiki/Polynomial_regression))

> In statistics, polynomial regression is a form of linear regression...

~~~
koblenski
I should have noted that and said polynomial regression is linear regression
of nonlinear data. Thanks for the correction.

------
jules
You have to be careful here. If you transform your data, and then do a linear
regression on the transformed data, and then un-transform your regression
line, that's _not_ the same as doing non-linear regression. Linear regression
makes the assumption that the data is a line + normally distributed errors
with 0 mean and constant variance.

For example lets take regression for an exponential relationship. If you
transform your data with log, and then do linear regression you are assuming
that your transformed data fits a line with normally distributed errors with 0
mean and constant variance. This means that you are assuming that your
original data fits an exponential + normally distributed errors with 0 mean
and _exponentially increasing variance_. This might be a good assumption, but
it might not be. It depends on what your data is. If you look at his data [1]
you see that the variance is increasing, so that indicates that his method is
OK for that data.

What is going to happen with his method is that points far away from 0 are
going to weigh far less into your regression than points near 0. It's not the
same as doing a non-linear regression, which would assume that your data fits
an exponential + normally distributed errors and constant variance. That would
give all points equal weight.

[1]
[http://1.bp.blogspot.com/-jL4JsLT5M6o/VEXCNmFesaI/AAAAAAAAD3...](http://1.bp.blogspot.com/-jL4JsLT5M6o/VEXCNmFesaI/AAAAAAAAD38/MT0bJSccnOk/s1600/exponential_data_with_trendline.png)

~~~
koblenski
That's a really good point. I didn't want to get into nonlinear solvers for
this post, but I should have mentioned the considerations, at least.

------
therealdrag0
Relatedly, here are some free ebooks directed at a similar audience and
subject material.

Think Stats: Probability and Statistics for Programmers
[http://greenteapress.com/thinkstats](http://greenteapress.com/thinkstats)

Think Bayes:
[http://greenteapress.com/thinkbayes](http://greenteapress.com/thinkbayes)

------
mierle
If you're using C++, consider looking at the Ceres Solver tutorial [1]

[http://ceres-solver.org/tutorial.html](http://ceres-solver.org/tutorial.html)

------
pfortuny
Oh dear! He does not even discuss the HUGE errors that may appear when taking
logarithms... If you have points near 0 and away from 0 the former get an
artificial enormous weight and bias the "interpolation" too much towards them.

CAREFUL!

Basically: the mean value of the logarithms is not the logarithm of the mean
value.

~~~
rprospero
Wouldn't the arithmetic mean of the logarithms be the logarithm of the
geometric mean?

1/n Σ ln xᵢ = 1/n ln (Π xᵢ) = ln ( n√(Π xᵢ) )

------
roel_v
Can somebody explain how a logistic regression fits into this explanation, if
at all?

~~~
clouden
Logistic regression is similar to his example on fitting a line to an
exponential trend. The differences is that in logistic regression, the
equation would be

ln(p)/ln(1 - p) = α + Σßᵢxᵢ

where p is in the interval [0, 1] and E is .

Note: in logistic regression, the notation is generally p (as in probability
of event), not y.

