> The standard best practices in STATS 101 is to compute R^2 coefficient (based on data of the sample), which is akin to reporting error estimates on your training data (in-sample predictive accuracy).
That's not the best practice at all. It ain't even standard because adj R^2 exist to penalized coefficient cheating and if you cheat on your degree of freedom. And on top of that we got other penalization functions other than R^2 such as AIC and AAIC.
All of them are just for comparison between similar type of models and it's not suppose to be use for generalization test for unseen data. We're taught CV too and it is of statistic invention. And also taught about training set and test set with CV.
If you want to debug for test data you have to do CV anyway for imbalance data. So this is comparing apple to banana with R^2 vs out of sample.
You can see all of this in the book that applied statistic uses for linear regression, Applied Linear Statistical Models by Kutner & et al.
That's not the best practice at all. It ain't even standard because adj R^2 exist to penalized coefficient cheating and if you cheat on your degree of freedom. And on top of that we got other penalization functions other than R^2 such as AIC and AAIC.
All of them are just for comparison between similar type of models and it's not suppose to be use for generalization test for unseen data. We're taught CV too and it is of statistic invention. And also taught about training set and test set with CV.
If you want to debug for test data you have to do CV anyway for imbalance data. So this is comparing apple to banana with R^2 vs out of sample.
You can see all of this in the book that applied statistic uses for linear regression, Applied Linear Statistical Models by Kutner & et al.