

State of Hyperparameter Selection - Zephyr314
http://www.startup.ml/blog/hyperparam

======
TheEzEzz
Can we train a deep neural net to pick the hyperparameters of deep neural
nets? Then we can train it on itself.

~~~
drsaltiel
Interesting idea. Generally, neural nets are not ideal for data sets with a
small number of observations (as in the case of hyperparameter selection). An
additional consideration is that if our hyperparameter optimization itself has
just as many hyperparameters, we aren't really helping much.

Most damning though is probably that neural nets do not adjust to new data
quickly - we would have to train a new one every time we acquire a new data
point and training neural nets is incredibly expensive (it might be cheaper to
train this new neural net than the original one, since hyperparameters are
almost always of lower dimension than the data being fit, but it will
certainly be more expensive than modeling a Gaussian Process).

