> the system tries every setting and graphs the results so it's easy to pick out the best setting. How would that happen? How does the system know what "good" is?
In a research field of Parameter Tuning we try to answer that question. The field is more focused to optimizing algorithm parameters, but it could be applied to physical world of robots etc., if it is feasible to automatically repeat the experiment few dozen to around hundred times.
If we would do like Bret proposed, by logging each experiment we would already have done some "probing" on the parameter space. This, in turn, would allow us to build a statistical model of the phenomena and then minimize/maximize on that. The resulting parameter configuration would then be evaluated, the model updated and the process repeated until satisfactory level of performance was reached.
See for example the recent works from Hutter et al. [1], where they use Random Forests with parameter tuning to make parameter "goodness" predictions (in order to reduce the actual experiments on the target algoritm/robot/whatever).
In a research field of Parameter Tuning we try to answer that question. The field is more focused to optimizing algorithm parameters, but it could be applied to physical world of robots etc., if it is feasible to automatically repeat the experiment few dozen to around hundred times.
If we would do like Bret proposed, by logging each experiment we would already have done some "probing" on the parameter space. This, in turn, would allow us to build a statistical model of the phenomena and then minimize/maximize on that. The resulting parameter configuration would then be evaluated, the model updated and the process repeated until satisfactory level of performance was reached.
See for example the recent works from Hutter et al. [1], where they use Random Forests with parameter tuning to make parameter "goodness" predictions (in order to reduce the actual experiments on the target algoritm/robot/whatever).
[1] http://www.cs.ubc.ca/labs/beta/Projects/SMAC/