* Instance-based methods can adapt to any kind of pattern if you have enough data. This is a well-known result in machine learning -- the interesting question is simply how much data do you need?
* It's quite remarkable how well RBF SVMs do in almost all cases. They even outperformed a 2nd degree polynomial kernel SVM on a 2nd degree polynomial!
* Logistic regression is pretty terrible for anything except "easy" data -- simple linear data.
* Random forests are sometimes ok, but tend to prefer axis-aligned data (at least in his formulation).
* Meta-point: all the tests shown are with "clean" data...i.e., there are no "wrong" training examples. This is unrealistic, and in practice makes a HUGE difference for some of these methods. E.g., a lot of the rule-based methods get demolished by even a little bit of wrong data. In contrast, SVMs have a slack variable that can tolerate some amount of noise, and would probably shine even more on such data.
Shouldn't an RBF kernel be expected to perform better than a 2nd degree poly kernel since, technically, it is infinite-dimensional? (I.e., the radial basis function can be expanded into an infinite-degree taylor series)
I think (not a ML expert by any means) that RBF should probably outperform most fixed-integer polynomial kernels, but it does so at the cost of introducing more structural risk into the model.
It does not really go like that. The higher the dimension (actually VC dimension, but polynomial dimension is a similar concept) the more points you would theoretically need to guarantee that the classifier is not overfitting (ie, representing the noise in the data set).
I think its interesting how well the instance based methods perform at comparative numbers of training samples to the SVM-RBF methods. I'd certainly choose instance based methods for tasks with similar data to these tests.
* Instance-based methods can adapt to any kind of pattern if you have enough data. This is a well-known result in machine learning -- the interesting question is simply how much data do you need?
* It's quite remarkable how well RBF SVMs do in almost all cases. They even outperformed a 2nd degree polynomial kernel SVM on a 2nd degree polynomial!
* Logistic regression is pretty terrible for anything except "easy" data -- simple linear data.
* Random forests are sometimes ok, but tend to prefer axis-aligned data (at least in his formulation).
* Meta-point: all the tests shown are with "clean" data...i.e., there are no "wrong" training examples. This is unrealistic, and in practice makes a HUGE difference for some of these methods. E.g., a lot of the rule-based methods get demolished by even a little bit of wrong data. In contrast, SVMs have a slack variable that can tolerate some amount of noise, and would probably shine even more on such data.