

Ask HN: Machine learning in 5 papers.  What should they be? - snikolov

Hey folks,<p>I am helping to organize a reading group to introduce people to machine learning in a way that is more informal and interactive than a class.  With only 5 meetings slated for this semester, what do you think would be a good selection of topics to cover?  Ideally one would come out of this reading group with high level appreciation of ways to think about problems in machine learning (feature selection, training, supervised vs. unsupervised learning, function approximation, etc) as well as familiarity with some high level tools (statistics, optimization, linear algebra).  This is not meant to be a class, which makes it all the more difficult to strike a balance between depth and breadth.<p>Any thoughts, as well as paper or topic suggestions would be greatly appreciated.
======
cschmidt
I would cover some fairly basic topics:

1\. Naive Bayes (with review of probability needed to understand)

2\. Linear Regression

3\. Decision Trees (for a nonlinear model)

4\. Boosting (and why it doesn't overfit much)

5\. k-means (and other clustering)

Be sure to emphasize cross validation or a holdout set to evaluate all these
models.

The new edition of the "WEKA" book would be a nice text, rather than the
original papers.

Data Mining: Practical Machine Learning Tools and Techniques, Third Edition
Ian H. Witten, Eibe Frank, Mark A. Hall

~~~
snikolov
Thanks! This is a great list.

------
rgens
I'd recommend the slides from Pedro Domingos' (UW) ML class. They give a broad
overview of the ML landscape. He follows Tom Mitchell's (CMU) book pretty
closely. <http://www.cs.washington.edu/education/courses/cse546/10wi/>

Both the slides and the book are highly readable. There are no good papers
that cover everything. I wouldn't necessarily dive in to a paper on
regularization or non-parametrics for a beginning class (is this a slug talk
series?). The Netflix paper really doesn't convey good understanding of ML
(they combine a large number of techniques at the problem) but you might want
to cover boosting using a different source.

------
mitko
Few things that I know & like (they are rather heavy but deep):

    
    
       F. Cucker and S. Smale. On The Mathematical Foundations of Learning. Bulletin of the American Mathematical Society, 2002.
    
       Sutton & McAllum : Conditional random fields for relational learning
    
       Pieter Abiel: apprenticeship learning with a helicopter (inverse reinforcement learning)
    

You can also add the paper of the team that won the Netflix challenge
(collaborative filtering), and something about non-parametric learning
(dirichlet processes etc.)

Also check out Lasso regularization (L1 norm) for sparse feature selection.

~~~
snikolov
Thanks, this is all great stuff. I've been meaning to read some of Smale's
papers on learning theory. It might be over a lot of people's heads though,
since we advertised this is an introduction to basic concepts that one might
encounter in a first course in Machine Learning. We are doing two groups of
10. The topics will have to depend on the kinds of people that applied and
their interests, but I like the idea of having a beginner group and an
advanced group (possibly based on mathematical level).

