
A Course in Machine Learning - sebg
http://ciml.info/
======
Smerity
For those who might not know, Hal Daumé III is a highly respected researcher
in machine learning, spending a lot of time working in Natural Language
Processing (NLP). He also contributes his knowledge to many interesting open
source projects. After creating an algorithm called SEARN for solving hard
structured prediction problems, Daumé et al. went and created a practical and
open implementation of the algorithm in Vowpal Wabbit, an online machine
learning library built for speed and used at Yahoo and Microsoft Research
amongst other places. He's the sort of academic who contributes not just to
theory, but also to practice. Browsing through his publications gives an idea
of his contributions[1].

Given all that, I'm looking forward to his take on machine learning and for
another valuable resource to be available to everyone.

[1]:
[http://www.umiacs.umd.edu/~hal/publications.html](http://www.umiacs.umd.edu/~hal/publications.html)

~~~
zenbowman
Nice. He was a fellow PhD student at USC/ISI, I remember him being the dude
who always walked around barefoot.

Seems like he's done really well since, I will definitely take a look at this.

------
tedks
Hal uses this as the textbook for his undergrad machine learning course.
Having recently taken that course (in the spring 2013 semester), I feel
justified saying that this book needs a _lot_ of work before it's usable as a
textbook or even a learning tool. Most of the time this text served primarily
as an initial "dereference" of ideas yielding not concrete information, but a
series of other pointers that I'd need to chase on Google before getting
anything usable.

He clearly has high hopes for CIML, based on all the infoboxes and chapters
that are still incomplete; it's a shame he'll likely need to get tenure before
finishing it.

(This should in no way be taken as a slight on Hal's teaching; his class was
the best sort of challenging in that it required a large amount of work, and
yielded a correspondingly large amount of insight. He's also a fantastic
lecturer and a fair grader.)

~~~
ninjin
> "... it's a shame he'll likely need to get tenure before finishing it."

Judging by his homepage [1] he is at least on the tenure track. Unless the
notion of an Assistant Professor is different over in the states. I agree
about the sentiment about the book, at least the chapter on Neural Networks
needs some serious work (I believe there are errors in there, maybe I should
mail Hal). As a person and researcher I only have good things to say about
him, do check out it his blog if you are into ML and NLP [2] and let's hope he
will find the time to continue polishing on this freely available book (I do
like his writing).

[1]: [http://www.umiacs.umd.edu/~hal/](http://www.umiacs.umd.edu/~hal/)

[2]: [http://nlpers.blogspot.com/](http://nlpers.blogspot.com/)

------
denzil_correa
I wish more people followed what Hal writes in here :

> A second goal of this book is to provide a view of machine learning that
> focuses on ideas and models, not on math. It is not possible (or even
> advisable) to avoid math. But math should be there to aid understanding, not
> hinder it.

No book (yes not even PRML - Bishop), follows this diligently.

~~~
rm999
This is a harder problem than it sounds, and something I've given a lot of
thought to. I think the underlying issue is that all machine learning was
discovered through a combination of applied math and intuitive ideas/models.
Without the intuitive model no one would have thought to discover the method,
and without the math the intuitive idea would be a pipe dream. Both are
fundamentally linked, and it's a bad idea to separate them.

For example, one of the most basic and oldest statistical methods is linear
regression. The first thing anyone will tell you when they are teaching it is
the basic "idea and model" \- finding a line that fits a scatter plot (and
then extending that idea). But this doesn't give you a real understanding of
linear regression: where does that line come from, and why does a clean,
algorithmic solution exist? Why does the standard solution often lead to
numerical errors, and why is regularization a valid solution?

These questions require an increasing amount of math, but they are essential
to really understanding linear regression. I agree that some textbooks just
throw you a wall of math, but many actually do a solid job of explaining what
is going on as they do so.

If you don't want to know the details of machine learning methods, which are
inherently mathematical, you might as well as just remember the names of
libraries that implement the solutions for you.

~~~
malandrew
Completely agree. However one thing that I think is missing in most books with
a lot of math is the opportunity to use programming to help teach and
communicate the math in question.

A great example of a book that communicates abstract mathematical concepts via
algorithms is the Little Schemer. Ironically, it doesn't even set out to
communicate the math, but actually just uses the math to communicate other
programming ideas like recursion. That, however, doesn't diminish the fact
that it demonstrates ways to teach math through programming and algorithms.

Another book which also does a good job at using programming to demonstrate
more concrete math is Allen Downey's "Think Stats" book. All through the book,
you learn the mathematical concepts of statistics through hands on programming
activities.

There definitely is a chicken and egg problem in areas like machine learning
because unlike the above resources which have only one layer of abstraction to
cross, machine learning presents two layers of programming<->math abstractions
to cross for most people who decide to learn it. To really understand and
apply machine learning you need to understand the math and models behind it.
However the math and models are presented in pure form that makes it difficult
to grok unless you arrived at the resource with a classically-trained
mathematical background. I would hope that given that the target market for
such learning resources it not mathematicians but programmers, that such
learning resources would present content to help you arrive at the math from a
programmer's point of view.

------
Discordian93
Edx is also offering the caltech ML course in mooc format:
[https://www.edx.org/course/caltechx/cs1156x/learning-
data/11...](https://www.edx.org/course/caltechx/cs1156x/learning-data/1120)

~~~
mikevm
Note that this is not a watered-down course! It's the same course as the
students at Caltech are taking, and in fact they will have the option not to
attend the lectures and watch them online instead. The homework assignments
will be the same ones as well (AFAIK).

------
tga
If you are interested in this you might want to also look at Andrew Ng's
(Stanford) Machine Learning course that is starting soon on Coursera.

[https://www.coursera.org/course/ml](https://www.coursera.org/course/ml)

~~~
kyro
Is this worth going through over picking up a textbook or two? I've found that
Coursera courses are actually quite bloated. Lots and lots of empty talking,
and very little substance.

~~~
sampo
I've taken the Coursera ML class. It's very easy if you have the adequate math
background. And it's not very comprehensive, there are lots of machine
learning methods that are not covered. So it's more like an introductory
course to machine learning.

But it's absolutely commendable how Andrew Ng takes the topic to such an
understandable level that a clever high schooler who knows a little about
programming could take the course. There's even extra videos serving as a
crash-course to linear algebra and octave programming in the first week.

So he really manages to make the topic accessible to a very large audience.

~~~
dominotw
>adequate math background

Do you know what kind of math is needed other than linear algebra.

~~~
sampo
Basic vector and matrix operations. The first half of a typical freshman
linear algebra course is more than enough. But like I said, there is a matrix
review in the beginning, so if you're willing to study those extra lectures,
then almost no prior knowledge is needed.

Also being able to take derivatives helps in a couple of places, but is not
necessary.

~~~
yelnatz
Thats linear algebra.

------
genevievemp
Just wanted to add another textbook option that's of comparable breadth and
quality, but easier to read than bishop. There are pre-print pdfs probably
available online with a little searching. The exercises and accompanying data
are really fun imo.
[http://www.cs.ubc.ca/~murphyk/MLbook/index.html](http://www.cs.ubc.ca/~murphyk/MLbook/index.html)

~~~
gtani
It's an excellent book, good balance of rigor and code/graphs/visuals to
develop intuition, but pls see errata before buying

[http://www.cs.ubc.ca/~murphyk/MLbook/errata.html](http://www.cs.ubc.ca/~murphyk/MLbook/errata.html)

(and the freely available content books by Barber, McKay and
Hastie/Tibshirani/Friedman (ESL) are all worth close reads

------
UK-AL
I think he needs to re-run latex a couple times before releasing that pdf.
Fair few ?? in there...

~~~
adiM
The lines are not properly justified, so it appears that the output is not
generated using TeX. I fail to see why would anyone choose left flushed
alignment (with hyphenation!) when using TeX?

------
nrox
The book seams accessible.

This subject is difficult and requires a lot of time to understand and put
something useful into practice. AI books appear frequently these days. I feel
that most of books and tutorials fail to deliver good practical examples and
meta-code, focusing more on mathematical proofs.

I am struggling to understand Peter Abeel's aprenticeship learning:
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.74....](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.74.1035&rep=rep1&type=pdf)

He can resume the thing in a couple of slides. It seams short and simple (!?)
but I think I am still away from truly understand it. Maybe next year.

------
chatman
The pages are watermarked, "Draft. Do not distribute". I wouldn't rely so much
on these lectures unless they stabilize.

~~~
winter_blue
Doesn't make them any less worth reading.

~~~
frozenport
If the content is wrong or suboptimal it makes them less worth reading. If the
guy hasn't spent time thinking about the content of his slides it might be a
confused and confusing jumble of nonsense: sometimes the best researchers are
the worst teachers.

------
genofon
please change the logo! I'm not trying to be jerky, it's always remarkable
that someone wants to share his knowledge and I know probably it's not your
priority, but the book and website will be better off without it.

also why do you have the topics "introduction to machine learning" and
"decision trees" in the same chapter?

------
mrcactu5
this looks like a pleasant introduction to machine learning

