
Level-Up Your Machine Learning - cjrd
http://metacademy.org/roadmaps/cjrd/level-up-your-ml
======
xiaoma
I think this is fantastic advice.

As someone who has spent an embarrassing amount of time on various independent
education, one of the key things I've taken from it is just how efficient text
books are. Not only can you read more quickly than people can speak, but it's
also active by nature. I've often found my attention wandering during videos,
but it's just not possible to read without putting in a minimum amount of
focus. It's also a lot easier to modulate your reading speed based on how easy
material is for you than it is to do the same during a lecture video.

Some general thoughts on MOOCs:

Coursera and edX tend to be great for small, self-contained topics and the
automated graders for programming assignments is great as well. The forums are
also useful, though not ideal (since there are no practice questions students
can get help with that don't fall under the honor code).

Where modern MOOCs really fall down is prerequisites. It's surprisingly
difficult to do something like structure an entire CS degree from Coursera
classes. Though many classes are taught by famous CS professors, they are from
different institutions that break material into courses in different ways.
Worse still a lot of the classes are either watered-down or shortened or both.

MIT's Open Courseware archives are actually a lot better for this. There are
no certificates, and no credentials, but nearly all the material is freely
available. The one biggest inefficiency though, is all the time spent in the
lectures. At least they can be played back at a higher speed, but the lectures
really do take a lot more time and cover less than the textbooks. For courses
that have good textbooks, I think the best approach is to skip the lectures
except in portions where you feel like you need more review.

Finally Khan Academy is fantastic for answering specific, mechanical questions
(e.g. how to calculate eigen values), but a bit light on material. I'd use it
as a supplement for the other resources.

~~~
furyofantares
I tend to watch lectures at 2x or 1.75x speed, and I tone it down for more
difficult stuff, or pause, or rewatch. I think at 2x it's pretty even with how
fast I can read, maybe even a bit faster. Modulating it isn't perfect but it's
workable. Also, I struggle very hard to pay attention to almost any lecturer,
my mind drifts very quickly and takes a while before I realize it's drifted.
But at 2x I find my ADD is mostly defeated.

~~~
sadkingbilly
I also watch at 2x speed. I do the same for audiobooks. Your brain adapts.
Sometimes I play them in 1x speed just for a bit and it sounds like extreme
slow motion, and I think, people really listen to this?

------
rfrey
I love textbooks and spend more of my childrens' inheritance on them than I
should.

But what MOOCs give me is the _exercises_. I often think I understand a
problem, but it's only after getting 2.1/10 on a Coursera quiz that I realize
I've missed a key step or concept.

Many textbooks have exercises but few have solutions. I've been working
through Barto and Sutton's Reinforcement Learning for example (again), and
although I do the exercises and programming questions, I never know if I've
gotten it _right_. My experience with MOOCs shows I probably haven't in a
large number of cases.

The best of both worlds is when I can follow a MOOC with the textbook to gain
more depth, for example with the PGM course on Coursera.

~~~
ced
Incidentally, I liked Barto and Sutton's book, but it feels a bit dated. Does
anyone have another reinforcement learning book to recommend?

~~~
camlinke
I chatted with Rich Sutton recently and he said he was writing and update -
not sure how far along in the process he is though.

Also Csaba Szepesvári (a colleague of Rich's at the U of A) has a free RL book
you can download.
[http://www.ualberta.ca/~szepesva/RLBook.html](http://www.ualberta.ca/~szepesva/RLBook.html)

~~~
noelwelsh
Csaba's book is the most up-to-date on RL I know of. Sutton and Barto is very
old by now. For the POMDP side of things there are no recent books I know of,
but [http://www.cs.mcgill.ca/~jpineau/files/sross-
jair08.pdf](http://www.cs.mcgill.ca/~jpineau/files/sross-jair08.pdf) is a
recent enough survey.

A related book is "Regret Analysis of Stochastic and Nonstochastic Multi-armed
Bandit Problems" which you can find at
[http://www.princeton.edu/~sbubeck/index.html](http://www.princeton.edu/~sbubeck/index.html)

The bandit problem is very strongly related to the reinforcement learning
problem, so you'll get some mileage out of studying bandits. Be aware this
area is very maths heavy, which is good or bad depending on your background.
If you like you like this stuff, also checkout "Prediction, Learning, and
Games" which deals more with the "adversarial" setup.

------
grayclhn
Two free books that I haven't seen mentioned, that are from more of a stats
perspective

* James, Witten, Hastie, and Tibshirani's _An Introduction to Statistical Learning, with Applications in R_

[http://www-bcf.usc.edu/~gareth/ISL/](http://www-bcf.usc.edu/~gareth/ISL/)

* Hastie, Tibshirani, and Freedman's _Elements of statistical learning_ (more advanced)

[http://statweb.stanford.edu/~tibs/ElemStatLearn/](http://statweb.stanford.edu/~tibs/ElemStatLearn/)

------
scottlocklin
PGM is a tough book. I'm not sure it's the right book for "level 3" unless you
want to be a level-3 who is good at PGMs.

The problem with ML is there are so many different kinds. Bishop's book is a
decent light weight survey, but it doesn't come close to covering all the
interesting fields. You could read that and Hastie/Tibshirani's book and still
know almost nothing about online training (hugely important for "big data" and
timeseries), reinforcement learning (mentioned, but not in any depth), agent
learning, "compression" sequence predicting techniques, time series oriented
techniques (recurrent ANNs for starters, but there is a ton to know here, and
most interesting data is time ordered), image recognition tools, conformal
prediction, speech recognition tools, ML in the presence of lots of noise, and
unsupervised learning. I don't own PGM, but it probably wouldn't help much in
these matters either. I know guys who are probably level 4 at machine learning
who don't know about most of these subjects. On the other hand, Peter Flach's
book "Machine Learning" at least mentions them and makes pointers to other
resources.

"Deep learning" is becoming kind of a buzzword for a big basket of tricks. I
think it's worth knowing about drop-out training, and the tricks used to do
semi-supervised learning, but the buzzword is silly. Technically "deep
learning" just means "improved gradient descent." I figure level-4 is anyone
making progress coming up with new techniques.

That said, reading good books is one way to make progress. Knowing the right
people is the other way.

~~~
scottlocklin
aaaand, after a late night coding session, this turned into a 2000 word tirade
which might be of interest:
[http://scottlocklin.wordpress.com/2014/07/22/neglected-
machi...](http://scottlocklin.wordpress.com/2014/07/22/neglected-machine-
learning-ideas/)

------
tel
PRML is great. I haven't read PGM, but I took a relatively intensive course on
it which had great lecture notes. Which I'd like to also suggest—lecture notes
are often "skeletal books" which can bring you up to speed on a topic quickly
given that you (a) are willing to work a bit more and (b) can fill in the
missing fleshy bits with your own experience.

I'd also really like to suggest DGL
([http://books.google.com/books/about/A_Probabilistic_Theory_o...](http://books.google.com/books/about/A_Probabilistic_Theory_of_Pattern_Recogn.html?id=5uCTngEACAAJ))
and Bickel and Doksum ([http://www.amazon.com/Mathematical-Statistics-Basic-
Selected...](http://www.amazon.com/Mathematical-Statistics-Basic-Selected-
Topics/dp/0132306379)). These are two of my _favorite_ core ML/stats books.

------
vkhuc
There are some (free) good books that haven't been mentioned yet:

1) "Data Mining and Analysis: Fundamental Concepts and Algorithms" by Zaki and
Meira
[http://www.cs.rpi.edu/~zaki/PaperDir/DMABOOK.pdf](http://www.cs.rpi.edu/~zaki/PaperDir/DMABOOK.pdf)

This book covers many ML topics with concrete examples.

2) "Computer Vision: Models, Learning, and Inference" by Simon Prince:
[http://web4.cs.ucl.ac.uk/staff/s.prince/book/book.pdf](http://web4.cs.ucl.ac.uk/staff/s.prince/book/book.pdf)

Despite a CV book, the first half of it is like a statistics book that comes
with examples in CV which are very easy to follow.

------
dimatura
I would also suggest K. Murphy's Machine Learning for the journeyman level. In
the intermediate apprentice-journeyman level Alpaydin's Introduction to
Machine Learning is very friendly.

~~~
ivan_ah
I second the recommendation of Murphy. It's very comprehensive and well
written.

[http://www.cs.ubc.ca/~murphyk/MLbook/](http://www.cs.ubc.ca/~murphyk/MLbook/)

------
kashifr
Form my own journey I would say that a good place to start for graphical
models might be "Bayesian Reasoning and Machine Learning" by Barber. It's free
([http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=...](http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online)).
I haven't read through it, but I've heard good things. However, it doesn't
cover some basic things like SVM, RVM, Neural Networks...

For those I'd suggest "Pattern Recognition and Machine Learning" by Bishop.
I've read throughout this and it's really well organized and thought out. For
more mathematically advanced ML stuff I'd suggest "Foundations of Machine
Learning" by Mohri. For a good reference for anything else I'd suggest
"Machine Learning: A Probabilistic Perspective" by Murphy. For more depth on
graphical models look at "Probabilistic Graphical Models: Principles and
Techniques" by Koller.

On the NLP front there's the standard texts "Speech and Language Processing"
by Jurafsky and "Foundations of Statistical Natural Language Processing" by
Manning.

I also like "An Introduction to Statistical Learning" by James, Witten, Hastie
and Tibshirani.

~~~
vkhuc
I skimmed over Mohri's book and I think the topics it covers are quite narrow.

For mathematical foundations of ML, I would recommend the book "Understanding
Machine Learning: From Theory to Algorithms" by Shai Shalev-Shwartz.

A brief version of the book is available to download on the author's website:
[http://www.cs.huji.ac.il/~shais/Handouts.pdf](http://www.cs.huji.ac.il/~shais/Handouts.pdf)

~~~
achompas
Yes, Mohri's book takes a strong learning theory approach.

At the same time, it's the only book I've seen that covers online learning
well. Can you think of any others?

------
cipher0
Great recommendations, some people might also find this interesting as a
general guideline to "Data science" [http://nirvacana.com/thoughts/becoming-a-
data-scientist/](http://nirvacana.com/thoughts/becoming-a-data-scientist/)

[edit] scroll down and look at the map.

------
eli_gottlieb
What I'd really appreciate is ideas on how to learn or review the core math
concepts. I haven't actually _done_ any multivariable calculus, vector/matrix
calculus, or linear algebra in _years_ , even though I took them in undergrad.

------
joaomsa
Wholeheartily agree with the author's sentiments on the value of textbooks.
Not because of the medium itself, but because they're (usually) accompanied by
well thoughtout examples and practice problems.

When initially starting a dense subject such as PGM, having my hands held
through the introductory material with incremental practice problems as the
topic elaborated, helped me get a much more intimate grasp. Initially only
reading superficially and watching lectures, I kept getting stumped trying to
form a cohesive mental map of all the interleaved concepts.

------
GabrielF00
What are the HN community's thoughts on Learning from Data by Abu Mostafa,
Magdon-Ismail and Lin ([http://amlbook.com/](http://amlbook.com/))? The
lectures from their course are here:
[http://work.caltech.edu/lectures.html](http://work.caltech.edu/lectures.html)

I haven't started it yet, but this book was recommended by some folks at my
company.

~~~
misiti3780
I really liked it - it was a lot smaller than I thought it was going to be (I
didn't look at the page count) but they definitely explain some really import
core ideas in ML like bias/variance tradeoff, curse of dimensionality, etc. in
a clear way.

~~~
misiti3780
important _

------
lowglow
I'm in the middle of PGM right now. It's actually really easy to follow if you
put some time into it. I'm reading PRML next. I didn't realize there was a
'path' to learning ML though, thanks for that.

We could use some more ML recs on
[https://books.techendo.com/](https://books.techendo.com/)

------
kp25
I would like to start my ML Journey in Python, then get to R.

How about learning things in python? Good recommendations?

~~~
wisty
If you go Python, you could start here:
[http://nbviewer.ipython.org/github/ptwobrussell/Mining-
the-S...](http://nbviewer.ipython.org/github/ptwobrussell/Mining-the-Social-
Web-2nd-Edition/tree/master/ipynb/)

------
jpeterson
For a really nice introductory book, try "Machine Learning" by Tom Mitchell.

------
orasis
Textbooks? Really?

How about start with a great lecturer like -

Nando de Freitas -
[https://www.youtube.com/channel/UC0z_jCi0XWqI8awUuQRFnyw](https://www.youtube.com/channel/UC0z_jCi0XWqI8awUuQRFnyw)

David Mackay
-[http://videolectures.net/course_information_theory_pattern_r...](http://videolectures.net/course_information_theory_pattern_recognition/)

or the (sometimes too dense) Andrew Ng -
[https://www.coursera.org/course/ml](https://www.coursera.org/course/ml)

~~~
incision
_> 'Textbooks?'_

The article addresses this almost immediately.

It's fine to disagree, but crapping a _' Really?'_ plus some contextless links
onto someone who put forth general reasoning for the nature of the
recommendations and spent 1300+ words describing expectations, key takeaways
and projects for those recommendations is just lame.

~~~
orasis
On a given topic, I believe the best textbook is an inferior medium to the
best lecture. Rather than blathering for 1300 words, I provided links to some
excellent machine learning lectures.

~~~
beejiu
You sound like you've had some bad experiences with books. If you learn better
with lectures, great. However, it's not for everybody -- I personally think
spending time away from the computer (until I'm programming something), with a
book, paper and a pen is very good time spent.

~~~
Houshalter
I read textbooks on my computer actually. Not by choice though, textbooks are
difficult to find and too expensive.

I used to hate lectures when I was in school but now I sort of prefer them.
It's easier for some reason. It's more passive; you just sit there and listen
rather than actively read. It doesn't seem to be slower like others complain,
and may even be faster. I read difficult texts very slowly and methodically,
and often have to reread stuff.

