
Practical Deep Learning for Coders - jph00
http://course.fast.ai
======
OmarIsmail
I've been coding for 20 years and professionally for almost 15 and I started
watching the first video and found it to be pretty difficult to follow. I
think you make a pretty common mistake that a lot of technical people do,
which is take for granted how much "institutional" knowledge you have.

The topics touched on in the first 30 minutes of the video include: AWS,
Jupityr Notebooks, Neural Networks, Tmux, and a few others. I understand that
this is the reality of the situation today (very large up front cost of
setting everything up) but it would be better to not even touch upon something
like Tmux because it's not absolutely essential and just results in
information overload. You can replace it with something like "I like using
Tmux to save my terminal sessions, check out the wiki to learn more" instead
of "here's this 3 minute tutorial on Tmux in the middle of how to use Jupyter
notebooks". Very few people are smart enough to be concentrating on following
what's going on with AWS/Jupyter notebooks and then pause that, process the
Tmux stuff, and then context switch back to AWS/Jupyter.

There's a reason why the wiki/forums are so invaluable. There's definitely
some really good information in the videos so if you guys had the time I
really hope you edit the videos to an "abridged" versions that focus on only 1
topic instead of jumping around so much.

~~~
jph00
That's a great point - but I would like to add that the expectation is that
people put in at least 10 hours a week. So the amount of material is designed
to be what you can learn in 10 hours of study, with the support of the wiki
and forums.

The lessons are not designed to stand alone, or to be watched just once
straight through.

The details around setting up an effective environment are a key part of the
course, not a diversion. We really want to show all the pieces of the applied
deep learning puzzle.

In a more standard MOOC setting this would have been many more lessons , each
one much shorter - but we felt the upside of having the live experience with
people asking questions was worth the compromise.

I hope you stick with it and if you do I'd be interested to here if you still
feel the same way at the end. We're doing another course next year so feedback
is important. On the whole most our in-person students ended up liking the
approach, although most found it more intensive than they expected. (We did
regular anonymous surveys. )

~~~
dhconnelly
Awesome stuff, but please let's not pretend that tmux is an important piece of
the applied deep learning puzzle that needs to have anything to do with the
intensiveness of the course.

~~~
jph00
I apologise if that's how my response came across. what I was trying to
express is that I wish to show what I believe are best practices, and these
tools are used throughout the course. I do not mean to suggest that tmux is
critical, or that there are not other good options.

Personally I enjoy seeing how others setup and use their tools, and feel it is
underappreciated in many courses; attempting to fill that gap is a little of
my particular bias.

~~~
annnnd
> Personally I enjoy seeing how others setup and use their tools, and feel it
> is underappreciated in many courses

Interesting - that explains a lot, thanks. Sorry to say, but I for one _hate_
courses / tutorials where the tutor tries to force me to use his/her pet
technology, totally unrelated to the topic at hand. I have always wondered
what made them do it, and I guess they just want to help.

Still, if you want to be helpful, show me this favorite tech of yours and
optionally point me to a separate resource for learning it, I might check it
out sometime. But I didn't come here to learn tmux, I came for deep learning.
My time is precious so please don't waste it. I would understand if you wanted
to teach me basics of TensorFlow (and TensorBoard) or similar, but tmux, vi,
emacs, bash,... are all outside the scope. IMHO of course. :)

------
Simorgh
This is really stunning. I can't wait to commence to course. I finished a
Masters from a top-50 worldwide university, and frankly, the approach to data
science was mediocre at best. The NLP module notes were plagiarised from
Stanford and we were quite happy with this! It gave us a break from 20 year
old textbooks that set the plodding pace for the Data Mining module. And don't
get me started on my deep learning dissertation. The only expert in the uni on
the topic got poached by facebook halfway through the project. The
universities are finding it difficult to keep up and are resorting to
'interesting' techniques to retain talent - witness the Turing Institute in
the UK. They gave out titles to many professors in several universities a year
or so ago... as I gather, as a precursor to pivotal data science research.

~~~
laughingman1234
I applied for Masters in ML , in one of these univ ,

CMU, UT Austin, Georgia Tech, UCSD..

I am not in US, thought getting MS in one of these would boost chances of
getting into something like google brain, or Open AI..

Is it waste of time and money in your opinion?

~~~
laughingman1234
If someone thought ^ this was sarcastic, sorry, I was just genuinely wondering
wether the exposure to top research in grad schools is worth investing

~~~
master_yoda_1
MS won't be so fruitful unless you do some research and publish (which is
difficult and also many university don't support giving research work to MS
student).

Its better to go for Ph.D. Remember for going into openai or google brain you
need to be among top even after Ph.D.

~~~
Smerity
I'll note that my MS was hugely useful and didn't result in publications
directly. The mileage of your MS or PhD is dependent on many factors.

OpenAI and Google Brain, like most other more research driven deep learning
institutions, are more interested in the results you can produce rather than
the accreditation you hold. Publications obviously count but well used or
written deep learning projects / packages would too. Many PhDs who come out
having spent many years in academia still wouldn't get an offer from these
places and many of the talented people I know in these places don't have a PhD
either.

To the parent of this post, I'd also look into what I'd refer to as "Masters
in industry" i.e. Google Brain Residency[1] and other similar opportunities.
From their page, "The residency program is similar to spending a year in a
Master's or Ph.D. program in deep learning. Residents are expected to read
papers, work on research projects, and publish their work in top tier venues.
By the end of the program, residents are expected to gain significant research
experience in deep learning.". This is likely an even more direct path than
most institutions would provide. Though obviously the competition is fierce,
many of my friends who participated in this ended up with a paper in a top
tier conference by the end of the.

[1]:
[https://www.google.com/about/careers/search#!t=jo&jid=147545...](https://www.google.com/about/careers/search#!t=jo&jid=147545001&)

------
jph00
Hi all, Jeremy Howard here. I'm the instructor for this MOOC - feel free to
ask any questions here.

~~~
kirkules
Just FYI: The AWS console UI is currently different in a number of ways from
the interface shown in the "how to set up your aws deep learning server"
video, beyond what is accounted for in the overlay-text amendments to the
video. (e.g. creating a user has 4 involved steps before you get to see the
security credentials, including setting up a user group and access policies
for that group -- I made my user have the AdministratorAccess policy...)

~~~
loco5niner
* I would amend the above comment to say that setting up a user group is not required (although it is not exactly clear that it is not).

------
ddlutz
Thanks for announcing this, looks amazing! As somebody that's been toying
around with deep learning and machine learning, I've been wondering what the
steps are to move from 'cool example' to viable product. I know somebody else
mentioned something general like that in another comment, but I had a concrete
example.

For instance, it's extremely easy to set up an MNIST clone and achieve almost
world-record performance for single character recognition with a simple CNN.
But how do you expand that to a real example, for instance to do license plate
OCR? Or receipt OCR? Do you have to do two models, 1 to perform localization
(detecting license plates or individual items in a receipt) and then a second
model which can perform OCR from the regions detected from the first model? Or
are these usually done with a single model that can do it all?

I'm not sure if answering these questions is a goal of your course, or if
they're perhaps naive questions to begin with.

~~~
jph00
They are excellent questions and that indeed is exactly the goal of this
course. I hope you try it out and let us know if you find the answers you
need.

For this particular question, a model that does localization and then
integrated classification is called an "attentional model". It's an area of
much active research. If your images aren't too big, or the thing you're
looking for isn't too small in the image, you probably won't need to worry
about it.

And if you do need to worry about it, then it can be done very easily - lesson
7 shows two methods to do localization, and you can just do a 2nd pass on the
cropped images manually. For a great step by step explanation, see the winners
of the Kaggle Right Whale competition:
[http://blog.kaggle.com/2016/01/29/noaa-right-whale-
recogniti...](http://blog.kaggle.com/2016/01/29/noaa-right-whale-recognition-
winners-interview-1st-place-deepsense-io/)

(There are more sophisticated integrated techniques, such as the one used by
Google for viewing street view house numbers. But you should only consider
that if you've tried the simple approaches and found they don't work.)

------
thrawy45678
I see a lot of criticism about tmux and other non-core items being included in
the overall curriculum. I think the author is trying to portray the workflow
he is currently on and exposing the full tool kit he uses. I don't think he is
saying - this is "THE" approach one has to follow. I for one think that this
is a perfectly legitimate way of teaching. People can leave out pieces which
do not interest them or substitute them with other tool sets if they chose to.

For me the key take away here is that - some one who has been a consistent
beater in the Kaggle competition for two years and the founder of a ML company
is teaching a "hands-on" course which fills a gap (from tech talk to step-by-
step hands on) and I think I can live with this method of teaching.

------
pkmital
Just going to through this out there, but this course is also geared for
beginners and has been loads of fun to create:
[https://www.kadenze.com/courses/creative-applications-of-
dee...](https://www.kadenze.com/courses/creative-applications-of-deep-
learning-with-tensorflow/info)

------
laughingman1234
How does this compare to udacity's deep learning course?

Or should I take a more theoretical course such as Andrew Ng's to get into ML?

~~~
tom_wilde
If you're getting started I totally recommend Andrew NGs course...

~~~
jph00
Strongly agree. Andrew's course is a great choice to take before this one, if
you have the time. It's not a prerequisite however.

The Udacity course has a very different aim - it covers much less territory
and takes much less time. If you're just wanting to get a taste of deep
learning, it's a very good option, but it's not a great platform to build
longer term capability on IMHO.

------
cr0sh
I'm currently taking the Udacity Self-Programming Car Engineer nanodegree; I'm
currently working on the lab right before the 2nd project lesson (I'm in the
November cohort). That lab is to re-create the LeNET 5 CNN in TensorFlow (we
have to re-create the layers of the convnet).

Last night I spent an hour or so getting my system (Ubuntu 14.04 LTS) set up
to use CUDA and cudnn with Python 3; setting up the drivers and everything for
TensorFlow under Anaconda - for my GTX-750ti.

That wasn't really straightforward, but I ultimately got it working. It
probably isn't needed for the course, but it was fun to learn how to do it.

I would like to take this fast.ai course as well, but so far the Udacity one
is eating all of my (free) time. Maybe I can give it a shot in the future.

~~~
mmcclellan
I have used and recommend [nvidia-docker]([https://github.com/NVIDIA/nvidia-
docker](https://github.com/NVIDIA/nvidia-docker)) for exploration (at least
for those with a docker-capable kernel as the base OS).

------
mikealche
Hey, so I know that 0.90 isn't much... But is there a difference between
running the programs on AWS than on my local computer with my mediocre video
card? Will they take much longer to trian? I honestly don't know the
difference so I'm asking

~~~
slashcom
Your mediocre graphics card may or may not have CUDA support. If it doesn't,
or it only supports an old version, then it's the same as not having a
graphics card at all. And of course, Nvidia only.

If it supported, then the major difference is GPU memory, which limits the
size of the network you can train. The newest models are faster than some 1-2
year old ones, but older hardware does the job fine.

~~~
jph00
In addition to this excellent advice, I'll just add that my experience is that
a 2GB CUDA compatible card is (just) enough to complete this course. Although
4GB makes life _much_ easier

~~~
webmaven
Any recommendations for the cheapest (but not "penny wise, pound foolish") HW
setup that meets these requirements for course completion?

~~~
lonesword
This is what I have and it's been holding up great for everything I've wanted
to do till now. Core i3 6100, Nvidia GTX 950 2GB, 8GB DDR4 RAM. Did not try
running a neural net on the GPU, but it should do the job. The whole thing -
[http://lonesword.in/hardware/2015/11/10/Assembled-a-
computer...](http://lonesword.in/hardware/2015/11/10/Assembled-a-
computer.html) \- including the cheap table and chair cost me around 60K
indian rupees which roughly translates to 1000 USD - but that's because
computer components are around 30-40% more expensive in India than in the US.

~~~
tmbsundar
In same boat here. Early this year I put together a PC with GTX 750 TI 2GB.

What do you think of this? Will this work fine?

~~~
cr0sh
I just set up CUDA and cudnn for python3 and tensorflow last night on my
machine - with that same card. That card should have 640 cuda cores; every
little bit counts, imho. I'm not really sure what the difference will be
compared to what I was doing before, but anything has to be better than the
quad-core cpu I'm currently using (some older AMD thing).

------
cicero19
Hi Jeremy, looking forward to going through these. Thanks for sharing. Very
noble decision to make them free.

I am wondering, why you chose to leave Enlitic and start fast.ai?

~~~
jph00
Probably too long a story for an HN comment! ;)

In short, my wife got sick and needed brain surgery while she was pregnant,
and I ended up being away from Enlitic for nearly a year. It made me reassess
what I really wanted to do with my time.

Now that I spend all my time coding and teaching, I'm much happier. And I
think that making deep learning more accessible for all will be more impactful
than working with just one company. Deep learning has been ridiculously
exclusive up until now, on the whole, and very few people are researching in
the areas that I think matter the most.

Finally, I think I achieved what I set out to do with Enlitic - deep learning
for medicine is now recognized as having great potential and lots of folks are
working on it.

------
codingdave
Thanks for sharing this -- I've been doing hobby-level work with computer
vision on the side for a couple years now, but always kinda hit a wall when
moving beyond anything trivial. I'll give this a shot and see where it takes
me!

~~~
jph00
Yeah I know just what you mean. Check out the feedback from Christopher Kelly
here, who describes something very similar:
[http://course.fast.ai/testimonials.html](http://course.fast.ai/testimonials.html)
. Perhaps you'll find some inspiration there...

I really hope that you get past the wall! If you do find yourself getting
stuck, demotivated, etc, please do come join the community on the forums,
since they can really help overcome any issues you have:
[http://forums.fast.ai/](http://forums.fast.ai/)

------
flik
>>you need access to a computer with an Nvidia GPU, along with a python-based
deep learning stack set up on it.

Is NVidia GPU a must have... can I workaround?

~~~
jph00
Yes, you can get by with an AMD GPU but it'll be a lot more work, and a lot
more frustration.

It's worth spending the $0.90 per hour to use AWS, or less if you get spot
instances.

You can do much of your prototyping on just a CPU, and only run on the GPU on
more data once it's working well.

------
girfan
This is absolutely amazing. Thanks for the effort. How do we join the slack
group for interactive help/questions?

~~~
jph00
Our preference is for most questions to go through the forums. When you join
the forums you'll get more information on why we think that's best for the
overall community (in short, because it's easier for others to find answers
when they are organized by topic), and also how to access the Slack channel.

------
kp25
Can i use google compute engine instead of aws ?

~~~
jph00
Last time I checked they didn't have GPU virtual machines. So no, probably
not. If there is a way to run jupyter notebooks on GPU machines on Google, I'd
certainly be interested in learning about it.

~~~
andresgottlieb
They announced they'll start offering GPU instances in early 2017:

[https://cloudplatform.googleblog.com/2016/11/announcing-
GPUs...](https://cloudplatform.googleblog.com/2016/11/announcing-GPUs-for-
Google-Cloud-Platform.html)

------
bogomipz
This is great! Thanks for sharing!

