You need to have a basic understanding of matrix products (from linear algebra) and the chain rule (from calculus). Both can be learnt on demand using Khan Academy - no need to go study math first, but just study what you need when you need it. We give brief explanations of both when they come up.
The wiki (http://wiki.fast.ai) has links to necessary learning resources for each lesson.
Deep learning is pretty much entirely linear algebra and calculus, at a mathematical level. But I really think that a code-first approach is much better. So I'd suggest just getting started working through the course, and really dive into the Jupyter notebooks for each lesson. Do lots of experiments.
>"But I really think that a code-first approach is much better."
This is really refreshing to hear Most of the books I have looked at seem to be mostly math-first and very densely so at that. As much as I appreciate math it's not always the warmest welcome.
I am looking forward to working through your course and seeing part 2 as well. Cheers.
If you want a solid understanding of DL⊂ML, you'll also need a good understanding of probability theory and information theory. While the mechanics require just matrix math and some vector calc, you'll also want to know linear algebra proper and these days, function spaces especially. A small helping of analysis, for measure theory, will not hurt either.
Interesting that you mentioned Information Theory, when I think of this I think of Shannon, encoding symbols, bauds, bit, noise and channels in the Telecom network sense.
But I guess this all applies quite similarly in the neural network sense in DL/ML as well?
There's a 1:1 relationship between likelihood functions and entropy, so a background in information theory is likely to make your understanding of loss functions come quicker.
First of all, let me say that what you all have put together is truly excellent. It covers the sort of things toy treatments and papers leave out or gloss over if you actually want to get something working in the real world.
That said, I strongly disagree with your disagreement. There was a recent paper whose abstract I read that made me think of homotopies between convolutional networks. Unfortunately I lost the paper behind some stream and never got to read it proper. In the context of that, I realized that the search for convnet design will likely soon be highly automatable, obsoleting much of the work that many DLers are doing now.
What will be future proof is understanding information theory so that loss functions become less magical. Information theory is needed to understand what aspects of the current approaches to Reinforcement learning are likely dead ends (typically to do with getting a good exploration strategy, also related to creativity). Concentration of measure is vital to understanding so many properties we find in optimization, dimensionality reduction and learning. Understanding learning stability and ideal properties of a learner/convergence means being comfortable with concepts like Jacobians and semi-positive definiteness for a start.
Probability theory is needed for the newer variational methods, whether in the context of autoencoders or a library like Edward (whose like I think is the future). Functionals and the variational calculus is becoming more important, in both deep learning and for understanding the brain. There's lots of work in game theory of dynamical systems (think evolutionary game theory) that can help contextualize GANs as a special case of a broader category of strategies.
Much to the contrary, the topics I mentioned are both the future of deep learning and future proof in general. This blog post by Ferenc captures my sentiment on the matter: http://www.inference.vc/deep-learning-is-easy/
I agree with all you say here. And indeed for researchers interested in building the next stage of deep learning, such topics may be important.
However this course is about teaching how to use deep learning today to solve applied problems. Most participants in the in-person course worked in business, medicine, or life sciences. It is to that audience that my comment was directed.
So we're perhaps each guilty of assuming the person asking the question wishes to study deep learning in the way we are focused on ourselves. :) Hopefully between the two of us we have now answered the question more fully than either of us individually!...
Yes, I certainly do think it possible. But it will be more difficult and slower going. Read widely, as the biggest hurdle for a self-learner will be in identifying and filling their gaps of knowledge. Read papers so you learn how to use language such that you are not setting off any crank or dangerous dabbler alarms. Reading lots is also necessary to habituate yourself to the jargon. Read the books by David MacKay and Christopher Bishop. Don't waste time if you're stuck on papers, it means either you're lacking either the shared knowledge or the authors are being purposely obtuse. Save anything you think is the prior for later (unless is a minor iteration of something, which most are). It might take multiple readings spread out over a large time span before you can truly understand the important concepts and papers.
If you can, try and organize or join study groups where the levels of skill are varying. Having a group will help in those times when it seems all too much and you're ready to quit.
Finally, don't try to compete with the well heeled industry titans and their GPU Factory Farms. Find an understudied but important niche where your lack of knowledge is not so much a setback because even if available mental tools will differ, everyone is equally ignorant on the terrain.
The wiki (http://wiki.fast.ai) has links to necessary learning resources for each lesson.