I'm a intermediate-level programmer, and would like to dip my toes in AI, starting with the simple stuff (linear regression, etc) and progressing to neural networks and the like. What's the best online way to get started?
I'm in the same boat. For long time, I was interested in AI but at the same time intimidated by math. I'm relatively comfortable with discrete mathematics and classical algorithms and at the same time calculus and linear algebra is completely foreign to me. Also, I do not accept way to learn ML without good understanding of core principles behind it. So math is a must.
A few months ago, I stumbled upon very amazing YouTube Channel 3Blue1Brown which explains math in very accessible way and at the same time I got feeling that I finally started understanding core ideas behind linear algebra and calculus.
Just recently he published 4 videos about deep neural networks:
Even if you think you grok matrices, have a go at the first few videos of that playlist, if just for the visualization. It really helped me see what matrices (and operations on matrices) represent!
3Blue1Brown is a treasure. The production value is excellent, and he's great at taking seemingly uninteresting ideas and painting a beautiful picture to connect them in twenty minutes. I used to go through a video before falling asleep each night.
Having watched the third one out of sequence, seeing the first two and then watching the third again helped me get a good understanding of the fundamentals. 3blue1brown as a narrator does a excellent job of allowing a rather tricky subject be more approachable, and inspired me to buy a course to allow a deeper dive into the math behind ML+NNs.
Nice to some one pointing the fundamentals. A good understanding about probabilist models is good too. After getting into too the basic math knowledge, I suggest this: https://classroom.udacity.com/courses/ud730
Classification and regression. Given examples, predict labels or values for new data. SVM used to be more «hot» than neural nets and are still very useful.
I highly recommend Andrew Ng's Coursera courses for both Machine Learning and Deep Learning. Good for beginners, Math is taught along with the course, and gets you a solid foundation:
Andrew Ng's courses are excellent. Another pretty good Coursera course is Machine Learning Foundations from the University of Washington. It is very high level and novice friendly. While it covers non of the math and very little programming it does give a nice quick introduction to the most popular ML techniques out there and when to use them. It all depends on what level you are interested in starting. They also have follow up courses that go deeper into the different techniques.
I would specifically recommend Machine Learning Foundations: A Case Study Approach - It is fantastic and helped me greatly start my ML journey last year.
Turi is awesome, I hope Apple is doing something great with it.
From my experience Andrew Ng wiped the floor with every other lecturer I've had. Both the ML and his new Deep Learning course.
If the lecturers aren't very interesting Coursera can be as hard as any other lectures. I gave up on the Scala functional programming and disappointingly have stalled with Geoffrey Hinton's Neural Networks courses.
But I really can't understate how good Andrew Ng is, he has a very relaxed manner and manages to make some very complex topics seem almost trivial.
The worst of the mathematics is derivatives and matrix multiplication. You can even avoid matrix multiplication mostly in the ML course, but in his Deep Learning course he takes you through the 300x performance benefit you get from using NumPy and matrix multiplication vs loops.
At your level yes, I would recommend starting with the ML course. It is really beneficial to understanding how the mathematics work.
The two most important things to remember, since the courses are challenging: 1) don't be in a hurry, and 2) don't give up! Take the time to learn every detail presented, do the optional exercises, and dig deep.
It's definitely challenging. The math and just seeing the complicated formulas really push me, but the reward is good too. I'm tired of pushing pixels and doing some meaty stuffy like ML is a nice change of pace.
I would recommend starting with deep learning first since that's what you are interested in and it covers all the ML principles you need to be familiar with. If you want to go deeper and get familiar with other ML techniques too you can easily follow the old course afterwards.
Its specifically geared towards visual recognition, but it starts with the basics of machine learning and moves on to feed forward nets and covnets and covers RNNs and attention towards the end.
The assignments are a great set of jupyter notebooks that really get your hands on the material and you can find a number of peoples complete assignments on github just by searching.
I've done hinton's and Ngs courses and as someone who already has a non-ai development background I found this to be the best introduction. Its really an extension of Andrej Karpathy's Neural Nets for Hackers (http://karpathy.github.io/neuralnets/)
It's really important not to skip the math. As a friend once said to me, doing deep learning without understanding the math is like gambling. It's fine to initially take a more practical, project-based approach for the sake of staying motivated, and you'll retain things better if you have project goals in mind, but, the math is that important.
The good news is that compared to other technical fields, the math is also relatively shallow. Here are some good resources that you don't need more than calculus/linalg for (I've used all of them and they got me off the ground):
Once you feel confident, the Deep Learning book is more math-heavy, but it is really very good. The authors are more or less deep learning gods. It'll teach you a tremendous amount about how/why neural nets work and the principles used to discover new architectures, and gain a strong intuition for how to use neural nets as a tool. Read it slowly---unless you're already good at math, it takes a while to get through. Don't skip the first five chapters. Use Google and Wikipedia to pick up concepts you don't understand along the way instead of skipping over them (it will bite you later).
It somewhat depends on if you are looking to build AI to address business problems or if you are more interested in the type of AI work you see companies like Google discussing.
I can speak to what "AI" means for most businesses outside Top Tech which more frequently work with tabular, relational, or log data rather than image and text. For these companies, this is what you need to learn how to do
1. Define a prediction problem and extract labels
2. Organize and clean the data for prediction
3. Perform feature engineering by applying domain expertise
4. Apply an off-the-shelf open source machine learning algorithm like a random forest
Assuming you have access to data and programming skills to clean your data, defining prediction problems and performing feature engineering are the most important skills you have to pick up. For machine learning you can you use open source libraries like scikit-learn or tensorflow.
At my company, we've noticed a lot of programmers are intimated by the feature engineering step in particular, so we tried to make it easier by creating an open source library called Featuretools [0].
it has Py2, Py3, R, F#, anaconda, TF, CNTK, etc. pre-installed.
There are some ML tutorials on it already + you can use the "load from github" feature to load, run, edit, ... many of the great tutorials already on github.
Other similar environments include colab by google and cocalc.
I'm involved with a startup that's specifically tackling this very problem -- how do you learn the theory & application of machine learning quickly (especially if you already know programming well). We teach using diagrams and interactive coding exercises in the browser: www.dataquest.io
1. Start with the fast.ai courses. It's an applied deep learning course using state of the art techniques.
2. For classical machine learning (regression, etc...), Andrew Ng's course on Coursera is widely considered "the basics"
3. As you progress, check out CS231 and CS224 from Stanford for state of the art image processing and natural language processing techniques. The lecture videos are on YouTube and the course assignments are available online. The third course I recommend is Geoffrey Hinton's neural networks course on Coursera (he is one of the most important researchers in the field).
4. If you're an application engineer, focus on using existing tooling to build cool projects. Keras and scikit-learn are great out of the box tools.
5. If you are more research oriented, you can start reading papers. In Silicon Valley, there's a meet up group that reads papers every Monday and tries to implement the algorithms in the paper. It takes a while to get to this level, but try not to get overwhelmed. Experts spend 7 years studying this stuff full time to get a PhD.
6. You really don't need much math to get started with ML. A high school understanding of calculus and some basic understanding of numerical optimization are the two main concepts you need to know. If you want to get into the research, there'll come a time when you will need more advanced math, but in my experience you can pick that up as you go along if you are curious.
Maybe you could start an AI study group online? The Silicon Valley study group was great, but I was just visiting.
If TensorFlow is what you're interested in I personally found "Hands-on Machine Learning with SciKit-Learn and TensorFlow by Aurélien Géron" to be the best introduction after introducing myself to the subject with Siraj's YouTube videos
I take a lot of different online courses, I have no affiliation with Udacity, but their courses are just too good.
I studied AI (focused on ML) in a decent grad school (and I like to think I had the best teachers there), and I think the quality of the courses is comparable.
Or is it an operant/classical conditioning sort of thing, where AI is specifically about training programs to act rather than to perceive/categorize things?
I suppose you can have AI that incorporates no ML (like most video game AI), but I'd imagine that will become vanishingly rare in the future.
In brief, AI uses existing knowledge and/or heuristics (to solve problems that lack a closed-form solution), while ML acquires knowledge and heuristics toward the same end, with the added goal of improving performance as it learns and adapting to changing conditions.
Traditionally, AI has been divided into distinct subfields (e.g. search, planning, natural language and speech processing, game playing, computer vision, robotics, knowledge representation, expert systems, logic, and ML). Today, ML is employed in all AI subfields, but until recently, most subject matter in each AI subfield had been unrelated to ML. In the past decade especially, that's changed as deep learning and probabilistic methods have gained mindshare and now are largely unavoidable when tackling AI-related problems.
In general, AI's subfields have focused on identifying fundamental obstacles and important features in their own problem domain and developing appropriate techniques that operate on those features when solving problems (like using object recognition and localization to solve vision problems like autonomous driving). I suspect AI's past emphasis on feature engineering has faded as NN-based ML has risen.
It would seem that many people have divergent definitions but I’ve always learned that AI was Artificial intelligence broadly construed (goal seeking, planning, rule based expertise, logic etc) while machine learning is the specific subset that is (roughly speaking) statistical (neural nets, regressions, svm and the like).
Don’t have strong feelings about these definitions.
In my opinion the best way to get started is first study statistical inference and modelling, in particular linear regression and the method of maximum likelihood. This will give you a critical eye later on for discerning when it's a good idea to actually use ML and when it's not (an important skill that apparently is in very short supply these days ;).
People like you are our primary audience :) it should take you exactly where you want to start and take you a good chunk of the way to where you wanna get.
Here's a recent video where he talks about how to create new Pokemon with Generative Adversarial Networks (https://en.wikipedia.org/wiki/Generative_adversarial_network).
Nice contrast from the usual MNIST dataset, especially if you want to be inspired to think about novel ways to apply this stuff:
I have two engineering degrees and studied neural networks in college. He cuts down about 3 months worth of math to a couple of lines of code in such a way that makes it make sense AND is productive. It may just be that his teaching style matches my learning style really well, but I'm enjoying going through it.
I am sitting in the same boat. Being a web developer for a couple of years, I wanted to try out a different domain. So I started to take Andrew Ng's course on Coursera [0]. Highly recommended. I supplement the course with audio and text by listening to the Machine Learning Guide Podcast [1] and by reading The Master Algorithm [2].
In addition, I started to apply my learnings in JavaScript [3]. Even though it's not the best language for ML, it makes it simpler to learn only one new thing and stick to known technologies for the rest. I have lined up ~7 articles about ML in JavaScript, so if you are interested, you can keep an eye on it :)
I got started being into algorithms and then making games with ai opponents like http://bitmultiplayerchess.com, http://wordsmashing.com
I took andrew ng’s coursera machine learning course which i would highly recommend, also his deep learning course is worth it too :)
His course inspired me to create a cryptocurrency trading bot which i spun into a business offering forecasting for altcoin markets: http://BitBank.nz - Crypto Market Predictions with Machine Learning
I managed to make much more successful forecasts by understanding the fundamentals taught in that course like under-fitting and over-fitting and how to visualize whats happening by plotting a learning curve ect.
The forecasting algorithm really just applies the fundamentals thoroughly in perhaps a novel way, e.g. some features we compute at the current time include the linear regression of trades over time weighted by their amount
So its definitely worth the investment i think :) try and apply the teaching to solve a real world problem which i think is the interesting part, although you’ll end up doing a lot of data engineering you’ll savor the AI/ML part even more and start to appreciate strategies for how you can improve your performance in your case and test them out.
Having a play around with the create your own deep neural net at playground.tensorflow.org is pretty helpful, try and conceptualize what youve been taught in the courses by playing around with that, e.g. add more layers/breadth to your network to watch it get more and more powerful and begin to overfit when you add noise to your data ect.
Just hijacking this question for my benefit as well. I am a product manager in enterprise focused software. I want to transition to the world of AI. Is Udacity's $600 Deep Learning Nano degree worth it ?
I don't know about Deep Learning Nanodegree, but I took Machine Learning and AI Nanodegrees at Udacity and they are definitely worth it. I would not recommend them to total beginners in the field. You need to have at least some experience with data science and Python to be able to follow along. Do some free courses on Udacity, Coursera, EdX and other platforms, try to implement these algorithms with your own data and problems and then take the Nanodegree to fill the gaps.
I was at the same point as you until I discovered the new Andrew Ng course on deep learning [1]
It's a good structured way to learn the core of ML while learning about Neural Networks and without having to become and linear algebra expert which for most people including like me was a deal breaker with other courses. The timing is great too as ML now is so much different than it was 2-3 years ago.
I think being effective in ML requires both theory, and practical knowledge you only get by doing.
Andrew Ng's ML course quickly provides a base in theory.
Ideally you couple that with some empirical work.
For that, I think sklearn is the best starting point (assuming you go down the python path). Modify some sample code and make a few simple models. Sklearn provides an excellent framework across all kinds of models (including deep learning if you use say keras.wrappers.scikit_learn), and can play well with pandas.
There are lots of practical concerns that come up that are not covered in intro ML courses.
As a follow up: I want to pursue a math degree study. What course titles and textbooks starting at the calculus level do you guys recommend? I want enough math chomps to then go onto a PhD in ML.
Linear Algebra (which is what you really need): Gilbert Strang - Linear Algebra and its applications
These two are all you need, with which you'll get a solid base. Then you're good to go on your own. These two combined are about 4 semesters worth of work. But if you really focus, I think you can get them done in a little less than 6 months.
If you want a 'just what I need' approach, Khan Academy.
For self-study, you don't necessarily need to follow the usual progression of math classes that start at calculus. It's more important to get comfortable with linear algebra than calculus, especially the way a lot of intro calculus courses focus on calculating integrals and derivatives. Maybe it's not the best message, but the worst grades in my math degree were in the intro calculus classes.
I don't remember what intro linear algebra books I used, but my college uses this: https://www.math.ucdavis.edu/~linear/ (I took the class before this free textbook was developed).
Possibly not what you're looking for (certainly not the cheapest option), but we (Lambda School - YC S17) just announced a live, remote class that trains engineers in AI & ML during weekday evenings for six months.
The next one starts in January, and is taught by an MIT grad that taught a similar course at MIT.
After you've got a grasp of what these things are doing then you can move into the how. For that you will need some math background, with emphasis in calculus and probability.
I liked the tutorials at Python Programming, sometimes the python details goes a bit fast and there are typos but over all it's the one that got me most understanding the practical parts.
I know you say you'd like to learn online, but I highly recommend picking up Duda and Hart's Pattern Classification to have a theoretical complement to the "hands on", programming type introductions. It's a very accessible intro to the topic, but also covers a lot of material in depth -- in particular, the topics you mention.
I'm currently working through "Deep Learning: A Practitioner's Approach" by Adam Gibson and Josh Patterson. Its a couple of years old but seems like a good book, and I'm certainly learning a lot. It doesn't consider some of the newer tooling, like TensorFlow, but the fundamentals plus a decent amount of theory are all covered.
What makes you say the book is 2 years old? Its publication date is Aug 1, 2017. It's based on Deeplearning4J and ND4J rather than TF, but that doesn't date it necessarily.
You're right, it is August 2017. I bought it from Amazon UK and they list the date as November 2015 [1] for some reason. I should have checked the book before posting. I've flagged the error with Amazon UK.
I can't speak to the rest of the content, but I found the introduction in the course accompanying the recently announced gluon library to be both comprehensive and comprehensible at the same time.
If you’re genuinely a novice programmer/lesser background in linear algebra, AI should be the last thing on your mind. Any attempts at a shortcut will enhance the difficulty in learning AI, and being able to code things besides simple examples. (which is why I am annoyed by many of the ML MOOCs which are targeted toward novice programmers)
I disagree. There is nothing magic or hard about basic ML. You can do real work with only some basic linear algebra and programming skills. Sure you won't be doing novel deep learning on 100 terabyte datasets, but most problems aren't that anyway.
Yeah, and to be honest while it's always important to be able to understand the math behind what you're doing, you can easily get started with just understanding what these algorithms do and why, and then work to expand your knowledge of the math over time from there.
Why does this annoy you? Some of the MOOCs are amazing and they certainly don't shortcut. I had many hard-learned lessons and realizations when going through the MOOCs I have taken. They also help in "knowing what you don't know"...which is essential in breaking out of ignorance.
I've learned computer science in college, so I do have _some_ linear algebra background. I'm by no means an expert coder, but maybe 'intermediary' would have been a better description than 'novice'. :-)
I say it for the opposite reason of elitism (and I’ve spoken out many time against elitism in ML/AI).
As other posts note, there are many resources available for teaching the concepts. But they don’t teach the limits of AI, and the rise of MOOCs is setting novice programmers up for a shock when they encounter real world data that is not as nice as the Titanic dataset, and requires making smart decisions to handle the data, and handle it in a way that does not invalidate the results.
Many romanticize ML/AI as something that can solve any problem, which is a dangerous approach.
It teaches you the foundational theory behind ML, and shows how the fancier stuff is built on it. Good to know the foundations, so you can branch outside of predefined ML techniques.
A few months ago, I stumbled upon very amazing YouTube Channel 3Blue1Brown which explains math in very accessible way and at the same time I got feeling that I finally started understanding core ideas behind linear algebra and calculus.
Just recently he published 4 videos about deep neural networks:
https://www.youtube.com/watch?v=aircAruvnKk
https://www.youtube.com/watch?v=IHZwWFHWa-w
https://www.youtube.com/watch?v=Ilg3gGewQ5U
https://www.youtube.com/watch?v=tIeHLnjs5U8
So my fear of ML was gone away and I'm very excited to explore whole new world for neural networks and other things like support vector machines etc