
Job placement program for top students in Stanford's online AI class - mlacitation
http://pastebin.com/JiczaBxb
======
WildUtah
Well, I haven't received any such email and my score in the class is 98% so
far (with only the final exam pending this week).

I expected that around 10,000 people would have a perfect score in the class
given my experience with Asian participation in online programming contests.
There are a few countries in that part of the world where organizing a group
to check each others' work and ensuring perfect scores is common and
encouraged even when the official requirement is to do your own work. And
those happen to be countries filled with millions of smart people. India,
China, and Russia seem to be the heart of the phenomenon. Google, Facebook,
and Topcoder have systems to deal with it but ai-class.org does not.

Anyway I'm neither surprised nor disappointed not to be in the top sliver. I'm
not hireable anyway. It's funny that there is an identifiable top 1,000 at all
instead of 10k+ perfect scores.

~~~
learc83
I got the letter, I made a 100% on everything except homework 6 and 7 (missed
1 each on those).

~~~
YokYok
I made only one mistake (98% in homework 6) and haven't got the letter.

~~~
zmj
Have you done all the other homeworks? I also missed only 1 question on HW6
and I did get the letter.

~~~
YokYok
I did them all and got 100% everywhere (except in HW6). I am kind of
disappointed. May be the quizzes are part of the grading?

~~~
wizzard
I think this has to be the case. My friend has a lower HW and midterm average
than me, but he got the letter and I did not.

------
sriram_sun
Getting new jobs is great. However, stay put in your jobs. Try to apply what
you learned to little problems at work. Everyone has a huge database these
days. I work for a medical device company and I write software that processes
blood (embedded stuff). Every year, a million or so run logs find their way
into one of our databases. There is a wealth of information there that we
could figure out. Eg. How do me maximize yield, what's the best we can do with
a certain type of donor. Do we perform worse on people in a certain geographic
region etc. Eventually make useful predictions based on that data. Granted,
that is not what I was hired to do, but I'm going to do it in my "free" time.

Why? Because it is interesting. Find something to apply to that might add
value to the business. If a enough people do that, AI techniques might become
common place. Get on Kaggle, GitHub your next project etc. Putting what we've
learned to practical use is how we can proliferate and disseminate this
knowledge. This was the intent of the class in the first place anyway.

tl;dr Using this course as a gateway to your next job is shortsighted IMO.

~~~
nosignal
I wish I could +1 this more than once. This is exactly the point, to me.
There's not really such a thing as "an AI job". There is application of AI
techniques to existing problems. Those problems are already found riddled
throughout every job everyone's ever worked in.

You can't sit back and wait for AI to "happen". What would that even look
like? The benefit of these techniques is in application to existing data; you
don't have to quit and found some kind of computer vision/kinect/big data
startup to take advantage of the knowledge.

You should be aiming to put yourself _out_ of a job by using AI.

------
tzs
Drat. If I'd known they were going to do that, I'd have made sure I was in the
top 1000.

edit: why is this getting voted down? I would expect there are a fair number
of people here who took the course with a casual approach, like I did. E.g., I
usually watched the lectures early Sunday, then did the homework that night.
Then, for those topics that I enjoyed and wanted in more depth, I read the
sections in the book on them. This approach is relaxing and fun, but does
increase the tendency to make silly mistakes and not catch them (which I in
fact have done a few times now).

I am, it turns out, enjoying the material enough that I think I would like, if
I ever find myself looking for a job, to find one where I can use this stuff.

~~~
buff-a
My guess is that you are being modded down because your statement is of a form
that many people have heard in real life and associate with poor performance.
It shows a lack of thinking about jobs. However, I'm going to give you the
benefit of the doubt and assume that you are young and not really worrying
about such things.

Basically your answer is like the classic "I could have got a 5.0, but I
partied", which is a statement that says a number of things. 1. "I think all
the kids who got 5.0 are boring and don't party" (untrue). 2. "I am not
capable of getting a 5.0 _and_ partying" (probably true).

That the person would even make the statement says something about their
general level of intelligence. Hence when you say "If I had but known, I would
have been in the top 100", you are saying "I had no idea that a hugely
publicized course at a renowned university might lead to job opportunities".
This lack of intelligence seems to contradict your first statement. Or it
could be that you are not in the habit of thinking about such things. I hope
you are either a kid, or an incredibly wealthy trust-fund baby living in
isolation. Usually, however, I hear such remarks from people who are not smart
enough to realize they are not smart enough, or they are quitters, or they are
victims.

~~~
tzs
Actually, the modding has reversed and my comment is now at +9.

The "partying" I was doing instead of putting a non-casual effort into the
class was dealing with some projects at work, and starting to read a book on
analytic number theory because I'm tired of being someone who doesn't
understand the proof of the prime number theorem.

> you are saying "I had no idea that a hugely publicized course at a renowned
> university might lead to job opportunities"

Yeah, I did not think it would lead to much in the way of job opportunities.
That's because the class is just an introduction. If you compare almost any
section of the class to the corresponding section in the Russell and Norvig
book, you'll see that they are leaving out a lot of detail in the class. The
homework in the class almost never pushes any limits of what was done in the
lectures. Many of the homework problems are just things that were done in
lecture with the numbers changed.

I would have expected the job opportunities to start at a deeper level than
the class reaches.

~~~
buff-a
_I would have expected the job opportunities to start at a deeper level than
the class reaches._

Aptitude and interest goes a long way.

------
rfrey
Sadly, I'm not expecting any such letter from Ng's ML class, because I'll bet
there's 10k people tied for top marks.

That's at least partly due to an extremely effective teaching system and
style, so I'd say it's a worthwhile tradeoff.

~~~
rglullis
I have only the best possible things to say about the video lectures, but the
lab assignments are just too easy. For something that is supposed to be the
advanced track, I was hoping that they would do something more than "take this
description of the function, implement it in Octave, make it pass our tests".

Of course, I know it's their first class and you can't get everything perfect
right off the bat. But I'd love to see the lab assignments being one larger
project, some kind of task where you need to apply the principles learned to
solve different problems. It would be also more effective in measuring/grading
students.

On an unrelated note: I wonder what is the license for the classes/videos. I
am working now in an adaptive e-learning platform. It would be cool to put
their videos and questions in our system and see how effective the system is.

~~~
jorgeortiz85
I agree the programming assignments have been very easy.

However, I doubt they have the bandwidth to manually grade thousands of
assignments. Given the scale of the class, the grading is forces to be
automated. It's hard to come up with such assignments and even harder to make
them challenging, given an automated grading script.

~~~
rglullis
The grading doesn't need to be manual. Think of the lab assignments being
something like a smaller scale of the Netflix challenge:

    
    
      1) They provide some set of data and establish rules for the competition.
      2) They implement their own solution to the challenge, and that is the benchmark.
      3) A "passing grade" is obtained by getting any working system.
      4) The actual grading is then given on a curve, compared against their benchmark.
    
    

If your project is better than the benchmark, you get an A+, 95%-100% an A,
85%-95% gets you a B... etc.

~~~
Natsu
Granted it wouldn't work in many other classes, but my teacher for assembly
language did something like this. First, your code had to work or you got
nothing. You also had some time limit, to avoid ridiculously slow, yet working
code. Finally, each working submission was graded by the number of additional
bytes you used above the reference implementation.

And he knew _all_ the tricks. I don't think anyone ever beat him. And he
didn't show anyone any of the solutions until after the final.

I felt like I learned more from the few minutes I spent reading those
solutions than I did during the rest of the course.

~~~
zmj
Lasher at NCSU?

~~~
Natsu
Lance at ASU, actually.

------
robryan
As someone doing the course i would say its a decent way to identify some
candidates but it wont be a really high correlation. It depends a lot on the
time you have to invest and how rigorously you go through your answers and
study the material the questions are drawn from. Some programming challenges
would be great to better separate whether people really understood the
material despite being harder to mark.

Personally I've always got a bit wrong in most of he homeworks but usually
felt that while I understood the material I didn't recite enough time to
rigorously check my answers and rewatch the materials to pick up on some of
the subtle things you need to remember or some questions.

~~~
Homunculiheaded
I took this course casually as well (wanted to fill in some gaps in my
undergrad education). But my guess is that people that complete this course
casual probably already have a lot on their plates. For me it was family,
working, teaching and taking a grad class. I never double checked my hws,
could only do a single pass through the lectures, and didn't spend any time
checking for nuance in many of the Qs and my grade shows accordingly. However
in the end I'm really excited that I was able to continue through it despite
all this (I'm really curious to see the percentage of participants who saw the
course through completion compared to those who signed up).

The correlation will be interesting. Undoubtedly the top 1000 is made of many
of the type of people that must be the best at whatever they try (not
exclusively of course) and with a decent amount of time on their hands, after
all there is no disincentive whatsoever for not completing something and no
previously known reward for perfection (I guess a piece of paper can count).
Given some wording in some of the Qs (actually my only complaint about this
course) a lot of care must be taken to ensure that every question is right, so
being in the top 1000 is no small effort.

~~~
jeremyarussell
Same here in regards to time, family, etc. I'm hoping they keep the videos up
for some time after the last week, I missed a few during the more hectic
times. Regardless I've learned tons that I'm already implementing in code (and
daily life, like actual planning methods and such.)

That said I have a job I like already, and plan on using what I learned here.

Those 1000 with more time and drive to be the best probably are some of the
best job candidates though. Plus chances are they do need a job after all.

~~~
robryan
Yeah I would agree, I have been allocating a block of time on a Saturday or
Sunday to get through the content.

Definitely learnt a lot, which surprised me because I did 3 classes that were
based around AI and probability in my course. It has enlightened me a bit on
the difference between a top of the range CS college and where I did my degree
just it terms of how much ground can be covered and covered well.

------
rhygar
A program like this is a great way to promote cheating.

------
plinkplonk
Prof Thrun mentioned in the last office hours that there were about 1650
people with "perfect scores" and he seemed a little thunderstruck by that and
said he would admit these folks into Stanford if he could, because they are
(paraphrasing, from memory)"Stanford quality".

What struck me was how much importance he gave to this metric, which isn't
_that_ hard to game on an online offering.

I know a few people (who shall remain nameless) who collaborate and check each
others answers and so on before submission, in direct violation of the
Stanford policy (and have 100s or close to it), and so probably have received
this mail, whereas more "deserving" (note quotes) people who honestly work
through the course material may not because they have, say, a 85% or 90%
score.

That said, my key takeaway from this is that professors are very impressed by
_perfect_ scores irrespective of how you got them. There must be something
magical about that row of 100s. Once you set up a grading/ranking system, it
is psychologically very hard not to admire people who end up at the top.

I am personally a little dubious that the people with the highest scores would
make the best pool of employees, especially given that this is an online
course _without_ the programming component, but what do I know?

I wrote Java code for most of the algorithms in AIMA as a side project a few
years ago [1], and after I read an online post by Peter Norvig saying a few of
his students had tried and failed a few times (to implement the code _in Java_
\- Common Lisp code existed and the Python version was in its infancy), I sent
him the code and this became the "official" java distribution for AIMA (
though I don't maintain it anymore- the immensely talented Ciaran O'Reilly of
the Stanford Research Institute does) and no one _ever_ invited me to Stanford
or offered me a cool AI job[2], sob! :-P.

No I am _not_ bitter I tell you, not even the teeniest bit :-p [3]

I wonder how this signalling will play into the upcoming courses? If there are
tangential real world benefits to be gained by attempting a "perfect" score,
then you can expect a lot more game playing wrt scores and exams.

[1] More about how Peter Norvig shredded my initial code etc here
<http://news.ycombinator.com/item?id=2405277>

[2] though eventually, after a lot more work, it did lead to my working on
good ML/robotics etc projects _from Bangalore_ , which is a hard thing to do
in the Great Outsourcing Wasteland.

[3] I am really _not_ bitter.

I wrote the code for the hell of it, not to get a job. AIMA was my
introduction to the fascinating field of AI. It is a great, great book and it
has a _lot_ more material than is covered in the course.

I once did want to go to Stanford and learn from the great profs there, but
now in a "mountain comes to Mohammed" fashion, Stanford is coming to me. I
don't care about the credentialling - I just want to learn. I took the AI
online course and enjoyed Peter's and Sebastian's teaching immensely. Fwiw I
should have a high 90's score, (I didn't add it all up) but nowhere near a
perfect score.

~~~
jpeterson
_I know a few people (who shall remain nameless) who collaborate and check
each others_

Why would someone do this for a free online course that gives no credit for a
degree? I mean, the whole point is to learn, not get the highest grade. I
really fail to understand people sometimes.

~~~
jonknee
People form study groups all the time, it's a great way to learn. Since it's a
free online course all of the benefit is what you actually learn, there is no
way to cheat. At least that's my point of view (I am not in any of the
classes, but wouldn't hesitate to co-work on stuff if I was).

~~~
briandon
_Homeworks must be completed individually, and while we encourage students to
help each other learn, homework assignments must be your own work and not done
with a group._

 _As with the homeworks, exams must be completed individually without the help
of other people._

<https://www.ai-class.com/overview>

I think that lots of people probably did collaborate on the homeworks and the
mid-term and likely will do so again on the final and it was and will be
cheating.

It's a shame, especially given that the instructors do seem to be attaching
some importance to students' scores and rankings, but I'm not letting it
detract too much from my enjoyment of the class.

~~~
jonknee
Again, maybe it's just me, but this is a free online course that everyone is
doing for their own knowledge. There's no degree being granted and it doesn't
count for anything. I wouldn't bother to read the rules and certainly not
attempt to follow them. I would try and learn the material as best I could,
however that is.

------
quasistar
Not sure I agree with the correlation that the 'most talented engineers' are
those that 'scored highest on the AI class HWs and Exams,' but I certainly
wouldn't refuse any of the folks who were able to solve the ApproximateAgent
PacMan Search problem in under 30 seconds from joining my team ;)

Its wonderful that Sebastian & Peter are reaching out like this after all they
have done so far. Congrats to everyone that slugged it out, and good luck on
the final this weekend!

~~~
bgutierrez
There are probably a lot of talented engineers not in the top 1,000, but there
are also a lot of non-talented engineers not in the top 1,000. I guess it's a
reasonable cut-off. Thankfully it's not the only way to get into a great
company.

------
clavalle
Likely they look at everything, not just the graded portions, to come up with
that kind of split. So if you got 100% on every quiz the first time, every
homework and test you'd be in that group. If you flubbed the quizzes here and
there, probably not.

They will also probably look closely at that optional programming assignment
as well.

I knew there was a reason Google let Peter Norvig take the time to do this ;)

~~~
brg
The optional programming assignment is a standard quiz. There is really not
much to it, and can be gamed without writing a line of code.

------
LogicX
Surprised no-one brought it up yet: What I'd like to know is if anyone
involved with the class is getting a referral bonus for bringing candidates to
companies...

Follow up is how we feel about that...

~~~
sbisker
I'm not sure why there's a down-vote here, because that was exactly the
question I had. There has been a system of professors referring their top
students to their colleagues for years, and on a small scale I don't see much
issue. However, with these large online classes, these professors can stand to
be gatekeepers to an incredibly massive - and valuable - source of leads. It
doesn't help when there is no true transparency to the process, and a desire
for the school to recoup costs - we assume the professor would just send the
top 1000 over regardless of source, but what's to stop them from only
referring the names of students who, say, enroll for credit and pay the
university? Or those who pay some sort of a separate "consideration fee"? I
can't put my finger on why, but I'm a little unnerved by this...

------
ih
Just want to mention that the start-up behind ai-class.com is looking for
great engineers. If you're interested in education shoot me an email at
ih@knowlabs.com.

------
jongraehl
That this inevitably leaked due to someone's bragging will encourage future
cheating.

On the other hand, it was inevitable that as long as "grades" or equivalent
are officially certified (even though the course is not for degree credit),
that people would collect online courses as credentials. Wherever there's
signaling value, there will be cheaters faking the signal.

Disclaimer: I got the email and missed two homework questions.

------
casualaistudent
Why are they sending out this letter before the final?

------
redcircle
This type of thing wouldn't help you get a job with me. It could make a
difference in getting an interview with me, if you had no real experience on
your resume/CV. I want a well-balanced team player, that has a good chance to
flourish on my team. Do course grades correlate with real-world flourishing on
my team? Of course not.

------
agentultra
Bugger. I had to drop out because my cat died and I fell behind on the home
work. I hope they do it again.

------
brg
It seems that this was sent to students with 100% aggregate score up to this
point (after dropping the bottom two quizes). One acquaintance had 1 wrong in
aggregate and another had 0 wrong in aggregate but two imperfect quizes, and
only the latter claims to have received the letter.

~~~
learc83
From what I've seen I think they are taking all scores, I've heard from
someone who got 6 100s and didn't get it, but he didn't attempt the last 2
quizes b/c he already had 6 100s.

I made 6 100's, a 98, and a 97, and a 100 on the midterm and I got it.

------
lemonad
It would be interesting to know what data they use for these kinds of
assessments outside the given ones for the course, that is the best 6
homeworks out of 8 account for 30% of the grade, the midterm for 30% and the
final exam for 40%.

------
jpeterson
Well, finding out that I've no chance of making the top 1,000 in the class or
any opportunity of a job is certainly a swift kick in the huevos, 2 days
before the final...

:(

------
dianaZ
Is this for real? Can anyone verify that this letter was in fact sent out?

~~~
learc83
Yes, I received one, and it's from the same stanford.edu email address that
sent the letter congratulating people on perfect 100s after homework 4.

------
HilbertSpace
Maybe, but be careful and think 'long shot':

I was one of three researchers in a team at Yorktown Heights that did some of
the best research in AI in the world, published a string of peer-reviewed
papers, was the source of two commercial products, gave a paper at the AAAI
IAAI conference in Stanford with for that year the 25 best AI applications in
the world, personally won an award, etc.

I've taught computer science at Georgetown University and Ohio State
University. My Ph.D. is in applied math.

Once in my career, just as a 'scientific programmer', in two weeks I sent a
few resume copies, went on seven interviews, and got five offers.

But after my Ph.D. and AI work, I sent over 1000 resumes to Google, Microsoft,
GE, FedEx, and hundreds more, got only five interviews, and no offers. I got a
nice letter back from Fisher Black (as in Black-Scholes) saying that he saw no
applications of applied math or AI at GS.

I ask you: Who will hire you in AI and why?

In business, hiring is because some manager has some work to do and a budget
to do it. That manager believes that they know nearly all that is needed to do
the work and otherwise would not be betting his career on the work. Thus, the
manager is not hiring high technical expertise he doesn't have. Instead the
manager is, as on a factory floor 100 years ago, hiring labor to add 'muscle'
to his work.

In particular, unless the manager knows AI, he won't be hiring for AI. And
there is at most only a tiny chance that the manager knows AI and even less
chance that his project will depend on AI.

Moreover, the manager does not want competition from below and does not want
his project 'disrupted' from below so really doesn't want technical expertise
above what is needed just to get his project done.

Net, if you know some AI and want to use it in business, then find an
application and start your own business. Then, since you know AI, you won't
have to hire anyone in AI either.

What I've said here for AI holds for essentially all advanced academic topics.

Sorry 'bout that.

~~~
patrickyeon
You were looking for someone to hire you to do AI work because you were great
in AI. This reads more to me as using a student's performance in ai-class as a
proxy for their overall software engineering performance. Seems like a better
metric than many others that employers use.

~~~
HilbertSpace
I was just trying to get HIRED for anything, anything at all, not seriously
illegal, immoral, or dangerous, and I would have compromised on those.

So, since this thread was about 'job placement' for students who did well in
an AI class, I posted about my experience getting hired where part of my
background was some expertise in AI.

Net, I have to conclude that, in getting a "job", expertise in AI will be from
very rarely helpful to often a serious disqualification as in "It appears that
you are overqualified for our position and would not be happy in it".

~~~
llimllib
> I was just trying to get HIRED for anything, anything at all

You are "overqualified". People will read your resume and assume that you will
be unhappy with a regular software engineering job, and therefore not hire you
for such a position.

Like it or not, you'd need to find an AI job where people needed your
particular skills. (Or something closely related.)

~~~
xtracto
I am now in a simlar position. I have a BS in Software Eng. and did a
Masters/PhD in Computer Science after a year and a half of software
development work.

After my PHD I got a postdoc in a 3 year EU project just finished. Now I am
craving to get out of "academia" and get into software development again. The
problem is that I would consider my development skills as a "junion" or "mid
level" developper but without hardcore expertise in a technology.

And the worst problem is that as you say, a lot of companies that see my
Resume see "PhD" and think "overqualified".

Recently I tried appliying to a group some company that is doing Machine
learning with the hope that they will see a PhD as a "feature" and not a bug.

