
A professor built a chatbot to be his teaching assistant - dlgeek
https://www.washingtonpost.com/news/innovations/wp/2016/05/11/this-professor-stunned-his-students-when-he-revealed-the-secret-identity-of-his-teaching-assistant/
======
Bartweiss
This is definitely an impressive result, but I'm pretty sure it was made
easier by the opposite pattern.

I've had a couple of TAs who I strongly suspected of being chatbots, even when
I was talking to them in person. I'm still not 100% convinced they were human.

~~~
tfm
More human than human is our motto ...

Next step will be to replace the prof, and then have the students replaced by
surprisingly attentive interrogative expert systems, and the cycle will be
complete.

~~~
itgoon
Reminds me of a scene in the movie "Real Genius". The class was empty of
people, was just tape recorders listening to another tape recorder set up by
the professor.

------
forrestbrazeal
I was a student in this class, and had at least one question on the forum
answered (correctly) by Jill. I can see this technology being hugely useful
for teachers who conduct large lecture classes in any subject on a regular
basis.

That said, I'd be even more excited if Jill had the ability to synthesize new
answers to questions through some type of case-based reasoning. This would
require Jill receiving feedback on "her" answers, which might mean the
students would have to know "who" "she" was in advance. (Sorry, got lost in
the quotes.) Right now, Jill is essentially an automated FAQ-retrieval bot.

~~~
SilasX
If you had some reddit/HN-style forum where the questions get replies and
votes, that would be valuable information for what counts as a "good answer".

~~~
SEJeff
Yeah I can see that going over well in Physics McPhysicsFace 101

------
lpage
For anyone interested, the class, which covers interactive intelligence and
knowledge based AI, is freely available at
[https://www.udacity.com/course/knowledge-based-ai-
cognitive-...](https://www.udacity.com/course/knowledge-based-ai-cognitive-
systems--ud409) . It's a great starting point if you're looking to automate
the sort of human intelligence tasks that don't lend themselves well to the
traditional searching/planning/proving/minimax/retrieval route that underpins
most AI.

I took it before they rolled out the chat bot TA, unfortunately.

~~~
patrickbolle
thanks for the link ! question though does anybody have any good resources for
doing something similar to this in javascript? I know a small amount of java,
no python, and want to brush up my js skills anyways.

~~~
vinodkd
IIRC there was nothing specific to java or python, except for using opencv for
the last exercise to visually solve ravens matrices. You could do the same
with nodejs, I'd presume. It might be easier too, considering most of the code
involved manipulating dynamic data structures.

------
ars
This is the key:

> The system is only allowed to answer questions if it calculates that it is
> 97 percent or more confident in its answer.

Knowing when you don't know the answer is something that has been lacking from
AI I've seen. Of course the problem is that not all AI has the option of
deferring to a human.

Anyone know what percent of questions the AI answered, vs redirected?

~~~
jobu
> _Anyone know what percent of questions the AI answered, vs redirected?_

That's actually a very good question. The article doesn't give many details
other than: _" There are many questions Jill can’t handle."_

For comparison, here's are the highlights of a study from 2014 on the mobile
assistants (Siri, Cortana, Google): [http://searchengineland.com/google-now-
beats-siri-cortana-di...](http://searchengineland.com/google-now-beats-siri-
cortana-direct-answers-study-205388)

Google was able to offer "enhanced" answers (i.e. more than a web search) 58%
of the time. Siri was at 29% and Cortana was at 20%. Siri improved some with
iOS9, but not enough to beat google:
[http://www.recode.net/2015/9/20/11618696/how-intelligent-
is-...](http://www.recode.net/2015/9/20/11618696/how-intelligent-is-siri-we-
put-her-through-a-few-tests)

From the actual study: _... we took 3086 different queries and compared them
across all three platforms. These were not random queries. In fact, they were
picked because we felt they were likely to trigger a knowledge panel._ (
[https://www.stonetemple.com/great-knowledge-box-
showdown/](https://www.stonetemple.com/great-knowledge-box-showdown/) )

~~~
ars
> That's actually a very good question. The article doesn't give many details
> other than: "There are many questions Jill can’t handle."

I emailed and asked. If I get a reply I'll post it here.

~~~
deckar01
It would be nice if Jill Watson could respond to your email.

------
collyw
Rather than inventing sophisticated chat bots, couldn't he have just had a
well organised web page? If the question were where is assignment two / when
is assignment due. The examples given seem like a "high tech" (possibly
unreliable) solution to a low tech problem.

~~~
adekok
Having written documentation for end users, that will work for only a portion
of them. Probably most, which is good.

But there are a significant percentage of users who can't (or won't) read
documentation. Even they do manage to read it, they don't understand it.

For some reason, human interaction works better for those people. They ask a
question, and someone copies the answer from the documentation, and pastes it
into the chat conversation. The user then goes "Oh, wow! That's helpful!"

But ask them to read a web page, and they get lost.

I would love to know the psychology behind this phenomenon.

~~~
olalonde
> I would love to know the psychology behind this phenomenon.

Laziness. It's usually faster to ask someone a question than to look up the
answer. I used to do this on IRC when learning a new programming language and
it saved me a lot of time. If you want an even faster answer, you guess the
wrong answer and wait for someone to correct you
([https://meta.wikimedia.org/wiki/Cunningham%27s_Law](https://meta.wikimedia.org/wiki/Cunningham%27s_Law)).

~~~
adekok
> Laziness. It's usually faster to ask someone a question than to look up the
> answer.

:(

My common answer is "If you're too lazy to read the documentation, I'm too
lazy to cut & paste it into the email".

Followed by them complaining about how I'm an asshole...

~~~
pm
I did this with my brother. Would always ask me beginner programming questions
instead of getting Google-fu (while running a business making FileMaker Pro
integrations, ugh).

Eventually I LMGTFY. He stopped asking.

~~~
specialist
I bitly my LMGTFY. The passive/aggressive nerd Rick Rolling.

~~~
pm
That's gold. I'm going to use that one next time.

------
mdorazio
"Goel plans to use Jill again in a class this fall, but will likely change its
name so students have the challenge of guessing which teaching assistant isn't
human."

This will be a fantastic Turing test, at least as far as basic question
understanding and natural response formation.

~~~
lordnacho
Hmm, does it count to use things that are not so much to do with the course?

For example, if a guy is able to answer questions at all times of day, really
fast, they're either an AI or Jon Skeet.

Also, real TAs have a life outside of the course. What will an AI say when you
ask them whether they enjoyed last night's episode of Game of Thrones?

~~~
tfm
The software worked remarkably well in this instance because the system was
well constrained, and the students weren't on the lookout for it. If they do
start quizzing their new unseen TA (Dr Colin V Olutionalnet) on random trivia
it could presumably deflect the query or have a few canards pre-programmed.

~~~
Coding_Cat
But likewise, if I were a TA for that course I'd probably try to act like a
bot just to confuse the students.

------
wyldfire
That's stunning. I studied AI in undergrad and I just didn't give it enough of
my focus at the time. Little did I know that in my lifetime I'd see real world
evidence of computers passing the Turing Test with flying colors.

Of course, Turing was right. What's outside of the set of computable sets? Not
much!

I recently read an article headline that talked about creeps using AI to stalk
porn stars. It's interesting to think that as computation power continues to
grow, there's a huge potential for evil uses of AI. All of the things we see
and hear and consider "public" have little or no nefarious uses to other
humans. But what about a computer that has human-like capabilities and
inexhaustible computation resources? The sci-fi scenarios of conscious AI
making decisions to save humanity from itself are not what I fear. The evil
humans sitting at the helm of powerful AI is what I fear.

A lot of society's norms are predicated on humans having private thoughts and
telling small and large lies as appropriate. What happens when AI-augmented
humans know when you're lying?

The future is so exciting and so terrifying.

~~~
eternalban
> What's outside of the set of computable sets?

Human _intution_ can jump. Mechanical deduction must expend energy.

Let me give you an example: Dmitri Mendeleev goes to bed, has a dream, and
then delivers the _periodic table_.

Can your machine do that?

~~~
lordnacho
The periodic table is the elements ordered such that common features occur
together. You'd think some sort of clustering algorithm would detect this.

A line of elements that almost never react with anything. A bunch of things
that gain one electron, some that lose one. And so on.

~~~
eternalban
Note: "must expend energy"

How many Joules did Mendeleev expend in his dream? How many Joules will your
clustering algorithm require? In fact, how many Joules before a meta-algorithm
supervisor decides on exclusively using clustering to solve the problem?

> A line of elements ...

Indeed. He arrived at the result with _incomplete_ information.

    
    
        Rightly traced and well ordered; what of that?
        ..
        Ah, but a man's reach should exceed his grasp

------
pesenti
For those wondering what technology is being used. It is using this Watson
product:
[http://www.ibm.com/smarterplanet/us/en/ibmwatson/engagement_...](http://www.ibm.com/smarterplanet/us/en/ibmwatson/engagement_advisor.html).
We are in the process of revamping it and making it a self-service API that
will be released this quarter in
[http://ibm.com/watsondevelopercloud](http://ibm.com/watsondevelopercloud).
Some of the core functionality is already available in
[http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercl...](http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/nl-
classifier.html) and
[http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercl...](http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/dialog.html)
in case you don't want to wait.

------
lordnacho
Makes sense as a use case. TAs must hear the same questions over and over
again from different students, both practical things like "what homework is
there" and reference stuff like "What does the line above the X mean?" (No pun
intended!)

I wonder how good it is when people don't know what they're asking. This used
to happen all the time when I was in tutorials. "I don't get why there's two
transistors in a chain". "What's the significance of the process being
adiabatic?"

If you can crack that, education will be changed forever, massively.

------
ikeboy
>Goel and his teaching assistants receive more than 10,000 questions a
semester from students on the course's online forum.

Is this normal?

~~~
burkaman
Not 10,000 questions, 10,000 messages:
[http://www.news.gatech.edu/2016/05/09/artificial-
intelligenc...](http://www.news.gatech.edu/2016/05/09/artificial-intelligence-
course-creates-ai-teaching-assistant)

That article says ~300 students, 30 posts each seems pretty normal.

~~~
ikeboy
So the original article:

1\. Didn't mention it was an online course

2\. Didn't mention there were 300 students

3\. Claimed it was questions instead of posts

~~~
burkaman
I don't think 1 and 2 are very important for the story, and the
questions/posts thing seems like an honest mistake. No harm done. The meaning
is basically the same, since the bot can presumably reply to any post, not
just a top level question.

------
srtjstjsj
Actual URL path:

    
    
        /this-professor-stunned-his-students-when-he-revealed-the\
        -secret-identity-of-his-teaching-assistant/
    

HN can do better than clickbait sites like the WaPo.

~~~
TillE
I've never read much of the Washington Post, so I can't say whether there's
actually been a decline since Bezos bought them. But there's been a remarkable
quantity of terrible reporting from them lately, so at least we can say it's
not very good under his ownership.

------
lucb1e
I can't help feeling a big FAQ page with a search function would be a lot more
useful. Especially if the entries contain aliases for common search terms,
like "length" on the entry of "word count", it should cover everything.

Now this is a forum where others can chip in as well, but the FAQ page could
prevent you from having to post in the first place. And if you don't have to
post publicly to get an answer from the chat bot (because it's a FAQ page and
not a bot) you can do 20 searches before deciding it's not in there.

~~~
BrandonSmithJ
Speaking from experience from being in this class, the unfortunate truth is
many people don't bother to read the documents for assignments (thoroughly, at
any rate). This one wasn't as bad as another one I took, but many people will
ask questions which are easily answered by looking at the materials they're
given. That's essentially the area in which this AI excels.

------
alexcaps
This sounded cool until, "Now Goel is forming a business to bring the chatbot
to the wider world of education.” // are all educators just trying to create
companies these days?

~~~
6stringmerc
In my honest opinion, innovating in the Educational sphere and showing value
to one or more groups - educators, students, administrative tasks - should
really be rewarded fiscally. An institution won't really do that, except in
the long-term by way of job stability and benefits (all subject to modern
fiscal pressures). So, while it might seem the case, I think this avenue is
perfectly reasonable.

Now, educators who write texts in concert with publishers and then somehow
coerse students into buying brand new copies every semester, that's what I'd
call profiteering. Not so highly esteemed to me. But, as noted, the profession
is a tradeoff a lot of times.

------
jerryhuang100
in a way i'm not sure why there is still no parents or students complaining
they're kind of being 'defrauded' for a chatbot while paying high tuition (cue
discover card tv ad.) one thing for sure is that there are a lot of undergrad
or grad students around looking for a part-time job and teaching experience. i
know this is a great achievement in AI with huge potential in open course, and
great for grant application, but on the very other end is someone paying with
an expectation of a human interaction & care. for years i've seen a lot of
complaints about foreign human TAs.

------
eternalban
"Education is such a huge priority for the entire human race."

Half a century into this current iteration of the game called 'Life as a Human
on planet Earth' I have come to the conclusion that our _primary_ focus must
be Education and Mental Health. Take care of these and we may yet see our full
positive potential.

------
mathheaven
The next step is to build a chatbot to replace the professor at teaching. If
someone is not going to enjoy the fruits why are you going to supply the
knowledge base for a chatbot that is going to replace you?

------
xufi
Pretty amazing. Considering that I've only really taken 1 online class so far
excluding tutorials that I've done. I wonder if one day how this would be used
for real responses for a real professor someday

------
zeeZ
The site comes with an obnoxious "enter your email" overlay that won't go away
and messes up scrolling when you block it. They do accept @washpost.com
addresses though, so...

------
thaw13579
I wonder how many of the responses were "check the syllabus"

------
sandaru1
Anyone know whether they used watson APIs to create this bot or used the
original watson codebase (with IBM collaboration)?

~~~
pesenti
It's using this:
[http://www.ibm.com/smarterplanet/us/en/ibmwatson/engagement_...](http://www.ibm.com/smarterplanet/us/en/ibmwatson/engagement_advisor.html)
but if you are smart you can reproduce a lot of the functionality using both
[http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercl...](http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/nl-
classifier.html) and
[http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercl...](http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/dialog.html).
We are also coming up this quarter with a fully integrated self-service API
that will make doing all this much easier (especially the dialog part). Stay
posted.

------
deepGem
I wonder what the 'layers of decision making software' on top of Watson mean.
Are these rule engines ?

------
cdnsteve
Anyone have technical details about this story? What kind of bot used,
languages, algorithms, etc?

~~~
orcasauce
IBM Watson

Here's a better article that was linked a few days ago:
[http://www.wsj.com/articles/if-your-teacher-sounds-like-a-
ro...](http://www.wsj.com/articles/if-your-teacher-sounds-like-a-robot-you-
might-be-on-to-something-1462546621)

------
exodust
Already posted in original form:
[https://news.ycombinator.com/item?id=11688061](https://news.ycombinator.com/item?id=11688061)

I don't know why SMH would take a Washington Post story and change the title
to something stupid, while keeping everything else the same, but that's what
they've done.

Wait.. there's one more difference. A pathetic picture of "an AI".

"an AI"? Seriously SMH, just use the original title. Your editorial blundering
is embarrassing, and could easily be replaced with "an AI" tasked with re-
wording headlines.

~~~
oarsinsync
Thanks for that, the original article was a much better read.

~~~
exodust
the original has the sample interaction with students, so its definitely worth
switching, otherwise its the same copy apart from butchered title.

SMH is a bottom of barrel source. Changing a story title to something more
click-baity, sums up the entire Fairfax network and Australian news media in
general.

