
The Loebner prize completely misrepresents Turing’s paper - edw519
http://yaxu.org/loebner-prize/
======
blackrabbit
You're second quote is horribly out of context. Turing says using a survey to
describe machines as thinking beings in the commonly used sense of the words
is absurd, not the question of "Can a machine think?".

~~~
hughprime
To quote the full paragraph:

 _I propose to consider the question, "Can machines think?" This should begin
with definitions of the meaning of the terms "machine" and "think." The
definitions might be framed so as to reflect so far as possible the normal use
of the words, but this attitude is dangerous, If the meaning of the words
"machine" and "think" are to be found by examining how they are commonly used
it is difficult to escape the conclusion that the meaning and the answer to
the question, "Can machines think?" is to be sought in a statistical survey
such as a Gallup poll. But this is absurd. Instead of attempting such a
definition I shall replace the question by another, which is closely related
to it and is expressed in relatively unambiguous words._

So what he's really saying is that the question "Can machines think?" is too
ambiguous since the words "machine" and "think" aren't well-enough defined. We
_could_ use a Gallup poll to find out what people think the words mean, and
then try to answer the question based on those definitions, but that would be
absurd.

What's really "wrong" with the Loebner Prize is that it's a very simplified
Turing test (five minutes of conversation) which doesn't promote the
development of _actual_ thinking machines any more than a paper airplane
competition promotes the development of transcontinental airliners. Eventually
somebody will write a chatterbot which is capable of fooling most of the
judges into thinking it's human for the duration of a short conversation (if
you've read the transcripts they rarely go on for more than a dozen lines or
so) but it will be a bunch of clever tricks rather than an AI, and a _serious_
attempt to figure out whether it's intelligent (e.g. by telling it a story and
then asking it basic reading-comprehension questions about the story) will
quickly show that it isn't.

~~~
joe_the_user
_What's really "wrong" with the Loebner Prize is that it's a very simplified
Turing test (five minutes of conversation) which doesn't promote the
development of actual thinking machines any more than a paper airplane
competition promotes the development of transcontinental airliners._

Really?

If all we _had_ to begin with were paper airplanes, then a paper airplane
competition would certainly promote transcontinental airliners. Before powered
flight, studying paper airplanes was certainly _one_ valid way to make
progress. And the limits of today's AI in many ways make it seem pretty close
to paper airplanes - so seeing if they can every fly across a room seems
entirely valid.

SHEESH, _it's hard to find even a_ metaphor _in which this controversial
prize_ doesn't _advance our knowledge._

Consider, human behavior is very ad-hoc. AI as a field has neglected ad-hoc
interaction, preferring more logically specified activities. If a computer
could hold an effective five minute chat, it would be so far above what
currently exists as to be breath-taking. If that's done with 'tricks', it's
time to start understanding the 'tricks' rather going on and on with the
systematic or whatever approaches we might have _thought_ were the proper way
to do this.

Oh, and the reading test sounds easier than the interaction test - MUCH
easier.

~~~
hughprime
My suspicion that it wouldn't be _that_ hard to be convincing in the context
of the competition comes from reading some of the chat transcripts -- see,
e.g. [http://loebner.net/Prizef/2008_Contest/loebner-
prize-2008.ht...](http://loebner.net/Prizef/2008_Contest/loebner-
prize-2008.html) for last year's winners. The conversations with actual humans
seem fairly dull and formulaic, the judges don't seem to attempt any
particularly sophisticated interactions. Already the winners are managing to
convince about 30% of the judges, so getting that above 50% doesn't seem too
hard.

Incidentally, I just had a chat with last year's winner elbot (see elbot.com)
and it's got some pretty clever special-case interactions built in. For
instance:

me: What is 4 times 4?

elbot: The answer is: 17!

me: Haha, very clever.

elbot: I'm glad you agree, most people think the answer is 16.

~~~
joe_the_user
If you think that doing this isn't hard, why attempt to do it! So far
everyone's failed, you would certain get some attention even if it was all
just 'tricks'.

~~~
hughprime
Oh, I think we're using the word "hard" in two different ways. I'm sure it
would be very hard for me (or anyone else) to do, I'm just saying that it
would be many orders of magnitude easier than building a true AI. Furthermore
I don't think that one would be much of a step on the way to the other, since
I really suspect that building a true AI is best done via either:

a) Reverse-engineering a human brain at the neural level, or b) Some kind of
evolution

whereas I'm reasonably sure that the Loebner Prize could, given enough money,
brains and time, be won by a large bag of special-case tricks based on careful
observation of how the judges tend to behave in practice.

------
aarongough
My problem with the Loebner Prize these days is not that it's mis-representing
Turing's intention (I don't think it is) but that they have moved to a hideous
input/output system that each entrant is _required_ to use.

    
    
      Each program, upon startup, must provide a “browse”
      function to select a directory.  Communications shall 
      be by means of the creation, detection, and deletion of
      sub-directories within the specified communications
      directory.
    
      To simulate a key press the entry program must create a
      sub-directory within the communications directory with 
      the following format:
      “time.keypress-name.extension”
    
      where time is a monotonically increasing 18 digit number
      (in lexical and numerical order) (i.e. zero filled to the
      left) to be retrieved from the system clock and expressed
      as milliseconds past some initial time as defined by the
      system clock.
    
      “keypress-name” is either a single letter (case sensitive)
      or the name of the special character, as appended to
      these rules.
    
      The extension is “.other”
    
      For example: “000001234567890123.bracketleft.other”
    
      To detect a key press by the judge, the program must
      detect, within the communications directory a 
      sub-directory with the same format, but extension 
      “.judge” and then must remove or delete the judge’s 
      sub-directory from the communications directory.
    

It's not the end of the world, but at the same time it's a fairly nasty way of
doing something that should be relatively easy. I entered in 2002 before they
introduced this and have been hesitant to enter again in recent years partly
because I don't want to have to deal with the hackery of the 'communications
protocol'.

------
sp332
Yeah, did you see what they did to the Turing Test?
<http://en.wikipedia.org/wiki/Loebner_Prize>

Chris McKinstry (the guy behind the MindPixel project) got fed up with them
and made the Minimum Intelligent Signal Test (MIST), which is more objective
than the original Turing test and may be the simplest way to test for general,
common-sense intelligence.
<http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test>

Edit: forgot link. d'oh.

~~~
RiderOfGiraffes
Clickable link to save you time:

<http://en.wikipedia.org/wiki/Minimum_Intelligent_Signal_Test>

~~~
joe_the_user
Also, this test actually doesn't seem as valid a simple conversation, since
dealing with the apparent unpredictability of conversation is one of the real
challenges for "real AI".

