Re: Article - Why the Loebner Prize Will Never Be Won...

"Ray Dillinger" wrote:
> Hugh Loebner wrote:
> >
> > The requirement that the computers operate on site is to prevent
> > fraud.  We all want that, don't we? Contestant who have special
> > hardware requirements are invited to provide the hardware.  In 1995
> > Thom Whalen provided (or caused to be provided) a Sun workstation.
> > What hardware did you have in mind?  What, in any case, is inadaquate
> > with today's 1+ Ghz 500+ Meg machines?  What more do you need?  Do you
> > have a handy neural net you want to enter?
>
> Basically, there are two "camps" of language processing.  The first
> camp does patternmatching (like eliza) and is very cheap in computer
<snip/>
> The second camp uses techniques that are much harder to handle and
> more computationally intractable, but which vastly exceed the limitations
> of pattern matching.  Usually they do parsing (like XTAG) and get much
> better analysis of sentences - but typically run on networks of a dozen
> or more computers, have a working set in excess of 6 Gbytes, and may
> need to stop and work on a given input for several minutes to an hour.
>
> This seems to continue to be true even as hardware capabilities advance;
> the guys doing parsing build better and more accurate grammars and
> parsing engines that occupy more space and take more cycles as the space
> and cycles become available.  Since what they are writing is grammatical
> rules and configurations that have a lot more interdependencies and
> implications than the matching rules in the ELIZA camp, the work is
> intellectually very much harder to do.  This bunch is mostly people
> who are doing hardcore research into what language is and how people
> use it, and typically they don't own all the systems they're running
> their stuff on and can't bring them to a contest.
>
> None of them are interested in a contest where they'd have to use
> language technique (even from their own camp) that is about six to
> seven years behind the curve, usually because their Ph.D committees
> require them to be covering *new* ground. The Loebner contest
> appears to be such a place.

Actually, PhD candidates aren't good sources for functional whole
systems.  They only have 2 to 4 years to build something, so they
have to specialize very deeply in some small aspect of linguistics.


> Basically, the perception is that if you want to enter the Loebner
> contest, you have either massive logistics and expense (not to mention
> approvals and signoffs and interdepartmental politics) to get a
> network of machines from your university on site, or you have to
> dumb it down and waste time on something that's not serious in order
> to fit it on one or maybe two boxes and respond within fifteen
> seconds to a minute at most.  So the research camp ignores the
> contest entirely and leaves it to the chatterbot hobbyists with
> their ELIZA clones and a few experimenters who are trying things
> that are completely different.

The winning approach will have to be somewhere in between these
two poles.  Enough dramatic mechanism to convince the judges that
there's a human there, and enough efficiency to do it in real time.


> One thing I've observed is that you are getting some experimental
> architectures in the contest; some researchers are showing up with
> things they couldn't get official support from their universities to
> investigate and which they're doing "on a lark."  These are neither
> pattern matchers nor parsers, and represent interesting new work
> or hybrid approaches.  But their absolute performance, while
> interesting, doesn't appear to be very convincing so far.

So far, yes.  But the fact that the contest is a stable feature of the
landscape means people are thinking about ways to meet the goal.
It all starts from there; some day, someone will put it all together
and the contest will make headlines for a long time.


> Bear

Rich