Re: Article - Why the Loebner Prize Will Never Be Won...

Hugh Loebner wrote:
> 
> The requirement that the computers operate on site is to prevent
> fraud.  We all want that, don't we? Contestant who have special
> hardware requirements are invited to provide the hardware.  In 1995
> Thom Whalen provided (or caused to be provided) a Sun workstation.
> What hardware did you have in mind?  What, in any case, is inadaquate
> with today's 1+ Ghz 500+ Meg machines?  What more do you need?  Do you
> have a handy neural net you want to enter?
 
Basically, there are two "camps" of language processing.  The first 
camp does patternmatching (like eliza) and is very cheap in computer
hardware requirements.  These programs can run on a single machine and 
can respond in under a second, unless implemented very badly.  These 
represent a "sweet spot" in implementation effort vs. performance, in 
that it is simple to construct something that performs well, within 
certain limits.  But those limits are very hard to overcome without 
ripping the whole thing down and starting over with a different approach
entirely. This is a "dead end" as far as building systems that have any 
actual intelligence, but one where interesting and possibly useful 
performance is available, and hobbyists like it. 

The second camp uses techniques that are much harder to handle and 
more computationally intractable, but which vastly exceed the limitations
of pattern matching.  Usually they do parsing (like XTAG) and get much 
better analysis of sentences - but typically run on networks of a dozen 
or more computers, have a working set in excess of 6 Gbytes, and may 
need to stop and work on a given input for several minutes to an hour.

This seems to continue to be true even as hardware capabilities advance; 
the guys doing parsing build better and more accurate grammars and 
parsing engines that occupy more space and take more cycles as the space 
and cycles become available.  Since what they are writing is grammatical 
rules and configurations that have a lot more interdependencies and 
implications than the matching rules in the ELIZA camp, the work is 
intellectually very much harder to do.  This bunch is mostly people 
who are doing hardcore research into what language is and how people 
use it, and typically they don't own all the systems they're running 
their stuff on and can't bring them to a contest. 

None of them are interested in a contest where they'd have to use 
language technique (even from their own camp) that is about six to 
seven years behind the curve, usually because their Ph.D committees 
require them to be covering *new* ground. The Loebner contest 
appears to be such a place. 

Basically, the perception is that if you want to enter the Loebner 
contest, you have either massive logistics and expense (not to mention
approvals and signoffs and interdepartmental politics) to get a 
network of machines from your university on site, or you have to 
dumb it down and waste time on something that's not serious in order 
to fit it on one or maybe two boxes and respond within fifteen 
seconds to a minute at most.  So the research camp ignores the 
contest entirely and leaves it to the chatterbot hobbyists with 
their ELIZA clones and a few experimenters who are trying things 
that are completely different. 

One thing I've observed is that you are getting some experimental 
architectures in the contest; some researchers are showing up with 
things they couldn't get official support from their universities to 
investigate and which they're doing "on a lark."  These are neither 
pattern matchers nor parsers, and represent interesting new work 
or hybrid approaches.  But their absolute performance, while 
interesting, doesn't appear to be very convincing so far.   

                        Bear