Re: Why is NLP such a hard topic?

On Thu, 06 Nov 2003 09:24:17 +0100, Niki Estner wrote:
The answer in simple in statement but incredibly complex in its
implications.

Question:  Why is NLP a hard topic?
Answer: Because natural language is a a rules based system where users
don't follow the rules.

Seems odd, but you have to understand that language is a social phenomena.
It reflects the users and is embedded in the larger communication context
of the language community.

For example, take ambiguity.  NLP hates ambiguity, but if you take
ambiguity out of a language, users put it back in.  Why? they want wiggle
room when explaining to their spouse who they were working late with. Or
what ever.

Two more specific problems:The linguistic content of an utterance is embedded in a 
larger non
lingusitic message.  This means that:

1. Some of the linguistic elements carry non-semantic information that
reflects on the speakers intentions, goals and attempts to create a
specific response in the listener:

    "Do you enjoy making me look like total fool in front of my
family"?  You can just smell the fight coming in that one right?

Metaphor, innuendo, sarcasm.  Think of the way meaning can turn based on
on intonation "Please no, don't let me inconvenience you."

2. Conversely context allows speakers to omit chunks of linguistic content
that can be inferred from content.

   "The University canceled the demonstration because they were afraid of
violence."  Context suggests "they" is the University -- anaphora
resolution by real world knowledge.  Listen to conversations, speakers do
not speak in sentences or well formed linguistic units:
   
   "Hey"
   "Uh?"
   "Hungry?"
   "Nah"
   "Wann do sumpn?"
   "I dunno, like what?"
   "you know, like hanging."

You can picture the context can't you?

Written text in somewhat better but it is a reflection of spoken language.
For example, many years ago, I lead a project that developed a machine
translation program form English to French.  The question was asked how
accurate was the translation -- 80%, 90%.  I suggested that the way to
answer this was to have two human translators translate the text first
then one they agreed on a benchmark translation, rate our programs
output.  No two translators ever agreed on a right translation for the
benchmark.  Several times we they even came to blows.

The problem is not our technology, but that we do not yet understand enough
about how language works to do good NLP.  NLP tries to model something we
do not fully understand.

> I have long been wandering about that question, and did not really find any
> good answers:
> There are many (profitalbe) uses for natural language processing (automated
> translation, search engines, spam filters, database querys...), and lexicons
> (even including kind of semantic information) are available, and yet (as far
> as I know) there is no really good implementation for natural language
> processing (Think of something like English in BNF).
> 
> Why is this the case?
> 
> It's surely not because it would be too much work to write down a few
> thousand rules or more (theres money in it).
> 
> Can anyone explain me (or point me to information that does) why this is
> such a hard topic?
> Is the number of rules simply too big???
> Can syntax not be analyzed without semantics?
> Is some mystical "soul" required to process natural language????
> 
> Thanks a lot
> 
> Niki

-- 
.................................................
I almost had a psychic girlfriend but she left me before we met.

Rod Davison -  Critical Knowledge Systems Inc.