Re: How do you test?

> i've spent the last few years writing software that is supposed to act
> like a human.

Correct me if I'm wrong, but from what follows, it looks like you
focused on a very reduced set of problems, and didn't build a full
"human being simulator". Right ?

> The closer the two
> sets of responses, the more i believe i designed the software
> properly.
...
> If the humans give wildly
> different answers, i don't know what to use as a "right answer"
...
> How are other people handling this?

First, my answer to the last question: I have no idea.
Still, my suggestion would be to model the human behavior as
probabilities, like P(human_answer|situation), that would be deduced
from your tests. Then, you can compare the two distributions
P(human_answer|situation) and P(cpu_answer|situation) (given you
implemented some kind of probabilistic output in your model), with the
Kullback-Leibler divergence for instance (but there are certainly
other methods for this).

Now, for your other questions, they're interesting, but I wish I had
more free time to think 'bout them ;)

Olivier