Re: Analysis technique sought for intersubject pairings

On 3 Dec 2003 00:24:30 -0800, [EMAIL PROTECTED] (Ray Koopman) wrote:

>Jay Weedon <[EMAIL PROTECTED]> wrote in message 
>news:<[EMAIL PROTECTED]>...
>> 
>> A researcher wants to find out whether there is a systematic
>> difference between two human populations (let's call them male &
>> female) in ability at some one-to-one competitive skill (let's call it
>> chess-playing). The methodology adopted is this: Supposedly
>> representative samples of N1 males and N2 females (different sample
>> sizes, for reasons I won't go into) are selected, and each of the N1
>> males plays a single chess game against each of the N2 females. The
>> results (win/lose, there are no draws) of these N1*N2 games are
>> recorded. 
>> 
>> I need a 95% confidence interval for the probability that a randomly
>> selected male will defeat a randomly selected female. Other than using
>> resampling methods, how can I estimate this? I need to take into
>> account the varying skill level of the players, and to allow for
>> correlated outcomes among successive games played by the same person.
>
>Consider first a simplified scenario, in which the outcome of each game
>is determined by a simple comparison of some fixed attribute of the
>contestants; e.g., the taller person wins. Even under those conditions
>there is no really good way to get a confidence interval without making
>some sort of distributional assumptions about the two populations. IMHO
>the current best distribution-free interval is given by eq 5.12, p 140,
>in Norm Cliff's book _Ordinal Methods for Behavioral Data Analysis_.
>
>Your problem is more complicated because the outcome of each game is not
>fixed once the two contestants are specified, but depends on a further
>random component -- this feature of the problem would seem to rule out
>the usual resampling methods -- that I presume represents within-person
>variability in the contestants' skills, and that you say must allow for
>(auto?)correlated variation in outcomes of successive games.
>
>This sounds like a paired-comparisons scaling problem in which you're
>more interested in the distributions of the scale values in the two
>populations than you are in the actual scale values of your contestants.
>I don't see how you can get a confidence interval without making lots of
>assumptions, and I can only suggest that you try several different sets
>of assumptions, to see how sensitive your interval is.

Many thanks Ray for your thoughtful suggestions. I'll locate the Cliff
book. I have now figured out how to put the problem into the framework
of a generalized mixed linear model, but a nonparametric approach may
be a valuable alternative here.

JW