Tuesday, May 06, 2008

Passports: Your Control Group Please?

Over at Rant, Tom Fine made the following comment that deserves wider attention:

So what would happen if you found 850 non-athletes with no reason to dope, and tested them 2100 times? Statistically speaking, are you likely to find zero anomalous measurements? That would be almost impossible. So how many anomalous readings to you expect to find? I suppose it depends on how you define anomalous.

And that’s been a big problem so far in cycling. A lot of the tests I’ve looked at define anomalous as just slightly outside of normal.

In a perfect world, the dopers would be strongly statistically isolated from the honest athletes. The honest athletes would for example get results between one and two, and the cheaters would measure at 100. Wouldn’t that be nice. Instead, we have tests where the honest athletes measure from one to two, more or less, and the cheaters have measurements of 2.5 (or so).

That’s little picture stuff though. Let’s look at the big picture.

1. Maybe nobody dopes, and these are all false positives.
2. Maybe doping is widespread, and most dopers are getting away with it.
3. Maybe the tests are perfect, and they’re finding exactly who’s doping.
4. Maybe there’s a lot of dopers, but they’re really good at cheating, and the tests suck, and the people getting caught are still all innocent.

Statistically, these cover the four corner cases of what these results mean. And sociologically, these are the corner cases of our current culture’s mythology. Item number 1 doesn’t really represent anyone’s mythology. Item 2 is the most popular mythology about the current state of the world. Item 3 is a mythology that seems to be held by many supporters of anti-doping.l And 4 is the mythology of cynics like me, that think anti-doping is doing more harm than good.

In sociology, “myth” doesn’t mean something is false. It’s a widely held belief that influences society, regardless of whether or not it is true. But in this case I think that none of these myths can be purely true. The real truth must lie in the middle of all of these myths. Statistically, it’s the most likely answer.

The biggest problem of anti-doping is it’s failure to try to predict false positive rates. We can’t rule out the possibility that the positive rates are similar to, or only two or three times higher than, whatever this false positive rate might be. In fact, a number of people have looked at some of these tests, and concluded that false positive rates could easily be one in a few hundred.

And so we get the the statistical heart of the problem. When positive rates are fairly low, and are not drastically different from best guess false positive rates, then we could easily be at the intersection of all of these myths:

1. A lot more people dope than are being caught.

AND

2. A significant portion of those who are caught are innocent.

Not only does statistics allow both of these to be true at the same time, all the numbers I’ve seen, including these new numbers, suggest to me that this is the most likely description of the world we live in.

Tom

4 comments:

Jon said...
This comment has been removed by the author.
Jon said...

Lies, damn lies, and statistics. Measures of central tendencies can be manipulated by mathematical models to achieve a desired result. Even then this does not account for individual differences within a population. The "biological passport" science is incomplete and those who refuse to admit that "false positives" are impossible because of anti-doping dogma will destroy the athletic careers of innocent athletes.

Cub said...

"So what would happen if you found 850 non-athletes with no reason to dope, and tested them 2100 times? Statistically speaking, are you likely to find zero anomalous measurements? That would be almost impossible. So how many anomalous readings to you expect to find?"

And how many more would you find if you subjected them to the kinds of physical and mental stresses and goofball diet of a hard training or in-competition endurance athelete? The point being that it is not valid to assume that you'll measure the same things in elite athletes that you do in non-athletes (e.g. a bunch of med students at UCLA, as in one of the big studies on the IRMS test for exogeneous testosterone).

But then again, anti-doping is as much about poltics, zealotry and commercialism as it is about science. I guess to some that justifies cutting some scientific corners.

jrdbutcher said...

Excellent points as usual from Tom Fine.

I'm a hopeful sceptic wrt the bio passport. Increased knowledge can yeild more accurate judgements, but only if interpreted correctly.

Several teams have funded their own version of the bio passport system. Again, I find that an excellent (perhaps the best?) insurance policy to defend their riders against bogus AAF's.

The only indication I've seen that the bio passport system might represent an era where the rules are better respected is that the name(s) of suspected athlete(s0 has not been leaked. That give me some small hope they will follow through on the rest of the package to actually make it work as advertised. Again, time will tell. I, and many others, will be watching.