Passports: Your Control Group Please?

Over at Rant, Tom Fine made the following comment that deserves wider attention:

So what would happen if you found 850 non-athletes with no reason to dope, and tested them 2100 times? Statistically speaking, are you likely to find zero anomalous measurements? That would be almost impossible. So how many anomalous readings to you expect to find? I suppose it depends on how you define anomalous.

And that’s been a big problem so far in cycling. A lot of the tests I’ve looked at define anomalous as just slightly outside of normal.

In a perfect world, the dopers would be strongly statistically isolated from the honest athletes. The honest athletes would for example get results between one and two, and the cheaters would measure at 100. Wouldn’t that be nice. Instead, we have tests where the honest athletes measure from one to two, more or less, and the cheaters have measurements of 2.5 (or so).

That’s little picture stuff though. Let’s look at the big picture.

1. Maybe nobody dopes, and these are all false positives.
2. Maybe doping is widespread, and most dopers are getting away with it.
3. Maybe the tests are perfect, and they’re finding exactly who’s doping.
4. Maybe there’s a lot of dopers, but they’re really good at cheating, and the tests suck, and the people getting caught are still all innocent.

Statistically, these cover the four corner cases of what these results mean. And sociologically, these are the corner cases of our current culture’s mythology. Item number 1 doesn’t really represent anyone’s mythology. Item 2 is the most popular mythology about the current state of the world. Item 3 is a mythology that seems to be held by many supporters of anti-doping.l And 4 is the mythology of cynics like me, that think anti-doping is doing more harm than good.

In sociology, “myth” doesn’t mean something is false. It’s a widely held belief that influences society, regardless of whether or not it is true. But in this case I think that none of these myths can be purely true. The real truth must lie in the middle of all of these myths. Statistically, it’s the most likely answer.

The biggest problem of anti-doping is it’s failure to try to predict false positive rates. We can’t rule out the possibility that the positive rates are similar to, or only two or three times higher than, whatever this false positive rate might be. In fact, a number of people have looked at some of these tests, and concluded that false positive rates could easily be one in a few hundred.

And so we get the the statistical heart of the problem. When positive rates are fairly low, and are not drastically different from best guess false positive rates, then we could easily be at the intersection of all of these myths:

1. A lot more people dope than are being caught.

AND

2. A significant portion of those who are caught are innocent.

Not only does statistics allow both of these to be true at the same time, all the numbers I’ve seen, including these new numbers, suggest to me that this is the most likely description of the world we live in.

Tom