Irregular Report 5
News
The AFP posts a story about biostatistics expert Donald Berry who says that the anti-doping tests used at the Olympics , which start this week, can be erroneous. He uses the Landis case to illustrate his point:
Donald Berry, an expert in biostatistics at the University of Texas, used the case of American cyclist Floyd Landis to point up flaws in anti-doping procedures, but cautioned that the problems he uncovered apply across the board to lab tests designed to ferret out athletes who cheat by using performance enhancing substances.
Writing in the British science journal Nature, Berry argues that the tests performed by the French national anti-doping laboratory (LNDD) that condemned Landis to ignominy and barred him from competition for two years were "non-informative" and potentially subject to error.
While Berry does not have an opinion as to the cyclist's guilt or innocence, he is highly critical of what he called the "inherent flaws" in current testing practices.
Outlets from SciScoop to the Telegraph UK have picked up the sensational story with differing spins and details. Like VeloNews.
The full original article is available at Nature, for $18. We got a copy and it's not long, but respecting their rights, will not quote at length. There is this figure addressing the positivity criteria/single metabolite issue available:
(click for larger)"Plots show the distribution of 167 samples of the metabolites etiocholanone and 5
-androstanediol (a, b), and androsterone and 5 
-androstanediol (c, d). Panels b and d show samples the French national anti-doping laboratory (LNDD) designate to be 'positive' (red crosses) or 'negative' (green dots); the values from Landis's second sample from stage 17 is shown as a blue dot. Axes display delta notation, expressing isotopic composition of a sample relative to a reference compound." (source: Nature)We think this oversimplifies by skipping two plots, being the same ones for the metabolites that were not positive at all in the Landis samples. It's a good visualization of the data of the LNDD tests in discovery, and a model we may try to follow if we ever want to type the 200+ rows of numbers into a spreadsheet of all the negative and positive tests LNDD had performed. We particularly wonder about the athlete who was declared a positive on a single metabolite when all the others were not measured at all -- what possible validation had been done on that metabolite? Did the athlete confess? Given the lack of transparency, it's unlikely we'll ever know.
There have been a number of good comments on the Berry/Nature article below, and we'll collect some of them in a post before the opening of The Games.
The CyclingNews says that Tom Boonen has learned his lesson, Ricardo Ricco has named his EPO supplier, and Kayle Leogrande will have his day in court as he sues Matt Decanio and Suzanne Sonye for slander. Leogrande is represented by Howard Jacobss. The big news in this edition is the revelation that the "welcome mat" may indeed out for Landis at Rock Racing:
Tyler Hamilton has welcomed Floyd Landis to join his American Rock Racing squad, once his ban from competition ends next year. The American tested positive for testosterone after the Tour de France in 2006. "Floyd is welcome to join us immediately," 37 year-old Hamilton said, according to Belgian publication HLN. "I know what he is going through, we can help him return to the top."
Landis is suspended until January 30, 2009 after having his Tour de France title stripped from him following a drawn-out legal battle.
Hey, if the paychecks are good enough and don't bounce, it'd be a job. We really want to see Landis go up Palomar in the Tour of California.
ESPN/AP reports on Emanuele Sella's out of competition positive for EPO. Last we knew Sella was a "he", so someone is probably confusing Sella for Marta Bastianelli.
Blogs
Racejunkie notes Emanuele Sella's doping bust and cries foul for poor Gilberto Simoni, and speaking of folks who have been screwed over will Floyd Landis really "marry" Rock Racing? Only time and Floyd will tell.
Recovox News also shares the Landis to Rock Racing story prompted by Tyler Hamilton's comments. Landis is pictured below in RR regalia, probably at the Amgen Tour of California earlier this year.
Picture from Recovox News
15 comments:
Well, it will be interesting to see how the cycling world reacts if Landis is riding again next year.
If he does go back to riding seriously, I hope he commits to it 100%. Stick it to the Man, Floyd!
syi
I sure aso will still ban him from the tour de France. If he joins rock racing I will have to become a fan.
Floyd signing on with RR seems to make him "...subject...to this farcical system again." Like many people though, sometimes you have to take a job under less than ideal conditions.
And surely ASO would pressure AToC to exclude him from that race.
I am already a RR fan. Would love to see Floyd racing again, and in RR colors would be an extra treat.
ASO has a big say now in the
Tour of California with its 49% ownership interest. While it is safe to say that Rock Racing is not ever likely to be granted a spot in The Tour de France, Rock is a significant sponsor of the TOC. What will ASO have to say about RR (a significant TOC sponsor as well as participant) bringing Floyd to the Tour of California?????????????
This communal time watch makes it more interesting. Or as the frogs say: "Time is fun when you're having flies."
there goes the neighborhood.
Oddly enough, a big fight in CA over Floyd's participation in the AToC between RR and ASO might be a good thing. Floyd is a CA resident, and past winner of the race. Mt. Palomar is his training ground. RR is a big sponsor of AToC. Amgen is a very large company based in CA. The governator of California is an ex-athlete (yeah, with a history of doing what....) The AToC relies on cooperation of the local government (like all race promoters) for use of the roads and cooperation with traffic control.
Now, a *French company* wants to tell all these poeple who can ride a bike in their race, because...well, they don't like Floyd? Imagine how the residents, voters and legislators, in even the very activist state of California will react to *that*.
Oh, bring that on....
I like the way you think, eightzero!
syi
Can anyone explain that Nature graph to me? I am no good at statistics.
syi
The AFP article also says:
The same type of tests that ruin careers would be rejected by regulatory agencies such as the US Food and Drug Administration as scientifically unsound for diagnostic tests to detect disease, he told AFP in an interview.
I think we already figured that out, but it is nice to hear another real scientist say it.
The response from the IOC?
International Olympic Committee (IOC) medical director Patrick Schamasch, contacted ahead the commentary's publication, would not comment directly on the study but said: "What we are doing in the area of doping is the most advanced in terms of certitude."
Wow. Worthy of Dick Pound himself. What a joke. Someone with nothing but an academic interest in the issue, and someone with a huge political and financial interest in the issue - who you gonna trust here? If you are the AAA or CAS arbs, obviously the later.
Sick.
syi
More good quotes from this Donald Berry, from some Australian radio show:
SIMON SANTOW: You're suggesting that drug testing is not effective at the moment. On what basis do you come to that conclusion?
DONALD BERRY: The major flaw is that they haven't quantified the accuracy of the testing process. They have a testing process that they use. I'm not certain that it's sufficiently well defined to be validated. It needs to be well defined.
For example, if the ratio of testosterone to epitestosterone is greater than four, than we do the following thing; and if this happens, then that. That is, the whole process has to be laid out.
Then that has to be validated in a study, an experiment in which there are known positives and known negatives so that we can assess what the false positive rate is, what the false negative rate is and therefore be able to conclude in a specific case how likely is it that the person was actually a doper, based on the results of the test.
SIMON SANTOW: So in your view, we simply don't know at the moment, that too much is assumed knowledge?
DONALD BERRY: That's correct.
SIMON SANTOW: We hear both from the drug-testing authorities that they're going to catch cheats and that they have a new test and that nobody should assume that they can get away with cheating.
How much of that do you think is bluff and how much of that is actually based on good science?
DONALD BERRY: I (laughs), I don't think any of it is good science, frankly, from what I've seen.
Some of it may be bluff and indeed that may be the best way of handling the circumstances. But when they've got a new test, that new test has to be validated. Indeed they are probably catching dopers but just because there is lots of doping going on, and not so much that their test is all that good.
SIMON SANTOW: But do you think there might be a degree of deliberate concealment of their methods because in revealing their methods and the science behind the test, they may be giving clues as to how to get around it?
DONALD BERRY: Yes, I think there is definitely an aspect of that. And it's not an easy question. Do you tell the mouse how the mousetrap works?
On the other hand, concealing the methods is also concealing them to critical appraisal and to scientific investigation.
On the last point, that is, of course, exactly the way it is intended to work.
Okay, I see this is my fifth post in row. But this stuff is amazing. Quotes from an MSNBC article:
Those working within the testing system also point out that proven false positives are rare. (DUH - THAT'S BECAUSE YOU MAKE IMPOSSIBLE TO PROVE A FALSE POSITIVE!) They say they’re acutely aware of the consequences of a false positive, and, if anything, err on the side of caution so that there is a much greater chance of a false negative allowing a guilty athlete to escape.
But Berry believes that if, according to the UNESCO treaty, the war on doping is about to become a legal matter like the war on drugs, there is no room for error.
“When I see things I do not agree with, I sometimes pass it off, but if it affects people, it bothers me. It eats at me,” he said.
WADA often reacts defensively to criticism, but Berry said he’s up for the fight.
“Recognizing the crap I will go through, this is worth it to me. We all want to make a difference,” he said.
Bring in on, baby!
I know where he can find some supporters!
syi
And another from a blog called "ars technica."
Herein lies the problem - the Wold Doping Agency has neither conducted nor published the necessary studies to establish sensitivity and specificity in these metabolite ratio tests. An analysis of the case of Floyd Landis from the Tour de France, assuming reasonable values of sensitvity and specificity, indicates that there was between an 8 and 34 percent chance of registering a false positive. Given that he had eight different opportunities to test positive (far more than the average Tour rider because Mr. Landis was a front-runner throughout the race), the case against him suddenly looks substantially weaker.
TBV, that one really ought to make it onto the front page!
syi
I think Donald Berry has a fight to pick!
Anyone know anything about Berry? Why is he sticking his neck out? Does he have a beef with WADA?
Mike -
Good to have you back!
In theory, the separate testing of B samples should do a lot to reduce the error rate, and I don't think Berry considered this. Of course, in order for the B testing to function in this way, the B samples would have to be tested in a truly "blind" way, and that kind of blind testing does not appear to happen (especially in cycling, where the A tests are announced and acted upon before the B testing is complete). But if you're just looking at the statistics, you'd have to consider the B testing. I have not seen the entire Berry article, so I don't know if he does this.
As for what the Figure 1 chart means ... I HAVE seen an excerpt from the article, but I'm not sure what point Berry is trying to make with this chart. He says that the chart shows results for 139 "negative" cases and 28 "positive" cases, including the Landis case, and that the chart shows how the WADA criteria for positive and negative results was applied to these cases. He argues that establishing threshold values like WADA does is good to "define a hypothesis", but that this data is useless without further investigation on known samples. Yeah ... this doesn't exactly explain why Berry included the chart. The point he seems to be making doesn't require a chart.
Berry may not realize it, but he's making a potentially damning condemnation of the "targeted testing" used by the AFLD during this year's Tour de France, where the lab tested a handful of "targeted" riders over and over and over again. Based on what I've read, there does not appear to have been any methodology to AFLD's selection of many of the riders on its target list ... and the repeated testing clearly raises the spectre of false positive results.
I wrote a piece last month for TBV about the possible meaning of the 20% error rate initially (and apparently, incorrectly) reported for the LNDD's CIR test by the LNDD's ISO 17025 accreditation body. I have been working on a part two to this article, where I address how LNDD came up with its +/- 0.8 margin of error for its CIR test. According to the documentation released with the CAS decision (see LNDD 0451-0456), LNDD came up with its CIR margin of error by performing 30 different analyses of their blank urine pool over a 7 month period, coming up with a range of delta-delta values of 0.4 at a single standard deviation. Per the ISL requirement that labs determine the "expanded uncertainty" of their methods, LNDD multiplied this maximum standard deviation by 2 to get its error rate. But as has been pointed out here before, this is not a proper way to determine a margin of error. LNDD has simply measured method "precision" (how close multiple measurements made by the same method are to one another), and ignored method "trueness" (how close the lab's measurement comes to the true value that the lab is trying to determine). The WADA rules require the lab to consider both precision AND trueness. This is discussed in some detail in parts 7 and 8 of my infamous "Curb Your Anticipation" series, if you want to know more.
In simplest terms, the LNDD procedure for determining margin of error looks solely at the consistency of its test, without ever determining whether its test is consistently wrong. It is a bit like purchasing an old adding machine that has been programmed to "think" that 2 + 2 = 5. You can test the adding machine over and over again, and it's always going to tell you that 2 + 2 is 5. According to the LNDD view of things, this adding machine is perfect, and can be used without need of a margin for error.
Post a Comment