Notes from the Present, and Future
In a comment, Tom Fine made an interesting observation, which we've taken as a starting point:
Once upon a time, there was a test used to convict dopers, based on what was believed to be the best science and validated by some studies. This was the T/E ratio test. We hoped and believed it was subject to rigorous scientific review. By WADA criteria, it was "deemed" suitable for catching dopers, and reliable.
Over time, we've learned that individual variation is much wider than originally thought in those "rigorous" validation studies demonstrated. There are genetic influences, and external influences (such as alcohol consumption) that were not known or understood originally, and that no one predicted.
It's now difficult to get a conviction on a T/E test without a lot more work, including a "longitudinal study" to determine what is really going on with the subject.
Why then, is the new and improved IRMS testosterone test automatically conferred the status of "gold standard?" The underlying issues are the same, and even more complicated. The science behind the test is believed to be solid, but is basic. It doesn't look for subtlety of variation across a broad group of subjects. It doesn't look for potential outside influences that could skew the test.
And it is very sensitive to the particular and non-standard chemistry and measurement done at specific laboratories.
Yet WADA has adopted some curious standards ("metabolite(s)") that are not clearly supported by validation studies relevant to the procedure in place at individual laboratories with their non-standard methods. By rule, these methods are "deemed" to be as reliable as the T/E test was a decade ago.
In another five years, will we be shaking our heads again at the state of the IRMS test in 2008, because we've learned more? Too bad for those who may have been innocent?
How lucky we'll be to have a NEW new test, which has these brief studies that show it seems to find things that we can use to indicate doping. Let's "deem" that to be reliable.
6 comments:
This really gets at an area of the Landis case about which we have very little information. The lab is required to do validation studies for its tests, studies that actually generate data about the scientific quality of the test. And ISO 17025 certification in some way checks those validation studies.
But the lab is NOT required to make the validation studies public, and they are not open to regular scientific peer review (which to a lot of scientists would simply mean it isn't science). I have read comments by WADA people, from administration and the labs, which talks about the great need for the labs to work together to achieve the best results possible, but it is not at all clear that much, or any, of that happened with the exo-t IRMS test before 2006.
I have not been able to figure out whether the ISO certification simply checks that the validation study has been done, or whether it checks the quality of the study. My impression is the former.
The recent news of the reliability of the T/E test shows that the whole validation side of WADA's process is flawed. But much of this area is completely and intentionally hidden from public view.
syi
Mike -
Terrific comments!
When I wrote the "Curb Your Anticipation" opus, I purposely decided NOT to dive into the process of ISO 17025 accreditation. I had too much stuff to cover as it was. And besides, I knew that there are people on this forum and on related forums who have worked at labs and have experiece with this accreditation process.
But ... IMHO there's no way that the ISO accreditation process could possibly include much of a check on the science underlying the lab tests. I think the MOST the ISO could do would be to make sure the lab has properly documented its validation process.
I base my HO on the fact that there's a wide range of lab tests out there, and the people who run the ISO cannot possibly be expert on every kind of test.
Also, most labs world-wide adopt existing test methodology rather than creating their own tests, as WADA labs are required to do. So I imagine that the ISO accreditation procedures are aimed at making sure that labs have done the things they need to do to properly adopt already-existing test methods.
There is a WADA accreditation procedure in addition to the ISO accreditation procedure. Logically, you would expect that the WADA accreditation procedure would pick up where the ISO procedure leaves off, and that WADA WOULD carefully examine the science behind the tests adopted by its labs. However, I know very little about how WADA accredits its labs.
Mike, we can wish that WADA would be on the forefront of scientific research on how to improve doping testing. I DO occasionally run into scientific papers where a co-author is the head of a WADA lab. And we know that Don Caitlin at UCLA has done some cutting-edge research and development work. But for the most part, I don't see the WADA labs in the forefront of research, and the research they do seems aimed at finding new tests rather than at perfecting (or questioning) established techniques. IMHO.
I don't know if it's reasonable to conclude from the recent studies on T/E tests that the WADA labs aren't properly validating their test methods. I think that test validation can properly rely on the existing state of scientific knowledge. So, if the scientific community is comfortable with the science supporting the existing T/E test, I don't think the lab has to perform basic research confirming this science in order to validate its test method. That situation changes, of course, as studies are published like the one cited by TBV. We could hope that WADA will now begin some kind of formal process to revalidate its existing T/E testing.
The process of when a lab must revalidate its methods would be covered by "Curb Your Anticipation Part 2, The Story of Lab Quality Control." Not coming to a blog site near you any time soon!
Larry wrote: I don't know if it's reasonable to conclude from the recent studies on T/E tests that the WADA labs aren't properly validating their test methods. I think that test validation can properly rely on the existing state of scientific knowledge.
Yes, I think that is right. But those who are hearing the evidence in any given case MUST take into account the quality of the validation of the test. They have to decide what degree of certainty they can gain from the test.
But as things stand now we have a black and white system. If the lab(s) decide that a test is good enough, then it is deemed perfect for all practical purposes. The labs are specifically given permission NOT to make their validation study public. You can question the science, and that is exactly what Landis has ended up doing (on the peak identification issue), but those hearing the evidence have no basis on which to judge how much trust should be placed in the test to begin with. The way the system is set up, I think it is pretty clear the "default" position is to give it 100% validity, which it may, or may not, deserve.
syi
The Velonews reports close of the hearing:
http://www.velonews.com/article/73612/landis-hearing-closes-in-new-york
They say the UCI took Landis' TdF "crown" but I still wonder what ASO will do when the CAS rules for Landis.
It's pure speculation, but given their comments in the past, I would think they would say that based on the French AFLD "investigation" they will not change their position and OP will still be the 2006 winner. It's an indefensible position given the way cycling, and doping control, us supposed to be organized, but that won't stop them. But again, pure speculation.
syi
I'm also a bit mystified as to how that particular problem will be resolved. Surely the UCI will have an opinion on that because if Floyd wins, by their rules, he will be the official winner of the TdF.
No doubt another potential bun fight between the UCI and ASO. This is what happens when you start making up rules on the fly. In this respect, both Oscar and Floyd have been poorly served by the system.
"Oh what a tangled web we weave ..."
Post a Comment