Larry's Curb Your Anticipation: Epilogue
Up to the Introduction; back to part 14; to the Table of Contents.
Larry:
The law is something more than just what the powers that be say it is. The law is something more than what the judges say it is, and it's something more than what proves to be practically useful.
Otherwise, I could have written two sentences:
1. If your "B" sample was analyzed by the same person who analyzed your "A" sample, there's no AAF.
2. Otherwise, you're screwed.
[MORE]
Rules are rules. They have a language and a meaning separate and apart from how they are applied. 500 years ago, wars were fought over whether the Bible would be published in the vernacular, because the powers that be knew that words have their own power. 80 years after the Declaration of Independence, some "idiots" read that all men were created equal, and decided that meant the end of slavery. Yeah, it's laughable to discuss professional cycling in the same breath with Reformation and Abolition, but the principle is a big one even if the questions we consider here are not as big.
"Stop" does not mean "go", "war" does not mean "peace", "freedom" does not mean "slavery", no matter what anyone else might say and how much power they might have.
It's interesting. In my preface to this series, I warned everyone that they'd come to the end of the series and wonder what it was all supposed to be about. And I've been thinking about what I might say if I was asked to write an epilogue to this opus.
Here's why I wrote the series. What I wanted was for people to read the Hamilton case, say "that's full of shit", and be able to employ the combined logical power of science and law to back their argument.
I wanted Mr. Idiot to read what the FL arbitrators said about matrix interference, say "that's full of shit", and use the combined logical power of science and law to back his argument.
If my article did what it was supposed to do, you guys can make your arguments not by plucking ISL rules from here and there, but by employing the ISL as an coherent whole, as a document with a logic and a purpose connected not to the exigencies of WADA World, but to the imperatives of science.
I wanted to put the lawyers and the scientists on the same page. There are many things wrong with the ISL, but even WADA World could not completely screw up a document that derives from ISO 17025 and is intended to provide standards for scientists. I was tired of hearing from you, and Ali, and Mr. Idiot, how the science and the law are at cross-purposes. If my article did what it was supposed to do, then you can now read the ISL (at least the portions of the ISL covered in the article) consistently with sound principles of good science.
I'm relatively confident that the article did not manage to do all I wanted it to do. I'm relatively confident that my article is a cautious step in the right direction. I'm reasonably happy with that.
TBV:
I wish that Larry is right, that actually understanding the regulations and rules would lead to an ability to argue them successfully given a set of facts. I'm afraid that in the geo-political circus that seems to be the environment here, that will not be the case. In honesty, we know Atticus Finch did not have a good chance on appeal with Tom Robinson, as much as we might wish it were otherwise, and as much as belief in the law might lead us to hope.
I'm very pleased to have Larry's work shed light into the rules that seem so likely to be misinterpreted. If you are failing to provide completely clear answers, it is because the situation has been made intentionally murky by those who wrote the rules and interpret them.
Thanks to Larry for the piece, and to those who've taken the time to get through all the pieces
Should Larry work up the steam to look at quality control, we'll be happy to run that as well.
D-47 and counting.
Up to the Introduction; back to part 14; to the Table of Contents
25 comments:
Larry,
Your opus is a very good piece of work - even if it is a bit like Chinese food in that I am already hungry for more. After reading it, and many other posts and comments on this site, I am led to the conclusion that taken as a whole all of the rules which govern and guide WADA and it's accredited labs make for neither good law nor good science. Heaven help the athlete, innocent or otherwise, that runs afoul of their messianic mission to rid sports of PEDs.
bill mc, thanks! But I can't promise that this particular take-out service will open again any time soon.
Memo to TBV: Atticus Finch WON. "To Kill A Mockingbird" sold 37 zillion copies, and Gregory Peck won the academy award.
I must have missed the happy ending for Tom Robinson and his family.
And the sequel has been a long time coming.
non-sanguinely,
TBV
TBV, if you read "To Kill A Mockingbird", and you cry when Tom Robinson dies, then Tom won. It cost him his life, but he won.
His fight continues, and we've made progress, though the fight is far from over.
If you're a realist, then I don't know how much more you can realistically ask for: progress in a good fight for a good cause. To expect more betrays the (good) heart of the idealist. Some of my best friends are idealists. A few of them like to pretend to be realists, but I know better, and I like them anyway.
Hamilton's case WAS full of shit.
The guy busted for using the baldness drug... that case was FULL OF SHIT.
Actually, I think Heras' and Perez' cases were FULL OF SHIT.
It all comes back to what I've been saying for 3 years or more... The line from "A Man for All Seaons" 'And if you were to chase the devil in to the woods, and if you were to cut down all the trees in the forest in your zeal for persecution, when, after you had cut down the last tree (the rules, or the laws), WHERE, WHERE would you then HIDE?'
Ladies and gents, I'm not an attorney, I'm not even a scientist. But the ADA's efforts to persecute when stepping over their own rules, or other ISO rules, is unfathomable. I can only pray that nothing close ever happens to myself or my athletes.
Thanks again for your incredible attention to detail, and to LOGIC. I think we all met on that 'other' forum, but the witch-burners have taken over in there, and it's good to get some clear-headed thinking in this arena.
Larry,
Absolutely spectacular piece of work. I've marveled for a long time at how guys like you and Mike could spend the time in the documents that you do. To write some thing like this in addition to that is WAY beyond the call. I hope I can buy you a beer one day!
I have to tell you that I'm still digesting it. About half way through it, I realized that there was as much (or more) to the legal strategy in the FL case as there was to figuring out if the science was done correctly. This kind of depressed me, because it shouldn't be that way. As a guy with a science background, if I were accused of doping, I'd want to see their method validation documents, their SOP, their documentation showing that they followed their SOP in determining the results of my test, and the results of the test. I think if the rules were structured to that they had to provide this for all cases, the cases would adjudicate in a much fairer, transparent, and straight-forward way. Right now, if my son turns out to be talented enough, I'd feel like steering him away from not only being a pro cyclist, but also an Olympic athelete. It's not worth the risk to your reputation if you can't defend yourself when wrongly accused.
Thanks again for providing this. You need to think about publishing this under a title like "WADA Doping Rules for Dummies" or something like that. Again, great work!
Regards
(BostonLondonTokyo) Larry - I've finally read the whole piece, and many thanks for the careful and thoughtful writing. You've made me understand a number of key issues that I didn't before - particularly 1) the meaning of 'fit for purpose' and 2) the seeming contradiction of WADA's assumption of compliance based on WADA's accredidation (ie, WADA assumes labs are working up to code because they have been accredited.) I also found this piece very interesting:
However, we should recognize that it is a difficult proposition for an athlete to prove that a test method was not properly validated. Any such proof requires the athlete to challenge the science surrounding the test method, a tall order under the best of circumstances. This task is made even more difficult by the fact that the WADA rules presume the validity of all WADA lab methods, and that the documentation of the lab’s method validation will not be included in the lab document package provided to the athlete.
I wasn't previously aware that the methods were not required to be part of a lab results document package. Whether this is fair or not is another topic, but it does speak to what is actually required by a lab.
What is interesting to me is understanding the relationship between a lab and WADA. From your article, I can see now how a lab can produce results that have passed the 'fit for purpose' requirement (which would mean compliant with WADA) BUT those results could be questioned in a different context (ie, a non-WADA context.) In that context (which is arguably the only context that ought to be considered in the FL case) the lab cannot take all the heat. This is a point I've tried to make on the board a few times, but lacked (and still lack) the language to express what I mean. We can certainly argue whether standards of specificity are generally relevant to evaluating 'true' results, but that's very separate from seeing things in the context of WADA.
Further, I think WADA is an issue that could be discussed more thoroughly, nicely raised as a 'talking point' from your article. I know WADA has already been talked to death, but your piece does unintentionally make us think about who watches who - I know Rant has discussed this as well. For many of us outside of the science of cycling, it would be very easy to say that WADA can't be trusted to make its own rules and police its own rules at the same time. But is this really the case? If we believe that every organization that has a mission statement is truly non-political and has no agenda whatsoever, yes, we can say that an organization can police itself. But if WADA were to step aside, and allow another body to police its labs, who would that possibly be? Who would select those people who are qualified? And aren't the people who make the decision of qualification PROBABLY alreay have a relationship with WADA? I think the discussion of WADA being policed by another agency is moot - the experts in the field (most of them) are peers, share information, and work generally for the betterment of scientific development and the futhering of research. WADA is (like it or not) a part of that. It seems to imply (to me) that WADA doesn't need to be disbanned, but it needs far better internal audit.
I'm moving way off topic, but I think in a way this was the intention of your article - it DOES bring to mind so many other issues. As far as leveling the playing field between the science and the law, I think your effort is admirable and very successful. Many thanks!
dbob, John aka BLT -
Thank you both!
dbob, it's probably my perspective, but I'm personally more disappointed with the scientists and the science establishment than I am with the lawyers and the legal apparatus. The law is complicated, in part because fairness requires complexity, and because when it comes to people, it's never a simple matter to determine the truth (if you have teenage children, then you don't need me to tell you this). Plus, the law is an adversary system, so I expect to see two sides fighting it out. I'm not always crazy about the adversarial process, except that it's better than any other process we've come up with.
In contrast, what happened with the scientists? We had some of the top scientists show up in Pepperdine. Did you hear their testimony? Why didn't the truth emerge from the combined testimony of the scientists? The scientists lined up on one side of the case or the other, depending on who paid them. And with very few exceptions, the only scientists we've heard from are the ones who were paid. Honestly, dbob: if the WADA rules interfere with the ability of science to decide these cases (and this IS what I believe), why doesn't the science community SAY something about this?
I'll also point out that the same science system that accredits the LNDD accredits the labs that look at my blood work. I've heard many say here, would I feel comfortable having the LNDD handle the testing for a sick family member? Well, the LNDD is ISO 17025 accredited, and so are the labs responsible for the long-term health of everyone I know. I try not to think about this too much.
I'm trying to get past the prevailing opinion, which (in simplest terms) is "science, good; law, bad." I'm trying to push us all to think about this question in terms of RULES. Science and law are both rule-based enterprises. It's possible for science and law to work together, for the law rules to be based on sound scientific principles and for the science rules to be based on sound legal principles of fairness and justice. All this starts with understanding the rules.
BLT, the laboratory documentation package (LDP) must contain a "description" of the lab's screening test and confirmation procedures. TBV has indicated that LNDD may have provided some documentation of method validation, but I don't think they were required to do so.
The issues you're raising are very good ones. I know, because I ducked most of them when I wrote the opus. It would be terrific to have a second opus on the question of lab accreditation, and the roles played by WADA, ISO and the various lab accreditation bodies. We're all dependent on the work performed by these labs - not just athletes, but all of us. Ultimately, the legal system cannot play a significant role in overseeing the quality of lab work. If you're of the belief (as I am) that the LNDD did a shoddy job in the Landis case, then one place we might look for improvement is to the accreditation process.
And you're right, the ultimate responsibility here belongs to WADA. WADA sets the tone and direction. They are the experts and the ultimate authority for the global anti-doping effort in sport. We need to remember that WADA is a relatively new agency, but I am personally very disappointed with their efforts so far. You question whether WADA can be trusted to make its own rules and police its own rules at the same time, and that's a great question. A parallel question is whether WADA can remain true to the imperatives of science while at the same time functioning as a political organization - balancing the needs of its constituent members, lobbying for money and influence, fighting for turf, and so forth. Probably my biggest complaint against WADA is the lack of any coherent vision for the future, any sense for a direction in which we should head.
Larry wrote back near the beginning of this magnificent work:
ISO 17025 provides rules for how labs should operate in order to produce consistently valid results.
With that in mind, Larry, I wonder if you could answer a basic question for me:
Where do the "fit for purpose" requirements come into play? Is that at the level of ISO 17025 or is that at the level of the ISL and TDs?
In other words, if a particular SOP is not "fit for purpose" would that keep a lab from getting ISO certification?
syi
As far as I can tell, there is no FFP review by any body except the lab itself. Because it is accredited, it appears to be given that power. There's sort of an implication you could challenge it in arbitration, but that is going to be hard to do with the documentation that will be provided.
It's more, "looks good to me, trust us."
TBV
But why would that depend on extra documentation? Couldn't you just look at the existing record of what the lab did, and say "Look, here is a hole is their process. Given this hole, the results could be incorrect because of x, y, or z" ?
Would it carry any weight for the lab/USADA to say, "Well, we have ISO certification, so our process is legit" ?
syi
Then you get the, "they are accredited, and no mistake they made affects the result" song and dance.
Unless the hearing panel wants to get into FFP, that is where it is going to sit, unless you find a procedural ISL violation.
An FFP review is a second order set of issues, and seems to me to be very hard to bring up in the context of a specific case under these rules. In real court, it takes some serious digging with discovery power to find things out, and that isn't available here. (On either side -- ADAs can't get search warrants).
TBV
From majority decision, paragraph 162:
As was previously described, the identification criteria the Lab uses must be documented in order
to obtain the ISO 17025 accreditation. That accreditation follows the principles that a Lab should do what they have written down; write down exactly what
they do and that in this case the identification criteria are fit for the purpose for which they are to be used. The identification for T and E by the ISO 17025 accreditation is the way in which the Lab carried out the confirmation tests.
Does that mean the Lab has complied with the TD2003IDCR because it holds the ISO 17025 accreditation?
Given the fact the panel threw out the T/E confirmation test, because it did not conform to TD2003IDCR (one ion analyzed, rather than three), it would seem that the lab got their ISO accreditation in spite of the fact that their process was deemed not to be fit for purpose, at least as "fit for purpose" is described in the TD.
So why the language of the text I bolded above? What does that mean?
It seems to me that USADA can make no claims about "fit for purpose" based on ISO accreditation. I think you agree, TBV.
But then you are saying that you think there is no way to challenge "fit for purpose" short of a clear ISL / TD violation?
syi
syi -
Quick thoughts (well, quick for ME, anyway):
My argument is that an athlete accused of an AAF can challenge a lab's method validation, but that it's a tough challenge.
I agree, the lab cannot use ISO 17025 accreditation as a defense to a challenge that its methods were properly validated.
I think the language you cited from the Landis decision is pretty close to on point.
I agree with your statement that, if there is a hole in the lab's method (i.e., the lab did not look at the full mass spectrum data), then the athlete can pounce and say, this method is not fit for purpose. Of course, the lab can come back and argue that there's really no hole in the lab's method (i.e., the lab's sample preparation is so good, there's no need to look at the full mass spectrum data). To resolve this kind of argument, you might want to see the lab's method validation studies. Otherwise you're forced to argue that a "hole" exists entirely from theory, or from the prevailing practice of the scientific community, and I'm not sure how well either argument would work.
In the opus, I specifically stated that I was not going to discuss the lab accreditation process. I don't work with labs in my real life, so I'm reluctant to talk about a process that I've never experienced -- particularly when I know there are others in this community who HAVE experienced an ISO audit. There is also the question of ISO audits and WADA audits, and what each is supposed to accomplish.
So syi, I can't answer the question whether the ISO would pull a lab's accreditation if the ISO found that the lab's methods were not fit for purpose. I think that the question of lab method "fitness for purpose" would be part of an ISO audit, but I know that TBV thinks otherwise, and I defer to him on this point.
Mike, I DO want to get into this question with you in some depth. The opus began with the assertion made by you, Duck and others that LNDD's method should have included examination of complete mass spectrum data. We're now in a position to reexamine this assertion.
Okay. Let see if we can agree on the question.
Obviously, in order for the assay to be fit for purpose, the assay would have to assure that when they measure the CIR of a peak they are only measuring the substance of interest, and nothing else. So the basic question is:
"What does LNDD do to assure that when they measure the CIR of a peak with IRMS they are only measuring the metabolite of interest and nothing else?"
Agreed?
Larry,
Thank you for taking the time and effort to write your Opus. Thank both you and TbV for sharing it with us and making it widely available for review/comment.
I share some of the sentiments expressed earlier. Like Whareagle, I’m offended by the process, as a whole, and am certain I don’t wish it/its capricious nature upon any athlete I have even an ounce of respect for. Hell, I don’t wish it on anyone. I’m still digesting your Opus, and like dailbob, I have serious concerns about a system that makes it so difficult/nearly impossible for athletes to defend themselves. For athletes with other viable options, steering clear of pro cycling and the Olympic movement might seem wise? That’s a sad commentary in, and of, itself. For the Olympic movement and all of its lofty ideals, it’s sadly ironic that a system set up to promote fairer competition (ped testing) would abandon a good portion of the very ideals the Olympics stand for. Another bit of bad commentary. Like you, I’m more disappointed in the scientists than the lawyers. There are some exceptions. On the lawyer’s side, I naturally blame Dick Pound and any of his cronies that helped write the WADA code, specifically the parts that tend to treat the athletes like chattel. It’s hard to excuse them. The lawyers writing the code have the intelligence, education, and training to have known they were ramming an unjust system down the throats of the athletes. The scientists involved are guilty of not standing up for the science. In my mind, that makes them something less than scientists. If the science is sound, it can stand scrutiny. Political, financial, or popular gain are poor reasons for an ethical person to compromise the very basis of their chosen profession.
Thanks again. I have some re-reading and more digesting to do.
"What does LNDD do to assure that when they measure the CIR of a peak with IRMS they are only measuring the metabolite of interest and nothing else?"
Agreed?
Yup.
TBV
jrd, thank you for the nice words and for taking the time to read the opus. I think your message is a great take on what I wrote. Thanks again.
Mike, I'd rephrase your basic question, using the language of method validation from the opus. I'd make it a two-part question. You might consider the rephrased question to be essentially the same as the question you posed. I encourage you to rephrase my rephrased question as you think necessary or advisable (you too, TBV).
"Both ISO 17025 and the ISL require lab method validation to determine and document method specificity, defined as the ability of the method to measure only the presence of what the method is supposed to measure, in the presence of potential interferences.
1. How did LNDD determine and document the specificity of its method of measuring the CIR of IRMS peaks? (this is really the most direct question to ask, given that our concern here arises in the context of method validation)
2. Assuming for the moment that LNDD would not agree to answer question no. 1, we move to question no. 2: does the LNDD method for measuring the CIR of IRMS peaks adequately determine specificity?"
We might then jump to a third question, which is whether a lab can adequately determine the specificity of a CIR peak without analysis of complete mass spectrum data.
Larry,
Honestly, dbob: if the WADA rules interfere with the ability of science to decide these cases (and this IS what I believe), why doesn't the science community SAY something about this?
I don't have a good answer for this, but I can offer some thoughts.
The one organization that came to my mind when I read your question was AOAC International (formerly the Association of Official Analytical Chemists). However, this group is a professional member organization that exists to provide continuing education, method standardization, etc., and it has no diciplinary arm. Also, it's just not part of their mission to be a watch-dog for incompetent labs or incongruous testimony.
In the state where I live, we have a Board of Examiners (or equivalent)for physicians, lawers, and engineers. These are part of the state government, and they have diciplinary authority to take your license. However, there is nothing like this for scientists or analytical chemists. If we're incompetent, we just get fired. Net, to answer your question, there is nothing that exists (structurally) to take scientists/chemists to task for behavior displayed during the FL trial. For what it's worth, I viewed them as tools (or maybe weapons?)in the adversarial system you describe (note that I'm not justifying them not stating the complete truth).
On a slightly different, but related, subject, I then asked myself, what would be the equivalent of "firing" LNDD? The best thing I could come up with is that they lose their ISO 17025 accredidation. As you are aware, this standard spells out in excruciating detail what a lab is required to do in order to be called competent (using white-out isn't one of them). I think this is the closest we could get to the scientific community saying something, i.e., it's the only structure that exists that I can see to get at what you're asking.
Just some thoughts.
Regards
dbob, thanks for the information. It's a bit surprising that the science community does not have a way to communicate to the rest of us when junk is being passed off as good science. It's sad - if Brenna had talked about pattern matching in a peer-reviewed science journal, I expect he would have received a lot of criticism. But if he says stuff in an arbitration, there's no forum to take him to task.
Okay Larry and other folks, here is my shot at the specificity question in the Post-Larry's-Opus era. It is not as well written as it could be. Sorry.
Preface - Larry wrote:"Both ISO 17025 and the ISL require lab method validation to determine and document method specificity, defined as the ability of the method to measure only the presence of what the method is supposed to measure, in the presence of potential interferences.
In that regard, I don't think we have any public documents that talk about method validation, at least wrt to specificity. We do know from USADA's Proposed Findings of Fact (PFF) after the first hearing that the sides had access to some of this material through Exhibit 26 (see USADA's PFF p. 52-54), but I don't think that exhibit has been made public.
In USADA's Proposed Findings of Fact after the first hearing they seem to make a big deal of the COFRAC/ISO accreditation of LNDD's IRMS process, but it is not at all clear to me what ISO accreditation really means. I suspect that USADA claims greater weight for ISO accreditation than it really deserves.
(Indeed, paragraphs 52-59 of USADA's PFF are fascinating, lawyerly sleight-of-hand. Read it and see if you agree. Available here, listed just above "various documents." Paragraph 56 is particularly devious, IMHO.)
Here's the payoff from Larry's opus: ISO 17025 5.4.5.1 speaks to what it means to validate a method: "Validation is the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled." So they really do have to do a study of the results of their method. And 5.4.5.2 says "The laboratory shall validate... laboratory designed/developed methods [note: which is what we have here]...to confirm that the methods are fit for the intended use."
(Obviously an important question is whether ISO accreditation means that ISO has confirmed the results of the validation study, or whether it means that ISO has confirmed that a validation study has been done, or whether it means something else. It seems to me that it means ISO has confirmed that a validation study has been done, but not that they have confirmed the results of the validation study. Anybody who knows more about ISO care to chime in?)
With that background, the first question:
1. How did LNDD determine and document the specificity of its method of measuring the CIR of IRMS peaks? (this is really the most direct question to ask, given that our concern here arises in the context of method validation)
Especially without the above mentioned Exhibit 26, we don't know. Did they do an internal study of some statistically significant number of samples, using their SOP, in which they determined whether they had specificity in the GCMS and then the IRMS? We don't know. Is this what would be checked in an ISO 17025 accreditation? If they did do that, did they use real, "live" urine samples (a "dirty matrix") rather than some sort of spiked blank? We don't know. If they did do such a study, is there any way it would account for the particularly "dirty" matrix of someone who had been injected with other banned (but TUE'd) substances? I doubt it. If they did such a study of their method, where is the documentation? In Exhibit 26? Never revealed? We don't know.
With so much we don't know, I suspect it is time to move on to Larry's second question: 2. Assuming for the moment that LNDD would not agree to answer question no. 1, we move to question no. 2: does the LNDD method for measuring the CIR of IRMS peaks adequately determine specificity?"
Here also, I think there are some key things we don't know. I think we have to consider three possibilities. But first, an important note. I am now convinced that TD2003IDCR is NOT relevant to this specificity question. The title of TD2003IDCR is "Identification Criteria for Qualitative Assays Incorporating Chromatography and Mass Spectrometry." It sets requirements for how a substance is to be adequately identified. There is no question that LNDD's method is able to identify the peaks of interest in the GCMS (but it's more questionable wrt to IRMS). They can say "This peak has 5aA in it," (identification) without being able to say (honestly) "This peak has ONLY 5aA in it" (specificity). So I don't think TD2003IDCR is relevant to the specificity question.
In fact, then, I think there is no detailed technical guidance in ISO or ISL or any Technical Document which talks about how to achieve specificity. That is left up to the lab.
So, the three possibilities:
One, the method LNDD uses to ensure specificity is simple chromatographic separation of peaks. If the peak looks like there is nothing else in it, then there is nothing else in it.
Two, the method LNDD uses to ensure specificity relies upon a "peak purity" subroutine in their GC/C/IRMS software. At least in current versions of software, you can basically select "check peak purity" from a drop down menu, and it will tell you if you have ONLY the substance of interest in that peak.
Three, the method LNDD uses to ensure specificity relies upon an examination of the data generated by a full mass spectral scan of the peak of interest. This would tell you EVERY ion that is present in the peak. If there were ions present that could not come from the substance of interest, then you would know there was other stuff present.
There may be other possibilities that I don't know about.
So (if those are the three options) what method does LNDD use to assure specificity? Well, based on information that has been made public, the only reasonable conclusion is that they use the first method and rely on simple chromatographic separation. If so, that is incredibly weak evidence of specificity, and I really doubt it would satisfy the requirement that the method assure specificity.
Re: the second possibility, if LNDD's version of the software did have a "peak purity" subroutine, you would think we would have heard about it by now. It would end the whole discussion about "matrix interference" along with this specificity question. So, either their software does not have that ability, or it has it, but they didn't use it for some reason, or they used it, but there is no evidence of it.
Re: the third possibility, we know that LNDD did do a low resolution full mass spectral scan. While this would not be absolute proof of specificity (because it was "low resolution"), it could be pretty good. However, we have no public evidence that they used this full mass spectral scan to check specificity (to check for peak purity) because that full scan mass spec data has not been made public. It is possible it was erased (why?) just before the reprocessing of the B samples took place in April.
Now, in the LDP, LNDD offers information that is loosely related to specificity. It offers (for example at USADA 321-322), an analysis of three ions for each metabolite of interest. But these three ions do not guarantee specificity/purity, they "merely" assure identity. There is definitely 5aA in the GCMS peak labeled 5aA (again, that does not speak the IRMS peak so named). But those three ions do not tell us that there is ONLY 5aA in that peak. There may be other ions present in that peak which are not analyzed in that chromatogram. If those other ions went into the IRMS they would be indistinguishable from the 5aA ions and would be counted right along with them.
So, as so far revealed in the public record, we see no evidence of a step in LNDD's method which assures specificity. So from what we know (and we don't know everything) the method is not "fit for the intended use" according to ISO 5.4.5.2, and is not "fit for purpose" according to ISL 5.4.4.2.1, specificity bullet.
syi
I think the part about standards for method validation is important. We know USADA has placed great weight (as Swim cites) on ISO accreditation being unquestionable in the context of the case, but seems to shingle past the edge of the roof by claiming ISO has explicitly accredited the LNDD IRMS method as fit for purpose.
I don't believe they do that, or did that, making FFP review fair game in arbitration.
This is worth continued exploration, with cites to supporting documents.
TBV
syi -
Great post.
I don't have time at the moment to give you the detailed reply that your post deserves. I will look at the PFF later today, I hope.
I agree with how you've structured your analysis, which is key (I think) to getting to the right result. I agree that we'll probably never know how LNDD did its method validation.
Onto question no. 2: the steps in LNDD's CIR method associated with specificity are (I think) sample preparation and determination of the best possible GC conditions. Obviously, if they could do the sample preparation to eliminate all possible interfering substances, then specificity would not be a problem. Just as obviously, they can't do that kind of sample preparation. But you have to take sample preparation into account as a specificity technique.
You should take a look at the article that "M" always cited, the one with the chromatograph that he wanted TBV and Ali to consider in their "idiot's" guides. In that article, Shackleton talks about how his initial chromatographic conditions failed to achieve specificity, so he switched GC columns and got the specificity he was looking for.
I think that these techniques are the only ones available to ACHIEVE specificity. The complete mass spectrum data is a way to CHECK that you've achieved specificity. Your argument is that the CHECK is necessary in order for the method to be fit for purpose ... and I'm wondering whether we're now dealing with a quality control issue.
Mike, these are quick thoughts, I'll try to provide you with something better later.
(Larry, I wrote this before I read your comments. I'll respond to those later.)
Here is some further background. It's not all we would want, but FWIW:
At the hearing, USADA presented some documents as Exhibit 26. I don't think that document has been made public yet, but from the hearing and USADA's Proposed Findings of Fact after the hearing, we can tell a little about what was in it. It includes some documentation about LNDD's COFRAC/ISO accreditation, but perhaps not very much. I suspect that USADA only submitted as evidence a section of the summary of whatever accreditation document they presented. In their opening statement at the hearing, USADA said:
At the end of the day, Respondent is asking the Panel to reject the laboratory's results in this case because of what he claims
is the laboratory's flawed methodology; however, under the scheme of the World Anti-Doping Code, ISO has already done the job of evaluating the Paris laboratory's methods. The Panel should not try to put itself in ISO's shoes.
Here's what ISO said in its audit report after inspecting the Paris laboratory five months before Respondent's sample was analyzed. This is their general comment on what they were going to do. This particular piece of the report is important because it shows that they specifically reviewed the Paris laboratory's IRMS method. This part of the report is important because it says, and this is the issue in this case, is whether Paris laboratory complied with international standard for laboratories, the technical document on IRMS and T/E, and the technical document on laboratory chain of custody, that they paid -- the ISO inspector paid particular attention to conformity with these three documents.
Their conclusion, with respect to chain of custody, was that chain of custody was conducted successfully and was considered very satisfactory.
Their conclusion with respect to the IRMS method was that it was well under control with maximum analytical precautions taken by the
section staff. Their summary was that LNDD had unquestionably had the capacity of implementing
good efficiency in the methods subject to review, which included the IRMS method.
They use very similar language after the hearing in the Proposed Findings of Fact.
Then during the hearing, see this. When Ayotte was testifying on direct examination, they asked her about the ISO accreditation:
Q. I will show you what is LNDD 0098,which is part of the ISO accreditation for the Paris lab, and would ask you to look at method
EC 31.
A. I see it.
Q. What does this mean --
MR. MC LAREN: Just a moment.
MR. BARNETT: Exhibit 26...
Q. What does it mean when you see this method listed in the ISO accreditation, this method EC 31?
A. EC 31 is the method that is for the determination...for the determination of the origin of metabolites and precursor of the
testosterone by GC/C-IRMS, delta carbon 13 -- because it has been specified -- and the following rows are just the extraction procedure
and the utilization of the IsoPrime MicroMass. So that means, since that method is under -- falls under their scope of accreditation, it has been audited and -- meaning reviewed carefully by auditors, and the paperwork has been reviewed, and it has deemed to be -- it was deemed by the ISO inspector to
be -- to be following the good quality management principle and the quality of a method following ISO. It has been found to be compliant with 17025 criteria, the norm. Was that clear? I'm sorry. I fumble.
Q. I understood.
Oh, yeah, perfectly clear! You fumble, really? Perhaps because you can't remember the exact way you were coached to answer this question so it sounded like you were saying the method was perfect without perjuring yourself?
Anyway, from that exchange I suspect that Exhibit 26 only included the summary statements that USADA (Young?) quoted in his opening remarks and that he asked Ayotte about.
It sounds to me like they are being VERY careful about what they claim for the ISO accreditation of Method 31.
As they should be! LNDD's ISO accreditation also applied to their T/E confirmation method. And that method was found by the majority of the panel to be non-compliant with TD2003IDCR. The majority decision says:
As was previously described, the identification criteria the Lab uses must be documented in order
to obtain the ISO 17025 accreditation. That accreditation follows the principles that a Lab should do what they have written down; write down exactly what they do and that in this case the identification criteria are fit for the purpose for which they are to be used. The identification for T and E by the ISO 17025
accreditation is the way in which the Lab carried out the confirmation tests.
Note: that last line is important. It specifically says that the way LNDD did the test was the way that was accredited by ISO. And then the arbs found the method non-compliant with the TD. So much for confirmation of methods!
The majority decision continues:
Does that mean the Lab has complied with the TD2003IDCR because it holds the
ISO 17025 accreditation?
163. The interplay of ISO17025 and Technical Documents is set out in Article 5.1 of the ISL. If an aspect of the testing is not specifically discussed in the ISL as expanded upon by the Technical Documents then the accreditation of ISO applies. Therefore, the Lab cannot reply to the alleged violation of TD2003IDCR by saying: “we have an ISO 17025 accreditation for what was done,” when the Technical Document specifically indicates that identification
criterion for confirmations requires at least three diagnostic ions.
Now our situation with specificity is different than that, because we do NOT have any real instruction in the ISL or any technical document which says how specificity is to be checked.
But let's slightly change the question at the beginning of that last quoted text: "Does that mean the Lab has complied with the ISL because it holds the ISO 17025 accreditation?"
The answer to the original question was "no," so the answer to this question must be "no" also.
That seems to contradict what USADA said in their opening remarks at the hearing, what Ayotte sort of said during the hearing, and what USADA said in their post hearing Proposed Findings of Fact. Or at least it contradicts what they seem to want us to think they said.
syi
Post a Comment