Larry's Curb Your Anticipation: Table of Contents

Larry's Curb Your Anticipation

A series of posts investigating the rules behind the validation of methods used by WADA Laboratories.

• Preface
• Introduction: Look at the Whole
  
• Part 1: Some Questions to Address
  
• Part 2: The WADA ISL
  
• Part 3: Introducing ISO 17025
  
• Part 4: Key Parts of ISO 17025
  
• Part 5: Rules Governing Test Development and Validation.
  
• Part 6: Fitness for Purpose
  
• Part 7: Uncertainty in Fitness for Purpose
  
• Part 8: Further Uncertainty, reproducibility and repeatability
  
• Part 9: Specificity/Selectivity in Fitness for Purpose
  
• Part 10: Specificity/Selectivity in the ISL
  
• Part 11: Methods v. Application
  
• Part 12: Arguing about validation in a case
  
• Part 13: Misapplication of validation rules
  
• Part 14: Quality Control matters.
  
• Epilogue

Full Post with Comments...

Larry's Curb Your Anticipation: Epilogue

Up to the Introduction; back to part 14; to the Table of Contents.

By Commenter Larry and TBV
(Extracted and modified from some comments)

Larry:

The law is something more than just what the powers that be say it is. The law is something more than what the judges say it is, and it's something more than what proves to be practically useful.

Otherwise, I could have written two sentences:

1. If your "B" sample was analyzed by the same person who analyzed your "A" sample, there's no AAF.

2. Otherwise, you're screwed.

[MORE]



Rules are rules. They have a language and a meaning separate and apart from how they are applied. 500 years ago, wars were fought over whether the Bible would be published in the vernacular, because the powers that be knew that words have their own power. 80 years after the Declaration of Independence, some "idiots" read that all men were created equal, and decided that meant the end of slavery. Yeah, it's laughable to discuss professional cycling in the same breath with Reformation and Abolition, but the principle is a big one even if the questions we consider here are not as big.

"Stop" does not mean "go", "war" does not mean "peace", "freedom" does not mean "slavery", no matter what anyone else might say and how much power they might have.

It's interesting. In my preface to this series, I warned everyone that they'd come to the end of the series and wonder what it was all supposed to be about. And I've been thinking about what I might say if I was asked to write an epilogue to this opus.

Here's why I wrote the series. What I wanted was for people to read the Hamilton case, say "that's full of shit", and be able to employ the combined logical power of science and law to back their argument.

I wanted Mr. Idiot to read what the FL arbitrators said about matrix interference, say "that's full of shit", and use the combined logical power of science and law to back his argument.

If my article did what it was supposed to do, you guys can make your arguments not by plucking ISL rules from here and there, but by employing the ISL as an coherent whole, as a document with a logic and a purpose connected not to the exigencies of WADA World, but to the imperatives of science.

I wanted to put the lawyers and the scientists on the same page. There are many things wrong with the ISL, but even WADA World could not completely screw up a document that derives from ISO 17025 and is intended to provide standards for scientists. I was tired of hearing from you, and Ali, and Mr. Idiot, how the science and the law are at cross-purposes. If my article did what it was supposed to do, then you can now read the ISL (at least the portions of the ISL covered in the article) consistently with sound principles of good science.

I'm relatively confident that the article did not manage to do all I wanted it to do. I'm relatively confident that my article is a cautious step in the right direction. I'm reasonably happy with that.


TBV:

I wish that Larry is right, that actually understanding the regulations and rules would lead to an ability to argue them successfully given a set of facts. I'm afraid that in the geo-political circus that seems to be the environment here, that will not be the case. In honesty, we know Atticus Finch did not have a good chance on appeal with Tom Robinson, as much as we might wish it were otherwise, and as much as belief in the law might lead us to hope.

I'm very pleased to have Larry's work shed light into the rules that seem so likely to be misinterpreted. If you are failing to provide completely clear answers, it is because the situation has been made intentionally murky by those who wrote the rules and interpret them.

Thanks to Larry for the piece, and to those who've taken the time to get through all the pieces

Should Larry work up the steam to look at quality control, we'll be happy to run that as well.

D-47 and counting.


Up to the Introduction; back to part 14; to the Table of Contents

Full Post with Comments...

Larry's Curb Your Anticipation, Part 14: Quality Control

Up to the Introduction; back to part 13; on to the Epilogue.

By Commenter Larry

We conclude by observing that specific results are intended to be validated by quality control, a major topic in and of itself.

[MORE]

• Are there rules in the ISL and in ISO 17025 that are addressed to determining the validity of a test result? Yes. There is at least one more concept under the ISL and ISO 17025 that we need to discuss: the concept of quality control. WADA labs are required to employ properly validated methods, but they are also required to use these methods correctly.
  

The concept of quality control is at least as complicated as the concept of method validation. It will require me to add a Part II to this article. It will probably take me longer to write that Part than it took me to write this one. Your friendly patience would be appreciated while your faithful (if somewhat overwhelmed and overmatched) author gears up for the effort to write Part II.

That's the end of this walk through method validation. We hope we've illuminated the important rules and regulations that ought to control the acceptance of the reliability of tests in the WADA system.


Up to the Introduction; back to part 13; on to the Epilogue.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 13: Misapplication

Up to the Introduction; back to part 12; on to part 14.

By Commenter Larry

At the same time, there are some things that would be misapplication of method validation rules to the assessment of results. The Majority of the AAA Panel seems to have made mistakes of this sort.

[MORE]

• Can the criteria for method validation be taken outside of the context of method validation, and used to judge the validity of a test result? The answer to this question seems to be a qualified “yes”. For example, in the FL case, FL argued that the LNDD violated ISL Rule 5.4.4.2.1 by failing to properly generate chromatograms that avoided matrix interference. During the arbitration hearing, both sides called expert witnesses to testify to the quality and reliability of these chromatograms. Predictably, the witnesses for USADA stated that these chromatograms were reliable, and the witnesses for FL disagreed. The majority of arbitrators ruled that these chromatograms “perhaps could have been better”, but ultimately found that the chromatograms were “fit for the purpose” and that the chromatography “unquestionably indicates the presence of exogenous testosterone in [FL’s] “A” and “B” samples.” In reaching this ruling, the majority arbitrators made the following statement, a statement that (in my opinion) completely misconstrues the nature, purpose and intent of the ISL:

In applying the language of the ISL what is required is that the “method should avoid interference.” The language is not mandatory. Had the drafters intended that matrix interference be avoided it would require wording such as “shall” or “must”. For this Panel to accept the submissions of the Respondent that matrix interference must be avoided would be a misconstruction of ISL 5.4.4.2.1. Dr. Ayotte confirms this statement in noting that a laboratory does not violate Article 5.4.4.2.1 of the ISL just because it produces a chromatogram that contains matrix interference. Therefore, even where matrix interference has occurred in the Stage 17 chromatograms it would not amount to a violation of the ISL. (FL decision paragraph 240.)

The majority’s reasoning here illustrates the danger inherent in reading rules out of context. The majority managed to interpret ISL Rule 5.4.4.2.1 to exclude any requirement to consider matrix interference – a reading that is possible only by failing to read the entire rule! (As we’ve seen, this rule includes a requirement that lab methods must achieve specificity, and the concept of “specificity” includes within it the need to avoid matrix interference.) But the majority’s reasoning is faulty on a more fundamental level, as the majority never considered ISL Rule 5.4.4.2.1 as applying to the lab’s method validation effort. In the context of method validation, a statement that a method “should avoid” matrix interference takes on a different meaning than the one found by the majority arbitrators. In the context of method validation, this statement would be read in light of the overall goal of method validation: to make a reasonable effort (within the constraints imposed by the cost and time required for the effort) to determine that a method is “fit for purpose”. In this context, the “should avoid interference” language in ISL Rule 5.4.4.2.1 might be interpreted to mean “do the best you can to avoid interference” or “make a reasonable effort to avoid interference”, consistent with the overall principle that a method cannot possibly be “fit for purpose” if it cannot provide us with a reasonable assurance that the method is measuring what we want it to measure.

So, let’s return to the question posed at the beginning of this bullet point: can the criteria for method validation be taken outside of the context of method validation, and used to judge the validity of a test result? The answer is “yes”, at least with respect to some of these criteria, but we need to be careful in how we do this. To be certain, if we want to assess the validity of a particular test result, we would want to consider criteria such as accuracy, uncertainty, specificity and interference. But we should avoid applying the specific language of ISL method validation rules embodying these criteria to our assessment of a particular test result. Instead, we should look to other rules within the ISL and ISO 17025 – rules that were written for the purpose of assessing a method result rather than the method itself – to provide us with the necessary guidelines for determining the validity of a particular method result.

Up to the Introduction; back to part 12; on to part 14.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 12:You should be able to argue...

Up to the Introduction; back to part 11 on to part 13.

By Commenter Larry

Now, some bullets about things you probably can argue, even if they may not be well received.

[MORE]

• Can we use method validation criteria as criteria for the evaluation of the validity of a particular test result? For example, can we look at the difference between the original results of the FL stage 17 testing and the results from the subsequent EDF reprocessing, and say that this difference indicates a problem of repeatability or intermediate precision? This is a tricky question to answer. From our discussion above, we understand that criteria such as repeatability and intermediate precision are method validation criteria. Simply because a method produces a bad result does not mean that the method is invalid or should be reevaluated. However, we also understand that method validation is an ongoing process, and that a lab’s quality control efforts may indicate the need to revalidate a method. So it is important (perhaps even required under the ISL and ISO 17025) for a lab to review its test method results against the various method validation criteria, to see if revalidation of the method is necessary or advisable.

• However, the failure of a particular lab result to comply with one or more method validation criteria does not necessarily mean that the result is invalid and should be thrown out. There is a difference between the criteria for evaluating test methods and the criteria for evaluating test results. To see this difference, all we need to do is to consider some of the method validation criteria discussed above. For example, the criterion of traceability addresses the test method only – either the method is tied to a known standard or it is not. The criterion of intermediate precision can be used to judge a test result only if the test is repeated at a different time or with different equipment or a different lab technician, but repeated testing of the same sample is not the norm.

• Can the criteria for method validation be used by an athlete to prove an ISL departure? Absolutely. This was the basis for the argument made by Tyler Hamilton (discussed above) in his anti-doping case. It is an ISL departure if an AAF is based on a method that was not properly validated. However, we should recognize that it is a difficult proposition for an athlete to prove that a test method was not properly validated. Any such proof requires the athlete to challenge the science surrounding the test method, a tall order under the best of circumstances. This task is made even more difficult by the fact that the WADA rules presume the validity of all WADA lab methods, and that the documentation of the lab’s method validation will not be included in the lab document package provided to the athlete. This difficulty may be the reason why the FL team seemed to avoid many of the criteria discussed above (in particular, specificity) in the case it made before the arbitrators last year.

Up to the Introduction; back to part 11 on to part 13.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 11: Methods v. Application

Up to the Introduction; back to part 10; on to part 12.

By Commenter Larry

We're going to start winding down, making some observations about the things we've discussed in previous parts with application to the Landis case. These aren't long, and each could easily become their own extended discussions, but TBV cut it off, so here they are as bullet points, a few at a time.

[MORE]


Let’s make some brief observations:

• One of our goals in this article is to place the ISL rules into a proper context. Here on TBV, we’ve frequently discussed ISL criteria such as specificity and interference, but we have not done so in the context of method validation. We need to understand that under the ISL, these criteria are method validation criteria. They speak to the fitness for purpose of a lab’s methods. So, we must be careful how we use these criteria. If we say, for example, that the chromatograms in the FL case lack the specificity required under ISL 5.4.4.2.1, we’re not saying that the lab failed to perform its test method correctly. We’re saying that there’s something wrong with the method itself. We’re saying that the method needs to be reevaluated, revised, or possibly thrown out and replaced with a better method.

• Let’s use two examples to illustrate this point. Both Duck and Mr. Idiot have argued that LNDD’s testing for exogenous testosterone should include analysis of complete mass spectrum data. This is an argument that goes to specificity; in effect, Duck and Mr. Idiot are questioning whether any method for testing for exogenous testosterone could have been properly validated if the method does not use mass spectrum data to determine specificity. In contrast, Ali has pointed out the wide variation between the results found by LNDD during its initial testing of the S17 sample, and the results reached later upon reanalysis of the S17 electronic data files (EDFs). Ali’s point here does not relate directly to method validation – Ali is not expressly questioning LNDD’s methods, but instead is pointing out an apparent failure of the lab to produce good results with its methods. The data pointed to by Ali may or may not indicate that LNDD’s method is flawed; the problem may lie elsewhere (for example, in LNDD’s sampling procedures, or with the training of LNDD’s personnel, or with LNDD’s equipment).

Up to the Introduction; back to part 10; on to part 12.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 10:Specificity/Selectivity meaning in the ISL

Up to the Introduction; back to part 9; on to part 11.

By Commenter Larry

We left our discussion in part 9 on the varying language in the ISL about specificity/selectivity, wondering how you can tell what really applies. We take note of an interesting distinction proposed by Mr. Idiot, but aren't sure it would be accepted.

[MORE]

I think that the ISL rules governing specificity are best understood in the context of our discussion of method validation. In this context, it is clear that selectivity must play a role in method validation, notwithstanding any permissive language we might find in ISL rules 5.4.4.2.1 and 5.4.4.2.2. It would make no sense to devote great effort to the determination of the accuracy of a test method, without devoting a roughly equivalent effort to determine that the method is measuring what it is supposed to measure. Understanding what we do about “fitness for purpose”, it would be impossible to argue that a test method lacking in selectivity could possibly be “fit for purpose.”

At the same time, it may be unreasonable to expect that WADA lab method validation should ensure complete selectivity. As we’ve stated repeatedly, the ISL is based on the assumption that standard methods are not available for doping testing, and that WADA labs will be required to develop their own test methods in house. It’s probably unrealistic to expect new and novel test methods to achieve perfect selectivity (notwithstanding the importance to the athletes being tested that the labs avoid making any mistakes). In addition, given that a urine sample is a complex matrix that might contain any number of unexpected substances, it might be impossible to prove that a lab method was capable of avoiding all possible interferences. Finally, we should recall that method validation is a finite effort, designed to determine fitness for purpose, but limited in important respects by factors of time and cost. Given all this, it seems most reasonable to define “specificity” for purposes of the ISL as a high degree of selectivity, but probably not as 100% selectivity.

But what of the difference highlighted above between the standards for avoiding (or limiting) matrix interference in ISL rules 5.4.4.2.1 and 5.4.4.2.2? Here on TBV, Mr. Idiot has advanced a good explanation for the difference in these standards. ISL 5.4.4.2.2 addresses the requirement to limit interference in testing for threshold substances, where the lab is required to measure the amount of the substance present in the standard. In contrast, ISL 5.4.4.2.1 governs the requirement to avoid interference in testing for non-threshold substances, where the lab is only required to determine the presence of the substance. Arguably, the standards for methods that need to measure the amount of a substance must be more exacting than the standards for methods that need only to detect the presence of a substance.

But I would caution against making too much of the difference in language between the standards in ISL rule 5.4.4.2.1 and 5.4.4.2.2. Recall what we’ve said so far about method validation. The goal of method validation is to ensure that a lab method is fit for purpose; a lab method that does not achieve good specificity (whether specificity is determined by avoiding interference or by limiting interference) is not fit for purpose. Moreover, given the unpredictability inherent in matrix interference (at least when dealing with a complex matrix such as urine), we can also expect that a lab will use the best means available to combat interferences, regardless of whether the lab is testing for a threshold or a non-threshold substance. At the same time, we expect labs to perform only a limited amount of method validation, so that method validation can be completed within a reasonable time and at a reasonable cost. In other words, we can predict that the main limitation on determining selectivity under the ISL will not be the standards set forth under ISL 5.4.4.2.1 and 5.4.4.2.2, but instead will be the limited effort required of labs under the ISL and ISO 17025 in performing method validation.

So, I think that the best way to understand the ISL rules on specificity is the way we’ve described above: under the ISL, specificity is a high degree of selectivity, but probably not perfect selectivity.

Up to the Introduction; back to part 9; on to part 11.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 9:Specificity/Selectivity

Up to the Introduction; back to part 8; on to part 10.

By Commenter Larry

We left part 8 discussing the ISO take on uncertainty, and the criteria that fall in that category. We now turn to the second of the two key parts in determining fitness for purpose, specificity/selectivity.

[MORE]

• Confirmation of Identity

A concept related to “selectivity” is “specificity”. “Specificity” is a desired state of selectivity. “Specificity” is one of those terms that is defined differently depending on where you look: specificity is either 100% selectivity (see Eurachem Guide paragraph 6.13) or a high degree of selectivity (see http://en.wikipedia.org/wiki/Selectivity).

As we’ve seen in earlier discussions on TBV, “specificity” is part of the ISL criteria for method validation (see ISL Rules 5.4.4.2.1 and 5.4.4.2.2). As per usual with rules written by WADA, it’s not clear exactly what the ISL means when it refers to specificity. The ISL contains separate rules for the validation of methods for threshold substances (ISL rule 5.4.4.2.2) and non-threshold substances (ISL rule 5.4.4.2.1). Both of these rules require validation of all WADA lab methods, and list “specificity” as an “[e]xample of factors relevant to determining if the method is fit for the purpose.” Given that specificity is described as an example of a criterion relevant to method validation, it’s not clear whether specificity is a required or an optional criterion. Later, rules 5.4.4.2.1 and 5.4.4.2.2 contain a bullet point for “Specificity” stating the following:

• Specificity. The ability of the assay to detect only the substance of interest must be determined and documented. The assay must be able to discriminate between compounds of closely related structures.

The specificity bullet point makes it sound as if specificity is both a required criterion for method validation under the ISL, and that the ISL defines specificity as 100% selectivity. However, to confuse matters further, ISL rules 5.4.4.2.1 and 5.4.4.2.2 contain a second bullet point for “Matrix interferences”, and there are differences in the way each rule describes “Matrix interferences”. The bullet point for matrix interferences under ISL rule 5.4.4.2.1 (for non-threshold substances) reads as follows:

• Matrix interferences. The method should avoid interference in the detection of Prohibited Substances or their Metabolites or Markers by components of the sample matrix.

And the bullet point for matrix interferences under ISL rule 5.4.4.2.2 (for threshold substances) reads as follows:

• Matrix interferences. The method must limit interference in the measurement of the amount of Prohibited Substances or their Metabolites or Markers by components of the sample matrix.

In short, the ISL language governing specificity and matrix interference wanders all over the place: permissive language (specificity and matrix interferences are merely examples of criteria for method validation), mandatory language (specificity must be determined and documented), and language somewhere in-between (some methods should avoid matrix interference; other methods must limit matrix interference). How, then, should we understand the ISL rules regarding specificity?

Up to the Introduction; back to part 8; on to part 10.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 8: Further Uncertainty.

Up to the Introduction; back to part 7; on to part 9.

By Commenter Larry

We left part 7 in the middle of talking about the ISO idea of uncertainty, one of two key criteria for fitness for purpose. We continue now, still discussing factors in uncertainty.

[MORE]

“Precision” is itself a measurement of two other criteria, “Repeatability” and “Reproducibility”. “Repeatability” is a method’s precision where the method is performed on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time. (Eurachem Guide paragraph A21). “Reproducibility” is a method’s precision where the method is performed on identical test items in different laboratories with different operators using different equipment. (Eurachem Guide paragraph A22). To complicate matters slightly, the ISL requires that method validation for threshold substances consider a criterion called “Intermediate Precision”. (See ISL Rule 5.4.4.3.2.1.) “Intermediate Precision” is the variation in results observed when one or more factors, such as time, equipment and operator, are varied within a laboratory. See http://www.measurementuncertainty.org/mu/guide/analytical.html. In other words, “intermediate precision” is a criterion that falls somewhere in-between repeatability and reproducibility.

(Interestingly, the ISL rules briefly refer to repeatability, see ISL Rule 5.4.4.3.2.1, but never to reproducibility. This omission may reflect WADA’s relative lack of concern with achieving consistent results among its various accredited labs. One further point: we can see that when Ali points to the variation between the LNDD S17 test results and the results achieved later upon the EDF re-analysis, he is pointing to a potential problem with the “intermediate precision” of LNDD’s test methods.)

Method “accuracy” is measured differently, depending on whether the method purpose is quantitative (as it would be for WADA threshold substances) or qualitative (as it would be for WADA non-threshold substances). (The Eurachem Guide says that this distinction applies to measurement of precision, see Eurachem Guide paragraph 6.37, but it would seem to apply equally to measurement of trueness.) If the method purpose is quantitative, “accuracy” is measured by looking at the amount that the test results differ from each other and from the reference value. If the method purpose is qualitative, “accuracy” is measured based on the percentage of the time that the test generates a false positive result or a false negative result. In either case, the method’s “purpose” should define the required test accuracy.

With the above discussion in hand, let’s look at how ISO 17025 and the ISL address the question of uncertainty. ISO 17025 Rule 5.4.6.2 addresses uncertainty in a general way, requiring testing laboratories to estimate uncertainty, or where such an estimate is impossible, to at least attempt to identify all of the components of uncertainty. The ISL requirements are similarly vague: ISL 5.4.4.3 notes the distinction we’ve already discussed between quantitative and qualitative uncertainty, and makes reference to concepts we’ve discussed above, such as repeatability, precision and bias. For quantitative methods, the ISL establishes a maximum uncertainty: “the expanded uncertainty using a coverage factor, k, to reflect a level of confidence of 95%.” ISL Rule 5.4.4.3.2.2. There is no corresponding maximum uncertainty for WADA lab qualitative methods – the ISL does not establish any maximum false positive or false negative percentages. However, it is clear from the ISL that all confirmation procedures – whether for threshold substances or non-threshold substances – must meet applicable uncertainty requirements. (ISL Rule 5.2.4.3)

(I should mention that the term “expanded uncertainty” used in ISL Rule 5.4.4.3.2.2 has a special meaning: it is a measure of uncertainty that defines a range within which we can expect to find a particular measurement result. The math for determining expanded uncertainty is beyond what I want to cover here. Anyone interested in learning more about “expanded uncertainty” can look here: http://physics.nist.gov/cuu/Uncertainty/coverage.html.)

I’ll conclude my discussion of uncertainty by stating the obvious: uncertainty is an enormously complicated topic, and my discussion here barely begins to explain this topic. There are important concepts that I have not touched upon, such as “measurement uncertainty” and “method uncertainty”. We have not covered enough information to look at how LNDD came up with its stated +/- 0.8 uncertainty for its CIR testing, nor have we figured out how this stated uncertainty relates to the ISL or to method validation in general.

We cannot become experts in “uncertainty” in the course of a single article like this one. Instead, what I’ve tried to do here is to introduce this topic, and more importantly, to place this topic in an overall context. “Uncertainty” (and related concepts such as bias, traceability, precision and repeatability), are method validation concepts that speak to whether a method is fit for purpose. We’ll return to a longer discussion of the importance of this context before this article is finished.

Up to the Introduction; back to part 7; on to part 9.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 7: Uncertainty

Up to the Introduction; back to part 6; on to part 8.

Hey folks, we're HALFWAY there!

By Commenter Larry

Criteria for Fitness for Purpose

There are a number of criteria a lab can use to determine the “fitness” of a particular method. Two of these criteria – uncertainty and selectivity/specificity – are arguably the most important, so we’ll limit our discussion of method validation to these two criteria – and to the myriad of sub-criteria related to these two principle criteria. It's going to take two posts to talk about uncertainty. We won't get to selectivity/specificity until part 9.

[MORE]

• Uncertainty. In simplest terms, “uncertainty” is a measure of the “accuracy” of a lab method. The “accuracy” of a lab method consists of two components: “trueness” (the closeness of a single result to the true value) and “precision” (how close multiple measurements made by the same method are to one another). Eurachem Guide 6.30.

Obviously, “trueness” is a tricky concept: how can you tell if a new method is providing a “true” result? According to the Eurachem Guide, a lab determines the “trueness” of a method by comparing method results against a known “reference value”. The lab has two techniques available to determine a “reference value”: (1) the lab can utilize a characterized (or reference) material, where the value the method is supposed to measure is already known, or (2) the lab can compare the results of its test method against a different method that has already been validated and approved for “trueness”. See Eurachem Guide paragraph 6.31.

The concept of “trueness” has a few related concepts that are referred to in the ISL and in ISO 17025. One of these concepts is “bias”. “Bias” is the difference between the expectation of the test results and an accepted reference value. See Eurachem Guide paragraph A2. There are, in turn, two types of bias: “method bias” and “laboratory bias”. “Method bias” is bias inherent in the method; “laboratory bias” is the additional bias peculiar to the laboratory and its interpretation of the method. See Eurachem Guide paragraph 6.35. I personally do not find “bias” to be a useful concept, but it is referred to in the ISL (see ISL 5.4.4.3.2.1), so I thought I should mention it here.

A second related concept mentioned in ISO 17025 and the ISL is “traceability”. “Traceability” refers to the ability of a test method to relate to a known standard. See Eurachem Guide paragraph A30. In the context of our discussion here, “traceability” means that the “trueness” of a method has been measured by testing the method against a standard that is well-accepted in the scientific community – for example, the method would be “traceable” if it could be tested on a reference material that is widely accepted in the field of doping control. Traceability merits its own section in ISO 17025 (5.6), but I think traceability is best understood as a property of “trueness”.

(Interestingly, the ISL by its terms seems to preclude any meaningful validation of the “trueness” of a lab method. WADA labs cannot determine “trueness” by utilizing a known reference material, since according to the ISL, “[f]ew of the available reference drug and drug Metabolite(s) are traceable to national or international standards.” ISL 5.4.6.1. And given WADA’s proclamation (discussed above) that standard methods are not available for doping control, it would be impossible for a WADA lab to determine the “trueness” of a method by comparing the method to a second, already validated, method. My guess is that WADA labs validate the “trueness” of their methods by utilizing whatever reference materials they can find, but I have no way to know for certain that this is what they do.)

As stated above, method “accuracy” is a factor of both method “trueness” and method “precision”. “Precision” is a measure of the closeness of method results when the method is repeated under the same conditions. See Eurachem Guide paragraph 15.1.

“Precision” is itself a measurement of two other criteria, “Repeatability” and “Reproducibility”. “Repeatability” is a method’s precision where the method is performed on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time. (Eurachem Guide paragraph A21). “Reproducibility” is a method’s precision where the method is performed on identical test items in different laboratories with different operators using different equipment. (Eurachem Guide paragraph A22). To complicate matters slightly, the ISL requires that method validation for threshold substances consider a criterion called “Intermediate Precision”. (See ISL Rule 5.4.4.3.2.1.) “Intermediate Precision” is the variation in results observed when one or more factors, such as time, equipment and operator, are varied within a laboratory. See http://www.measurementuncertainty.org/mu/guide/analytical.html. In other words, “intermediate precision” is a criterion that falls somewhere in-between repeatability and reproducibility.

(Interestingly, the ISL rules briefly refer to repeatability, see ISL Rule 5.4.4.3.2.1, but never to reproducibility. This omission may reflect WADA’s relative lack of concern with achieving consistent results among its various accredited labs. One further point: we can see that when Ali points to the variation between the LNDD S17 test results and the results achieved later upon the EDF re-analysis, he is pointing to a potential problem with the “intermediate precision” of LNDD’s test methods.)

Method “accuracy” is measured differently, depending on whether the method purpose is quantitative (as it would be for WADA threshold substances) or qualitative (as it would be for WADA non-threshold substances). (The Eurachem Guide says that this distinction applies to measurement of precision, see Eurachem Guide paragraph 6.37, but it would seem to apply equally to measurement of trueness.) If the method purpose is quantitative, “accuracy” is measured by looking at the amount that the test results differ from each other and from the reference value. If the method purpose is qualitative, “accuracy” is measured based on the percentage of the time that the test generates a false positive result or a false negative result. In either case, the method’s “purpose” should define the required test accuracy.

Up to the Introduction; back to part 6; on to part 8.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 6: On Fitness For Purpose

Up to the Introduction; back to part 5; on to part 7

By Commenter Larry

Fitness For Purpose

As should be obvious, in order to determine whether a lab method is “fit for purpose”, the lab must determine the method’s intended purpose, or in the words of the Eurachem Guide, “the performance requirements that a method must have to be suitable for solving the analytical problem.” The process of method validation can be understood as a cycle of method development and evaluation that continues until the method is found to be capable of meeting the defined performance requirement. See Eurachem Guide paragraph 6.9.

[MORE]

ISO 17025 is written for laboratories doing work for clients, and as you would expect, ISO 17025 is primarily addressed to the responsibilities that labs must assume when they do work for clients. However, when it comes to the “fitness for purpose” standard, the client bears some responsibility for defining the purpose of the method to be selected or developed by the lab. It is up to the client in the first instance to define the requirements of the method they want the lab to perform. As stated in the Eurachem Guide, “ideally the laboratory should first agree with the customer [on] an analytical requirement which defines the performance requirements that a method must have to be suitable for solving the analytical problem.” See Eurachem Guide paragraph 6.9. Of course, most lab customers are not sophisticated enough to come up with any such definition. The Eurachem Guide acknowledges that most customers “define their requirements in terms of cost and/or time and rarely know how well methods need to perform.” Eurachem Guide paragraph 6.10.

But WADA is not a typical lab customer – WADA is a leading authority in the field of doping control, and it contracts with numbers of labs world-wide to perform doping tests. Consequently, WADA bears a significant responsibility for defining the purpose of these tests, and WADA has at least made an effort in the ISL to meet this responsibility. Under the ISL, each test developed by a WADA lab must be capable of measuring a specified minimum amount of a particular prohibited substance. This specified minimum amount is sometimes called a “limit of detection”. See ISL Rule 5.4.4.1.3. (For a discussion of “limit of detection”, see Eurachem Guide paragraph 6.20.) If the test is to detect a “non-threshold substance” (a substance that can be the basis for an AAF if found in the athlete’s system in any amount), then the test must be able to identify the substance at the limit of detection. See ISL Rule 5.4.4.1.1. If the test is supposed to detect a “threshold substance” (a substance that must be present in an athlete’s system in more than a specified amount in order for the lab to find an AAF), then the “purpose” of the test is more demanding: the test must be capable of identifying both the substance and the amount of the substance with an acceptable “uncertainty”. See ISL Rule 5.4.4.1.2. (The issue of “uncertainty” is important, and we’ll address it shortly.)

(Here at TBV, we’ve debated the fitness of methods in use at WADA labs, but we have not considered as carefully the stated purpose for these methods. Assuming for the moment that there are deficiencies in the way that these labs perform their doping tests, we should consider whether the fault may lie in part with WADA’s failure to articulate a clear and definitive purpose for these tests.)

Once a lab determines the purpose of a particular method and develops a method to meet this purpose, the lab can then consider whether the method is “fit” for this purpose. This analysis is not limited to looking at the method in a vacuum. The determination of “fitness for purpose” turns on how the method performs when used at a particular lab, by the analysts employed at that lab, using the equipment and facilities available at that lab. See Eurachem Guide paragraph 6.9. A method that might work perfectly well at a state-of-the-art lab might not be “fit for purpose” at a less sophisticated lab.

The effort involved in method validation will vary. ISO 17025 rule 5.4.5.2 states vaguely that “[t]he validation shall be as extensive as is necessary to meet the needs of the given application.” The Eurachem Guide is a bit more helpful. The Eurachem Guide states that “[c]haracterisation of method performance is an expensive process and inevitably it may be constrained by time and cost considerations,” and that the lab must strike “the balance between time and costs constraints and the need to validate the method.” How does the lab manage to strike such a critical balance? The Eurachem Guide states only that “the laboratory should do the best it can within the constraints imposed.” (Eurachem Guide paragraphs 6.6 and 6.7.) So, we should require that the WADA labs do a reasonable job of method validation, but we should expect that “perfect” validation is probably beyond what we have a right to expect from any lab.

Up to the Introduction; back to part 5; on to part 7

Full Post with Comments...

Larry's Curb Your Anticipation, Part 5:Test Development and Validation

Up to the Introduction; Back to part 4; On to part 6.

By Commenter Larry

Let’s get started looking at the ISO laboratory rules that apply, finishing here with "Fitness for Purpose".

Rules Governing Test Method Development and Validation

Our first category, test method and development, is probably our most complicated category. It is one of the most detailed categories under ISO 17025, and the longest single section in the ISL. Arguably, this is also the most important single category applicable to anti-doping testing. Why should this be the case?

[MORE]

The answer is set forth in ISL Rule 5.4.4.1:

Standard methods are generally not available for Doping Control analyses. The Laboratory shall develop, validate, and document in-house methods for compounds present on the Prohibited List and for related substances. The methods shall be selected and validated so they are fit for the purpose.

ISL Rule 5.4.4.1 points out a critically important feature of anti-doping testing: standard methods are generally not available for Doping Control analyses. Consequently, the ISL requires each WADA lab to come up with its own tests “in house”. Given the importance of these tests, it makes sense that a substantial portion of the ISL addresses the process of developing and validating these tests.

We can divide our discussion under this heading into two parts: method development and method validation. We will examine each of these concepts in turn. In the process, we’ll look at a number of the terms we’ve used in our discussions here at TBV: terms such as “specificity”, and “traceability”, and “uncertainty”. I will do my best to define these terms, but you should note that these terms can have different meanings in different contexts, and that some of these terms do not have universally accepted definitions. I will try to define these terms in a way I think is consistent with the ISL and ISO 17025, and I will always provide a citation to support the definition I have chosen, but you may well be able to find alternative definitions for some of these terms.

1. Test Method Development

ISO 17025 assumes that a lab will select existing methods whenever possible, so the ISO 17025 standard has comparatively little to say about method development. ISO 17025 Rule 5.4.4 contains general requirements for new test procedures: each new test procedure must set forth an appropriate scope, parameters, pre-test checks, a method of recording observations, and interestingly, “criteria and/or requirements for approval/rejection.” (What does ISO 17025 Rule 5.4.4 mean when it refers to “approval/rejection”? Does this rule require that a test method contain criteria for when the results of a test have been performed correctly and can be approved, and when the results are invalid and must be rejected? ISO 17025 does not say. What criteria did LNDD use for “approval/rejection” under ISO 17025 Rule 5.4.4? It would be interesting to know the answers to these questions.)

While the ISL has rules addressed to “Selection of Methods”, see ISL Rule 5.4.4.1, I think that these rules fit better into our discussion of Test Method Validation below.

2. Test Method Validation

Method validation rules are arguably the most important rules in the ISL. I will argue that most of the lab testing criteria we discuss here on TBV are method validation rules. However, “method validation” has a specific meaning in the world of lab testing. Thanks to Russ, we have a very good discussion of method validation available to us in the form of a Eurachem Guide on “Fitness for Purpose”. See http://www.eurachem.org/guides/valid.pdf. I will rely heavily on this Eurachem Guide in the analysis that follows.

ISO 17025 defines “method validation” as “the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled.” (ISO 17025 Rule 5.4.5.1.) A shorter definition of method validation, more to the point, is “the process of verifying that a method is fit for purpose, i.e. for use for solving a particular analytical problem.” (See Eurachem Guide Annex A paragraph A33.2)

Method validation is closely connected with method development: according to the Eurachem guide, it is often not possible to determine exactly where method development finishes and validation begins. (Eurachem Guide 3.1) However, method validation is an ongoing process that does not end after the method has been developed. According to paragraph 5.1 of the Eurachem Guide:

A method should be validated when it is necessary to verify that its performance parameters are adequate for use for a particular analytical problem. For example:

• new method developed for particular problem;

• established method revised to incorporate improvements or extended to a new problem;

• when quality control indicates an established method is changing with time;

• established method used in a different laboratory, or with different analysts or different instrumentation;

• to demonstrate the equivalence between two methods, e.g. a new method and a standard.

In other words, method validation should take place not only as a new method is adopted, but also at any time the method is changed or is employed in a different way, or where the lab’s quality control efforts indicate that the method should be re-examined.

Central to method validation is the concept of “fitness for purpose” – the determination that the method under consideration has performance capabilities consistent with what the application requires. If a method is “fit for purpose”, then it is trustworthy – it provides the “right answer to the analytical part of the problem.” (See Eurachem Guide Section 4.3) So, implicit in method validation is a determination by the lab that the method is good enough, that it is worth performing, that on an absolute basis the test will meet its designed purpose.

Up to the Introduction; Back to part 4; On to part 6.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 4: What are the key bits of 17025

Up to the Introduction; Back to part 3; On to part 5.

By Commenter Larry

The Structure of ISO 17025

There are two main groups of requirements under ISO/IEC 17025 - Management Requirements and Technical Requirements. Management Requirements are primarily related to the operation and effectiveness of the quality management system within the laboratory. Technical Requirements address the competence of staff, methodology and test/calibration equipment, but also address issues critical to lab testing, such as method validation, traceability and sampling. See http://en.wikipedia.org/wiki/ISO_17025. In order to understand how ISO 17025 regulates lab testing, we’ll need to look at both the Management Requirements and the Technical Requirements.

[MORE]

There are fifteen Management Requirement and ten Technical Requirements under ISO 17025. The Management Requirements are:

• 4.1 Organization and management

• 4.2 Quality system

• 4.3 Document control

• 4.4 Request, tender and contract review

• 4.5 Sub-contracting of tests and calibrations

• 4.6 Purchasing services and supplies

• 4.7 Service to the client

• 4.8 Complaints

• 4.9 Control of nonconforming testing and/or calibration work

• 4.10 Improvement

• 4.11 Corrective action

• 4.12 Preventive action

• 4.13 Records

• 4.14 Internal audits

• 4.15 Management reviews

The Technical Requirements are:

• 5.1 General

• 5.2 Personnel

• 5.3 Accommodation and environmental conditions

• 5.4 Test and calibration methods including sampling

• 5.5 Equipment

• 5.6 Measurement traceability

• 5.7 Sampling

• 5.8 Handling of test and calibration items

• 5.9 Assuring the quality of test and calibration results

• 5.10 Reporting the results

In this article, we’re interested only in those ISO 17025 rules (as supplemented by the ISL) that deal with lab testing, and that might serve as the grounds for an “ISL departure” under the WADA rules. So we’re not interested in a number of the ISO 17025 requirements, such as how the lab does its contracting or how the lab hires its personnel. Also, you’ll probably note that the ISO 17025 requirements are not laid out in a logical order for reviewing how a lab performs its tests. Some of the steps we’d expect the lab to perform at the end of the testing process, such as corrective action, are listed before steps that the lab would need to take at the beginning of the testing process, such as test validation and sampling.

So for our purposes, I’d like to reorganize the ISO 17025 Requirements in a different way, so that we can focus just on those requirements dealing with lab testing, and so that we can consider those requirements in a more logical order. The order I propose to use is a sort of “chronological” order, with steps listed in the order that the lab must perform them in order to reach a valid result.

I propose to look at the ISO 17025 and ISL rules in the following order:

• Rules governing test method development and validation (ISO Rule 5.4; ISL Rule 5.4.4)

• Rules governing “quality control” (ISO Rule 5.9)

• Rules governing sampling and other steps taken prior to actual testing (ISO Rules 5.7 and 5.8)

• Rules governing how a lab must perform its tests

• Rules governing how a lab must interpret the tests it performs

• Rules governing the lab’s internal review of its work product (ISO Rules 4.9 and 4.11)

As should be apparent, we’re looking at only a handful of ISO Rules. Nevertheless, the task ahead of us is steep. The analysis of these six categories of rules will take us through a number of difficult and complex topics. We’ll only be able to examine the first of these six categories in this article.

Up to the Introduction; Back to part 3; On to part 5.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 3: ISO 17025

Up to the Introduction; Back to part 2; On to part 4.

By commenter Larry

ISO 17025: An Introduction

The main standard used by testing laboratories world-wide is ISO 17025 (sometimes called ISO/IEC 17025), issued by the International Organization for Standardization. ISO 17025 provides rules for how labs should operate in order to produce consistently valid results. ISO 17025 is also the basis for lab accreditation by bodies such as the American Association for Lab Accreditation.

[MORE]


The complete text of ISO 17025 is available at http://www.usocpressbox.org/usoc/pressbox.nsf/ac7bf642f496016a87256d0d006a340c/1b39aafc829ba275852572fc000de8ee/$FILE/023.PDF. The best general discussion I’ve seen of ISO 17025 is at http://www.labcompliance.com/tutorial/iso17025/default.aspx?sm=d_e.

(The version of ISO 17025 referred to in the ISL is the 1999 version, but at the time of the testing at issue in the FL arbitration, the 2005 version of ISO 17025 was in effect. USADA posted both the 1999 and 2005 versions of ISO 17025 as exhibits in the FL case, so it is not clear which version of ISO 17025 should be applied to the FL case. I have decided to use the 2005 version of ISO 17025 in this analysis. I do not believe that there are material differences between the two versions of ISO 17025 that would affect my analysis here.)

ISO 17025 applies one set of rules to be followed by all kinds of testing laboratories – not just anti-doping laboratories, or medical laboratories, or even laboratories that perform testing on human samples. For this reason, the standards in ISO 17025 are universal and general in nature. However, ISO 17025 recognizes that more specific requirements may be needed in order to ensure that specialty labs are competent to perform their jobs. For this reason, Annex B of ISO 17025 provides guidelines that certain specialty labs must use to supplement the general requirements of ISO 17025. In particular, Annex B.4 of ISO 17025 provides as follows:

[I]t may be necessary to develop a separate document of applications to supplement [ISO 17025] for specific types or groups of tests … Such a document should provide only the necessary supplementary information, while maintaining [ISO 17025] as the governing document through reference.

By its terms, Article 5 of the ISL is a supplementary document under Annex B.4 of ISO 17025, for the field of doping control. As required by ISO 17025, the ISL recognizes that any aspect of testing or management not addressed in the ISL is to be governed by ISO 17025. See ISL Rule 5.1. Moreover, all WADA laboratories are required to be accredited by a national accreditation body and periodically audited according to ISO 17025 (see ISL Rules 4.1.1 and 6.4.7.2), and all WADA lab methods and procedures must eventually be included in the scope of these periodic ISO audits. See ISL Rule 4.2.2.

The principle that ISO 17025 is applicable to anti-doping testing is expressly recognized in the decision of the majority arbitrators in the FL case, who ruled that “violations of the ISO 17025 …can be violations of the ISL for purposes of rebutting the initial presumption favouring the Lab that an AAF has been established.” See http://ia341243.us.archive.org/0/items/Floyd_Landis_USADA_Case_Decision_Documents/UsadaAndLandis-FinalAward20-09-07.pdf paragraph 156-157.

In some of my prior posts, I’ve noted that ISL Rule 7.1 contains language that appears to limit the application of ISO 17025 to anti-doping cases. The applicable language of ISL Rule 7.1 is set forth in full below:

The Laboratory is not required to provide any documentation not specifically included in the Laboratory Documentation Package. Therefore, the Laboratory is not required to support an Adverse Analytical Finding by producing, either to the Testing Authority or in response to discovery requests related to the hearing, standard operating procedures, general quality management documents (e.g., ISO compliance documents) or any other documents not specifically required by Technical Document on Laboratory Documentation Packages. References in the International Standard for Laboratories to ISO requirements are for general quality control purposes only and have no applicability to any adjudication of any specific Adverse Analytical Finding. (emphasis added)

Read broadly, the language in italics above would indicate that a lab’s violation of applicable ISO 17025 standards would not give an athlete a potential defense to an AAF found by the lab. However, this interpretation of ISL Rule 7.1 is contradicted by ISL Rule 5.1, by the express language of the FL decision, and by the fact that specific ISO 17025 requirements are cited more than a dozen times in the ISL, in ways that make it clear that ISO 17025 is an integral part of WADA drug testing. (See, for example, ISL Rules 4.1.1, 4.2.1, 4.2.2, 4.3.1, 5.2.6.6, 5.3.1.1, 5.3.2.1, 5.3.3, 5.3.4, 5.3.7.1, 5.3.7.3.4, 5.3.8, 5.3.10, 5.3.11, 5.3.13.1, 5.3.14.1, 5.4.1, 5.4.5.2, 6.2.1, 6.3.1, and 6.4.7). Consequently, the language in italics in Rule 7.1 should be understood as limited to the matters addressed in that rule. In other words, ISL Rule 7.1 should be understood to state that ISO 17025 requirements are not applicable to determine the documents that a lab is required to include in its Laboratory Document Package (LDP), but that ISO 17025 is applicable to other aspects of the lab’s work.

Let’s summarize. It is clear that in order to understand the ISL, we must also understand ISO 17025. ISO 17025 is the master document, while the ISL provides more specific rules to supplement the general rules in ISO 17025. These two documents are complementary documents that must be read together to understand the rules governing anti-doping testing in cycling. Where the ISL is silent, we must look to ISO 17025 for the governing rules. And as the ISL was written to “follow the format of the ISO 17025 document” (see ISL Rule 5.1), we should read the ISL rules within the context of ISO 17025 and within the structure set forth in ISO 17025.

Up to the Introduction; Back to part 2; On to part 4.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 2: The ISL

Up to the Introduction; Back to part 1; On to part 3.

By commenter Larry

In part 1, we set up the legal framework for doping testing in cycling, so let’s turn our focus to the rules governing how WADA labs perform drug testing. The applicable rules are the WADA ISL (International Standard for Laboratories), and the ISO 17025 standard. Here in part 2, we'll introduce the ISL, and in part 3, ISO 17025.

[MORE]

The ISL: An Introduction

The document at the heart of the WADA Rules governing lab testing is the ISL. You can find the ISL at http://www.wada-ama.org/rtecontent/document/lab_aug_04.pdf. (Note that the version of the ISL applicable to the FL case is version 4.0. The current version of the ISL, version 5.0, came into effect on January 1, 2008.) The ISL also includes all WADA technical documents (TDs), including TD2003 LCOC (governing chain of custody issues) and TD2003IDCR (governing identification of substances in chromatographic testing). You can find all of these TDs listed and available for download at http://www.wada-ama.org/en/dynamic.ch2?pageCategory.id=372.

Let’s quickly revisit how the WADA Rules incorporate the ISL. Once a WADA lab has proved an AAF, the accused athlete has the burden of proving a “departure” by the lab from the ISL. If the athlete demonstrates such a departure, then the prosecuting ADA has the burden of proving that such departure did not cause the AAF or the factual basis for the anti-doping rule violation. (WADA Rule paragraph 3.2.2.)

From the above description, you might assume that the ISL consists of a list of rules that a lab must follow in testing an athlete’s sample. Not so. The ISL is considerably more complicated, and more difficult to interpret.

The ISL contains two kinds of rules (see ISL Rule 1.0 for this summary):

• Rules Governing the Accreditation Procedure. These rules set forth the process that a lab must follow in order to receive WADA accreditation. These rules do not directly address how a lab performs its testing. For example, ISL Rule 4.1.2 requires that a lab seeking accreditation must receive a letter of support from a public authority responsible for a national anti-doping program (such as USADA). I believe that all rules under ISL Articles 4 and 6 fall under this category of accreditation procedure rules.

Some rules governing the accreditation procedure are of interest to us here at TBV. For example, ISL Rule 4.2.4 requires a WADA lab to analyze a minimum of 1500 doping samples a year, or else risk the loss or suspension of its accreditation. This gives us a picture of the minimum volume of work at a WADA lab. ISL Rule 4.2.2 allows a WADA lab to add or modify its methods without the approval of the organization that provided its accreditation, with the understanding that the method must be validated the next time the lab is audited. This lets us know that the methods used at LNDD are supposed to be reviewed by an accreditation body.

Since our primary focus here is to understand lab testing, and specifically to identify the ISL rules that can serve as the grounds for an ISL “departure” under the WADA Rules, I think we can ignore all ISL rules that govern the lab accreditation process.

• Operating standards for laboratory performance. This is the place where we would look to find the kind of substantive rules that a lab must follow in order to prove an AAF. These rules are contained in ISL Article 5, as well as in the TDs. Consequently, our primary focus in this article will be on the ISL rules found in Article 5

The first rule set forth in ISL Article 5 may be the single most important rule in the ISL.

ISL Rule 5.1 states that Article 5 of the ISL is a specialty document under Annex B.4 of ISO 17025, for the field of doping control. I will argue that the key to understanding the ISL is set forth in this Rule 5.1. But to understand ISL Rule 5.1, we must first answer the questions: what is ISO 17025? And why is ISO 17025 referred to in the ISL?

(We should note that ISL Rule 5.1 refers to ISO 9001 as well as to ISO 17025. We will not have time in this article to consider ISO 9001, but we will probably want to examine ISO 9001 in a later article.)

Up to the Introduction; Back to part 1; On to part 3.

Full Post with Comments...

Larry's Curb Your Anticipation, Part 1:Some General Questions

Up to the Introduction; On to part 2.

By Commenter Larry

Preliminary Questions

Why Are the WADA Rules Binding on International Cycling?

Let’s begin our discussion of anti-doping law at the most basic level. All professional cyclists are licensed by a national federation, such as USA Cycling in the U.S. The terms of a national cycling license require that the cyclist follow the anti-doping rules of the national federation, the Union Cycliste Internationale (UCI) and the World Anti-Doping Agency (WADA).

[MORE]

For example, the terms of the International License available through USA Cycling include the following:

I shall submit to disciplinary measures taken against me and shall take any appeals and litigation before the authorities provided for in the regulations. I accept the Court of Arbitration for Sport (CAS) as the only competent jurisdiction of appeal in the cases provided for by the Regulations and in compliance with the terms thereof. ... I agree to submit to drug testing and to comply with and to be bound by the UCI anti-doping regulations, the World Anti-Doping Code and its International Standards to which the UCI anti-doping regulations refer as well as the anti-doping regulations of other competent instances as foreseen by the UCI Regulations, the World Anti-Doping Code, or the U.S. Anti-Doping Agency (USADA), provided such regulations comply with the World Anti-Doping Code. …. I accept these conditions regarding blood testing and agree to undergo all tests required of me.

See http://www.usacycling.org/forms/intl_license.pdf.

The primary authority in international cycling is the UCI. WADA’s authority in cycling derives from UCI’s adoption of the World Anti-Doping Code (the WADA Rules). The WADA Rules are incorporated in the UCI Cycling Regulations (UCI Rules). See UCI Rule Part 14, Introduction, at http://www.uci.ch/imgArchive/Rules/14ant-E.pdf.

Whenever possible, I will cite to both the WADA Rules and the UCI Rules in the following discussion.

A Brief Description of the WADA Rules and the UCI Rules

In an anti-doping case, the anti-doping organization (ADA) has the initial burden of establishing that an anti-doping rule violation has occurred. UCI Rules Article 16, WADA Rules Article 3.1. However, once the Anti-Doping Organization introduces evidence of an anti-doping violation from a WADA-accredited laboratory, the results are presumed correct. UCI Rules Article 18, WADA Rules Article 2.4.3. The athlete is then entitled to rebut this presumption by establishing that the lab departed from the ISL. UCI Rules Article 18, WADA Code Article 3.2.1. If the athlete is able to prove an ISL departure, then the ADA must prove that the departure did not cause the adverse analytical finding (AAF) against the athlete. Id.

The process described above sets up what we’ve called the “burden shift” or “burden flip”. The athlete has the initial burden to show an ISL departure, at which point the burden “flips” to the prosecuting ADA to show that the ISL departure did not cause the AAF. In this article, we’ll focus almost exclusively on the first part of the “flip”: we’ll be looking at what it takes to prove an ISL departure. But keep in mind that this discussion is only half of the story: the ADA may be able to meet its burden of proof even if the burden is “flipped”. Remember that in the FL case, the FL legal team was able to prove a number of departures from the ISL, but USADA was able to prove (to the satisfaction of two of the three arbitrators on the panel) that these departures did not cause the AAF against FL.

Let’s look closer at the standard of proof required to establish an AAF. Under the UCI and WADA Rules, the ADA must prove the AAF “to the comfortable satisfaction of the hearing body bearing in mind the seriousness of the allegation which is made.” According to the UCI Rules and the WADA rules, this standard of proof is greater than the mere balance of probability required to win a case in a U.S. civil court proceeding, but is less than the proof beyond a reasonable doubt required for a U.S. criminal conviction. See UCI Rules Article 16, WADA Rules Article 3.1. In the case of USADA v. Montgomery, http://www.usantidoping.org/files/active/arbitration_rulings/CAS%20Decision_Tim%20Montgomery_Dec%202005[1].pdf, the Court of Arbitration for Sport (CAS) defined "comfortable satisfaction" to impose an “extremely high” level of proof on the prosecuting ADA. In the Montgomery case, the CAS ruled that there was “little, if indeed any, difference” between the standard of proof imposed on the ADA and proof beyond a reasonable doubt. The FL legal team concluded that USADA’s burden of proof in the FL case was “very close to ‘proof beyond a reasonable doubt’", see http://ia351413.us.archive.org/1/items/Floyd_Landis_Case_Documents_14/RespondentsProposedFindingsofFact.pdf paragraph 2.4.7, and I think that this conclusion is correct.

Is An SOP Violation the Equivalent of an ISL Departure?

In our discussions on TBV, we’ve discussed instances where the lab in the FL case may have violated its own standard operating procedures (SOP) in conducting its testing. For example, we’ve discussed at length whether the lab’s machinery was set up in accordance with the lab’s SOP. See the discussion at http://trustbut.blogspot.com/2007/12/different-columns-detailed.html. If a lab violates its own SOP, is this treated under the WADA rules like an ISL departure, where the lab must prove that its SOP violation did not cause the AAF?

This issue came up in the FL case. The lab in the FL case, the LNDD, had a SOP requiring it to do a monthly check of the linearity of its Isotope Ratio Mass Spectrometer (IRMS). USADA admitted that LNDD did not make these monthly checks. The arbitrators ruled as a result that “the Lab failed to adhere to the ISL in failing to check the linearity of the IRMS instrument on a monthly basis as provided for in its ISO 17025 accreditation.” See FL decision paragraph 218 (emphasis added). Later in the decision, focusing on this same point, the arbitrators stated that “[t]he SOPs are part of the ISO 17025 accreditation”.

It’s clear from the FL decision that at least some SOP violations will be treated as ISL departures under the WADA Rules. The key appears to be whether the SOP is a part of the lab procedures that are reviewed in the ISO 17025 accreditation process. While this question is not clearly answered in the FL decision, I would argue that any violation of an SOP that can clearly be connected to a requirement under the ISL or ISO 17025 should be treated as an ISL departure. For example, the LNDD’s failure to do its monthly linearity check of its IRMS machine arguably violated ISO 17025 Rule 5.4.5.3, which requires the lab to assess linearity as part of its method validation, and ISO 17025 Rule 5.5, which governs proper maintenance of lab equipment.

Can The Athlete Challenge the Scientific Basis of an AAF?

We’ve stated on TBV that the WADA Rules prohibit an athlete from challenging the science used by a WADA lab to prove an AAF. As it turns out, this statement is not 100% correct. As we’ll see, many anti-doping cases are based on the athlete’s argument that the AAF is based on bad science.

The actual rule in this area is as follows:

• The UCI and WADA rules establish a presumption that a WADA lab has carried out its anti-doping tests in accordance with acceptable current scientific standards. See USADA v. Bergman, http://www.usantidoping.org/files/active/arbitration_rulings/CAS%20Decision%20-%20Bergman.pdf, p. 12.

• This presumption of validity extends into three potential areas of challenge: the scientific basis for the test, the analytical result (for example, the chromatograms on which the FL AAF was based), and the interpretation of the analytical result. Id.

• The athlete can overcome this presumption of validity by presenting proof that the test as designed, performed or interpreted is not scientifically valid.

There are a number of anti-doping cases where the athlete has attempted to challenge the scientific basis of the AAF. For example, in USADA v. Hamilton, http://www.usantidoping.org/files/active/arbitration_rulings/AAA_CAS%20Decision%20-%20Hamilton[1].pdf, Tyler Hamilton challenged the scientific basis of his blood doping AAF, arguing that the blood doping test used to convict him was insufficiently validated, should have been based on a quantitative rather than a visual standard, and proved only that he had a “mixed red cell population” (which could be explained by circumstances other than blood doping). See Hamilton Decision p. 4. The arbitration panel in the Hamilton case rejected Hamilton’s arguments on scientific grounds, but there is no suggestion that Hamilton was precluded by the WADA Rules from raising scientific objections to the anti-doping tests.

Similarly, in the USADA v. Bergman case cited above, Bergman unsuccessfully challenged the scientific basis of the WADA lab’s interpretation of the rEPO testing used to prove Bergman’s AAF. As with the Hamilton case, the CAS rejected Bergman’s arguments on scientific grounds, and noted the presumption under the WADA and UCI rules in favor of the lab’s interpretation of Bergman’s rEPO tests. However, there is no suggestion in the Bergman case that the WADA or UCI rules barred Bergman from challenging the science behind the rEPO testing.

From the above, we learn that the WADA and UCI rules set up a presumption that all AAFs have a valid basis in science. The athlete can challenge this presumption – there is nothing in the rules preventing the athlete from challenging any aspect of the science behind an AAF. However, to my knowledge, no such challenge has succeeded to date.

Up to the Introduction; On to part 2.

Full Post with Comments...