There are a number of criteria a lab can use to determine the “fitness” of a particular method. Two of these criteria – uncertainty and selectivity/specificity – are arguably the most important, so we’ll limit our discussion of method validation to these two criteria – and to the myriad of sub-criteria related to these two principle criteria. It's going to take two posts to talk about uncertainty. We won't get to selectivity/specificity until part 9.
Uncertainty. In simplest terms, “uncertainty” is a measure of the “accuracy” of a lab method. The “accuracy” of a lab method consists of two components: “trueness” (the closeness of a single result to the true value) and “precision” (how close multiple measurements made by the same method are to one another). Eurachem Guide 6.30.
Obviously, “trueness” is a tricky concept: how can you tell if a new method is providing a “true” result? According to the Eurachem Guide, a lab determines the “trueness” of a method by comparing method results against a known “reference value”. The lab has two techniques available to determine a “reference value”: (1) the lab can utilize a characterized (or reference) material, where the value the method is supposed to measure is already known, or (2) the lab can compare the results of its test method against a different method that has already been validated and approved for “trueness”. See Eurachem Guide paragraph 6.31.
The concept of “trueness” has a few related concepts that are referred to in the ISL and in ISO 17025. One of these concepts is “bias”. “Bias” is the difference between the expectation of the test results and an accepted reference value. See Eurachem Guide paragraph A2. There are, in turn, two types of bias: “method bias” and “laboratory bias”. “Method bias” is bias inherent in the method; “laboratory bias” is the additional bias peculiar to the laboratory and its interpretation of the method. See Eurachem Guide paragraph 6.35. I personally do not find “bias” to be a useful concept, but it is referred to in the ISL (see ISL 188.8.131.52.2.1), so I thought I should mention it here.
A second related concept mentioned in ISO 17025 and the ISL is “traceability”. “Traceability” refers to the ability of a test method to relate to a known standard. See Eurachem Guide paragraph A30. In the context of our discussion here, “traceability” means that the “trueness” of a method has been measured by testing the method against a standard that is well-accepted in the scientific community – for example, the method would be “traceable” if it could be tested on a reference material that is widely accepted in the field of doping control. Traceability merits its own section in ISO 17025 (5.6), but I think traceability is best understood as a property of “trueness”.
(Interestingly, the ISL by its terms seems to preclude any meaningful validation of the “trueness” of a lab method. WADA labs cannot determine “trueness” by utilizing a known reference material, since according to the ISL, “[f]ew of the available reference drug and drug Metabolite(s) are traceable to national or international standards.” ISL 184.108.40.206. And given WADA’s proclamation (discussed above) that standard methods are not available for doping control, it would be impossible for a WADA lab to determine the “trueness” of a method by comparing the method to a second, already validated, method. My guess is that WADA labs validate the “trueness” of their methods by utilizing whatever reference materials they can find, but I have no way to know for certain that this is what they do.)
As stated above, method “accuracy” is a factor of both method “trueness” and method “precision”. “Precision” is a measure of the closeness of method results when the method is repeated under the same conditions. See Eurachem Guide paragraph 15.1.
“Precision” is itself a measurement of two other criteria, “Repeatability” and “Reproducibility”. “Repeatability” is a method’s precision where the method is performed on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time. (Eurachem Guide paragraph A21). “Reproducibility” is a method’s precision where the method is performed on identical test items in different laboratories with different operators using different equipment. (Eurachem Guide paragraph A22). To complicate matters slightly, the ISL requires that method validation for threshold substances consider a criterion called “Intermediate Precision”. (See ISL Rule 220.127.116.11.2.1.) “Intermediate Precision” is the variation in results observed when one or more factors, such as time, equipment and operator, are varied within a laboratory. See http://www.measurementuncertainty.org/mu/guide/analytical.html. In other words, “intermediate precision” is a criterion that falls somewhere in-between repeatability and reproducibility.
(Interestingly, the ISL rules briefly refer to repeatability, see ISL Rule 18.104.22.168.2.1, but never to reproducibility. This omission may reflect WADA’s relative lack of concern with achieving consistent results among its various accredited labs. One further point: we can see that when Ali points to the variation between the LNDD S17 test results and the results achieved later upon the EDF re-analysis, he is pointing to a potential problem with the “intermediate precision” of LNDD’s test methods.)
Method “accuracy” is measured differently, depending on whether the method purpose is quantitative (as it would be for WADA threshold substances) or qualitative (as it would be for WADA non-threshold substances). (The Eurachem Guide says that this distinction applies to measurement of precision, see Eurachem Guide paragraph 6.37, but it would seem to apply equally to measurement of trueness.) If the method purpose is quantitative, “accuracy” is measured by looking at the amount that the test results differ from each other and from the reference value. If the method purpose is qualitative, “accuracy” is measured based on the percentage of the time that the test generates a false positive result or a false negative result. In either case, the method’s “purpose” should define the required test accuracy.
Up to the Introduction; back to part 6; on to part 8.