Uncertainty -- what about that 0.8 delta unit?

In all editions of the slide show, Arnie Baker has made a point to mention the 0.8 delta unit claimed accuracy of the CIR test, and removed it from the -3.57 value to get a value smaller than the 3.0 "positive" level. This has been met with skepticism here at TBV and elsewhere. The main argument against this is that the accuracy had to have been factored into the 3.0 cutoff level.

In a recent slideshow, Baker attempts to justify this, with a reference to The role of measurement uncertainty in doping analysis", which we've come across for the first time now. The abstract seems clearly on point:

The determination of measurement uncertainty is a critical issue in all fields of experimental science; its importance becomes maximal in the specific case of forensic analytical chemistry (including doping analysis), where uncertainty has not only to be calculated with precision, but it also has to be both small and reliable enough to support effective decision making. This contribution gives a general view of the organisation of the activity of the network of the anti-doping laboratories accredited by the World AntiDoping Agency (WADA), focusing in particular on the current situation related to the determination of measurement uncertainty in doping analysis. Representative examples referring to the procedures followed in the case of threshold substances are also presented and discussed.

In particular, on page 378, it says the following, which we'll call The Paragraph.

In practice, to report an adverse analytical finding, it is necessary to identify a prohibited substance, and, in the case of substances with a reporting threshold, to measure a value exceeding the threshold; in the latter case, it is mandatory to express the measurement uncertainty. It is not unusual for a result apparently exceeding the threshold, if taken as a single value or even as a mean value to not be correctly reported as `above the threshold' if the measurement uncertainty is not taken into account. In Figure 2 it is evident that only case `E' shows a value that is above the threshold also taking into account the measurement uncertainty.

On the surface, this supports Baker's approach, so were left with some questions:

1. It this paper in any way authoritative?
2. Is the CIR a "threshold" test to which the approach applies?
   

Authority

The paper is not an official WADA publication, so it's difficult to claim it is a controlling document that must be obeyed by the arbitrators. However, it is by Spiroto and Botre, the Quality Manager and Scientific Director of the the Rome laboratory. The Paragraph says, "In practice, to report...", so we are in the realm of interpretation, not hard rule.

At the same time, the conclusion of the paper asserts that the goal of the labs and WADA is to,

ensure that all methods followed by the laboratories are `fit to purpose', the latter being to provide `clear and convincing' evidence of any detectable doping offence based on an adverse analytical finding.

which would argue that failure to consider the uncertainty may fail the 'clear and convincing' part of the goal.

It appears to us the authority of this argument rests on ethical and moral grounds, not on a de-jure reading of the rules. This can be seen as a failing in the rules.


Applicability to the CIR

Should we conclude that the uncertainty bars need to be taken into account for substances with threshold amounts, we then need to determine whether the CIR is a threshold test or not. This is not clear cut either. The description in WADA TD2004EAAS, in the same paragraph with the dreaded "metabolite(s)", does not use the word "threshold":

The results will be reported as consistent with the administration of a steroid when the 13C/12C value measured for the metabolite(s) differs significantly i.e. by 3 delta units or more from that of the urinary reference steroid chosen. In some Samples, the measure of the 13C/12C value of the urinary reference steroid(s) may not be possible due to their low concentration. The results of such analyses will be reported as “inconclusive” unless the ratio measured for the metabolite(s) is below -28‰ based on non-derivatised steroid.

But, we also notice for the first time that the "3 delta units" is not offered as a hard value either -- it is used an an example ("i.e.") of what ought to be the controlling restriction, "differs significantly".

If one takes 3.0 as a fixed value, then it looks like a threshold value, and it's a fair argument to make that the uncertainty should be taken into account.

If one does not take 3.0 as a fixed value, and latches on to the "differs significantly" wording, then there is clearly discretion allowed in the rule, and one can equally argue the uncertainty should be taken into account. In fact, the rule as written encourages this view, because it doesn't say three-point-oh, it says "3 delta units", indicating (to us) an intentional vagueness of the precision.

Because of the vagueness of the rule as written, there is clearly opportunity for variance in lab reporting of the same measured values. This is inconsistent with the WADA goal of harmonization, but it's not clear that needs to be considered by the arbitration.

All in all, it's a nice mess were left with, Ollie.

Attempts to "simplify" the issues all seem to come back to the same Rorshach -- do you think they are all dopers, so you convict shades of gray, or do you think grey means "probably not black?"

Our opinion, after looking at the paper and reading the rule again is changed. We're inclined to accept the argument that the 3.57 metabolite should not be considered a positive.

That leaves us with the previously unresolved problem of whether the single metabolite at 6 delta units is sufficient to declare a positive. And we're not going there now.

update: Further discussion of this topic continues at DPF.

update 2: An emailer sends the following on 6-Mar-2007:

Readers interested in learning more about the importance of measurement uncertainty may want to visit the following National Institute of Standards and Technology (NIST) webpage: http://physics.nist.gov/cuu/Uncertainty/index.html

NIST Technical Note 1297 is available for free via: http://physics.nist.gov/Pubs/pdf.html

If some enterprising readers wish to look at the Uncertainty paper and think about how it applies to the Landis case, we'd be obliged. I have a suspicion the LNDD quoted uncertainties are not in conformance with the NIST recommendations, but I don't understand the implications.

[end]