Larry: How traceable is traceable?

Tracing Traceability

One surprising aspect of the CAS decision in the Landis case is the decision’s heavy reliance on the accreditation of the French lab as an assurance of lab quality. We may discuss a single issue that casts doubt on whether the accreditation body truly understood the methods being used at the French lab.

[MORE]


Let’s start with some basic concepts. In the Landis case, the French lab (LNDD) reported an “adverse analytical finding” (AAF) that Landis doped with exogenous (artificial) testosterone during the 2006 Tour de France. The LNDD was authorized to perform drug testing for the Tour de France, based on the lab’s accreditation as a drug testing laboratory under the rules of the World Anti-Doping Agency (WADA).

The primary set of WADA rules governing labs is WADA’s International Standard for Laboratories (ISL). Incorporated within the ISL is ISO 17025 (sometimes called ISO/IEC 17025), issued by the International Organization for Standardization. ISO 17025 provides operating rules governing testing labs world-wide; the ISL provides specific rules for the field of doping control. The ISL recognizes that any aspect of testing or management not addressed in the ISL is to be governed by ISO 17025. See ISL Rule 5.1.

The ISL provides that all WADA laboratories are required to be accredited by a national accreditation body and periodically audited according to ISO 17025 (see ISL Rules 4.1.1 and 6.4.7.2), and that all WADA lab methods and procedures must eventually be included in the scope of these periodic audits. See ISL Rule 4.2.2. In addition, all WADA labs are separately accredited by WADA. See ISL Rule 4.1. To be certain, the national accreditation body is supposed to consider whether the lab’s methods and procedures comply with ISO 17025. To my knowledge, there’s nothing in the WADA rules that requires the national accreditation body to consider whether a lab’s methods and procedures comply with the ISL, though of course WADA would consider this compliance when it performs its own audits.

From the CAS decision, we’ve learned that the national accreditation body for LNDD is the Comité Français d’accrédtation, or COFRAC.

One question we’ve kicked around here at TBV is exactly how much is accomplished by accreditation of laboratories. Is lab accreditation a reasonable assurance that all lab methods have been properly validated, and that the lab’s quality control is up to par? We’ve also wondered about the dual accreditation process provided for by the ISL. Given that WADA accredits its labs according to ISL standards, is it reasonable to expect that the national accreditation body will ALSO closely examine the ISL and learn all of the rules applicable to drug testing? Or is it more reasonable to assume that the national accreditation body would focus solely or primarily on the ISO 17025 rules, and leave it to WADA to worry about the ISL. After all, a body such as COFRAC has only a single WADA lab under its jurisdiction. It’s a lot to expect for COFRAC to learn and apply all of the WADA ISL rules, when this knowledge will be applicable only to its review of LNDD.

My view of lab accreditation is based in part on a discussion of this issue in the first (AAA) arbitration decision in the Landis case. The AAA decision appears to have viewed the COFRAC accreditation as covering ISO 17025 only. On this basis, the AAA panel rules that the lab could not claim that its national accreditation was proof of its compliance with the ISL. See AAA decision paragraph 163.

But the CAS decision took a different view of the COFRAC accreditation. The CAS panel found that the COFRAC accreditation was evidence that LNDD’s procedures were valid and in compliance with the ISL. See for example Decision paragraph 76 on compliance with ISL rules on quality control, paragraphs 101 and 108 on peak identification methods, and paragraph 115 on the lab’s use of manual integration.

The reasoning of the CAS panel is based on its assumption that the COFRAC auditor was familiar with the ISL rules, familiar enough to spot any aspect of LNDD’s procedures that might be in violation of the ISL. For example, the CAS panel ruled that COFRAC’s accreditation is proof that the LNDD’s manual integration methods were OK under the ISL. Had the LNDD’s SOP for manual integration “been a violation of the ISL,” ruled the panel, “the COFRAC auditor who reviewed the SOP and watched [the lab technician] perform manual integration would surely have noted a deficiency.” (Decision paragraph 120).

Was the CAS panel correct on this point? Did COFRAC have a full and complete understanding of the ISL?

We do not have the complete COFRAC accreditation document available to us – at least I don’t THINK we do (it’s hard to keep track of what we have and what we don’t have). But we have bits and pieces of the COFRAC accreditation. And one of these bits and pieces is STRONG EVIDENCE that COFRAC did not understand the ISL.

The evidence in question comes from the statement of Dr. Christine Ayotte, the head of the WADA lab in Montreal. This statement is now available at TBV. See the Ayotte Statement.  In her statement, Dr. Ayotte wrote that COFRAC’s audit confirmed the lab’s compliance with the ISL, and she even translated a portion of the COFRAC accreditation from the original French. Her translation reads as follows:

“i) the competence of the laboratory personnel, its very good knowledge of the methods, know-how and knowledge of mass spectrometry, their extensive training record; ii) that the quality management complies with the norm, traceability is satisfactory; iii) that the number of instruments was appropriate as well as their quality; iv) that the IRMS method was well controlled; v) that the criteria for identification were conform to WADA requirements; vi) that the laboratory had highly satisfactory results in the WADA proficiency tests.”

We’ll put to one side any complaint we might have about the grammar of Dr. Ayotte’s translation (who here could do a better job translating English lab documents into French?). We’ll also pass over any disagreement we might have concerning the competence and training of LNDD personnel. Let’s focus instead on one little statement from the COFRAC accreditation:

“traceability is satisfactory”

LNDD’s traceability is satisfactory? This HAS to be news to everyone who carefully read my “Curb Your Anticipation” series. I discussed traceability at length in that series. Your ears all perked up at the mention of traceability, right?

Right?

Well, maybe not. Let’s review some portions of the “Curb Your Anticipation” series.

A fundamental ISL concept is stated in ISL Rule 5.4.4.1: “Standard methods are generally not available for Doping Control analyses.” In other words, lab methods to detect doping are in their infancy – they’re not well established at this point – so the WADA labs have to develop new methods for testing. This situation is the exception for lab testing world-wide, not the rule. More often, if a lab wants to validate a testing method, it does so by comparing the method to another validated method already in use. But as indicated in the ISL, WADA labs cannot validate their methods by comparing them to other validated methods – instead, these labs must “develop, validate, and document in-house methods” for drug testing.

A second way that a lab might validate a test method is to perform the method on a known standard. For example, the lab might run its CIR tests on a “standard” mix that is known to have a particular CIR measurement, and see how the results of its tests match up against the known standard. But once again, the ISL tells us that WADA labs cannot determine the “trueness” of their methods by utilizing a known reference material. In the language of the ISL, “[f]ew of the available reference drug and drug Metabolite(s) are traceable to national or international standards.” ISL rule 5.4.6.1.

Why am I reminding you of this esoteric discussion from months ago? Because COFRAC says that LNDD’s “traceability is satisfactory”. And “traceability” refers to the ability of a test method to relate to a known standard. See the Eurachem Guide, at paragraph A30. In other words, “traceability” means that the “trueness” of a method has been measured by testing the method against a standard that is well-accepted in the scientific community. But because the ISL acknowledges the lack of national or international standards in the field of doping control, TRACEABILITY DOES NOT APPLY TO MOST FORMS OF WADA LAB TESTING.

How can the LNDD’s traceability be satisfactory, if the ISL tells us that “[f]ew of the available reference drug and drug Metabolite(s) are traceable to national or international standards”? Obviously, LNDD’s traceability is not “satisfactory” – no WADA lab’s traceability can be satisfactory in the absence of accepted standards. COFRAC did not even need to visit LNDD to determine that the lab’s traceability was not “satisfactory” – a mere understanding of the ISL rules would be sufficient to indicate that LNDD traceability was not “satisfactory”.

OK, but if traceability does not apply to WADA drug testing, then why would COFRAC have bothered to mention it? COFRAC’s mention of traceability is likely due to the importance of this concept under ISO 17025. In fact, ISO 17025 devotes an entire section to traceability, see ISO 17025 section 5.6. And COFRAC, which must be very familiar with ISO 17025 concepts like traceability, threw this concept into the list of the nice things they had to say about the LNDD’s methods.

Not realizing that traceability is barely a relevant concept inside of a WADA lab.

This is just my best guess, my not a scientist or anything. Others who know more are free to weigh in here.

But I think it’s safe to say, if COFRAC specifically saw fit to mention traceability in its write-up, they probably don’t know much about the ISL. Despite the CAS panel’s statement to the contrary. This would indicate that COFRAC’s review of the LNDD procedures says NOTHING about whether these procedures are in compliance with the ISL.

Then there’s the question of why the esteemed Dr. Ayotte thought it would help her case to bring COFRAC’s endorsement of LNDD traceability to our attention. Do you suppose that she … ? Or maybe that she … ?

Naah. Couldn’t be.