Thursday, January 31, 2008

Larry's Curb Your Anticipation, Part 13: Misapplication

Up to the Introduction; back to part 12; on to part 14.

By Commenter Larry

At the same time, there are some things that would be misapplication of method validation rules to the assessment of results. The Majority of the AAA Panel seems to have made mistakes of this sort.


  • Can the criteria for method validation be taken outside of the context of method validation, and used to judge the validity of a test result? The answer to this question seems to be a qualified “yes”. For example, in the FL case, FL argued that the LNDD violated ISL Rule by failing to properly generate chromatograms that avoided matrix interference. During the arbitration hearing, both sides called expert witnesses to testify to the quality and reliability of these chromatograms. Predictably, the witnesses for USADA stated that these chromatograms were reliable, and the witnesses for FL disagreed. The majority of arbitrators ruled that these chromatograms “perhaps could have been better”, but ultimately found that the chromatograms were “fit for the purpose” and that the chromatography “unquestionably indicates the presence of exogenous testosterone in [FL’s] “A” and “B” samples.” In reaching this ruling, the majority arbitrators made the following statement, a statement that (in my opinion) completely misconstrues the nature, purpose and intent of the ISL:

In applying the language of the ISL what is required is that the “method should avoid interference.” The language is not mandatory. Had the drafters intended that matrix interference be avoided it would require wording such as “shall” or “must”. For this Panel to accept the submissions of the Respondent that matrix interference must be avoided would be a misconstruction of ISL Dr. Ayotte confirms this statement in noting that a laboratory does not violate Article of the ISL just because it produces a chromatogram that contains matrix interference. Therefore, even where matrix interference has occurred in the Stage 17 chromatograms it would not amount to a violation of the ISL. (FL decision paragraph 240.)

The majority’s reasoning here illustrates the danger inherent in reading rules out of context. The majority managed to interpret ISL Rule to exclude any requirement to consider matrix interference – a reading that is possible only by failing to read the entire rule! (As we’ve seen, this rule includes a requirement that lab methods must achieve specificity, and the concept of “specificity” includes within it the need to avoid matrix interference.) But the majority’s reasoning is faulty on a more fundamental level, as the majority never considered ISL Rule as applying to the lab’s method validation effort. In the context of method validation, a statement that a method “should avoid” matrix interference takes on a different meaning than the one found by the majority arbitrators. In the context of method validation, this statement would be read in light of the overall goal of method validation: to make a reasonable effort (within the constraints imposed by the cost and time required for the effort) to determine that a method is “fit for purpose”. In this context, the “should avoid interference” language in ISL Rule might be interpreted to mean “do the best you can to avoid interference” or “make a reasonable effort to avoid interference”, consistent with the overall principle that a method cannot possibly be “fit for purpose” if it cannot provide us with a reasonable assurance that the method is measuring what we want it to measure.

So, let’s return to the question posed at the beginning of this bullet point: can the criteria for method validation be taken outside of the context of method validation, and used to judge the validity of a test result? The answer is “yes”, at least with respect to some of these criteria, but we need to be careful in how we do this. To be certain, if we want to assess the validity of a particular test result, we would want to consider criteria such as accuracy, uncertainty, specificity and interference. But we should avoid applying the specific language of ISL method validation rules embodying these criteria to our assessment of a particular test result. Instead, we should look to other rules within the ISL and ISO 17025 – rules that were written for the purpose of assessing a method result rather than the method itself – to provide us with the necessary guidelines for determining the validity of a particular method result.

Up to the Introduction; back to part 12; on to part 14.


Mike Solberg said...

Excellent Larry. Imagine that, further confirmation that the arbs didn't really understand what they were dealing with.

Someone said a few days ago that instead of the hearing panel consisting of three lawyers and one scientific advisor, it should be made up of three (expert in the field) scientists and one legal advisor. That seems right me.


Larry said...

syi, I'd take Suh, Jacobs and Judge Hue over Brenna, Ayotte and Botre. I'd probably take a jury of 12 over the panel of experts, but that just might be my current mood.

I'm particularly glad you liked this part.

Mike Solberg said...

If you remove the obvious conflicts of interest from Brenna, Ayotte and Botre, I'd even take them over a bunch of lawyers.