Monday, January 28, 2008

Larry's Curb Your Anticipation, Part 3:
ISO 17025

Up to the Introduction; Back to part 2; On to part 4.

By commenter Larry

ISO 17025: An Introduction

The main standard used by testing laboratories world-wide is ISO 17025 (sometimes called ISO/IEC 17025), issued by the International Organization for Standardization. ISO 17025 provides rules for how labs should operate in order to produce consistently valid results. ISO 17025 is also the basis for lab accreditation by bodies such as the American Association for Lab Accreditation.

[MORE]


The complete text of ISO 17025 is available at http://www.usocpressbox.org/usoc/pressbox.nsf/ac7bf642f496016a87256d0d006a340c/1b39aafc829ba275852572fc000de8ee/$FILE/023.PDF. The best general discussion I’ve seen of ISO 17025 is at http://www.labcompliance.com/tutorial/iso17025/default.aspx?sm=d_e.

(The version of ISO 17025 referred to in the ISL is the 1999 version, but at the time of the testing at issue in the FL arbitration, the 2005 version of ISO 17025 was in effect. USADA posted both the 1999 and 2005 versions of ISO 17025 as exhibits in the FL case, so it is not clear which version of ISO 17025 should be applied to the FL case. I have decided to use the 2005 version of ISO 17025 in this analysis. I do not believe that there are material differences between the two versions of ISO 17025 that would affect my analysis here.)

ISO 17025 applies one set of rules to be followed by all kinds of testing laboratories – not just anti-doping laboratories, or medical laboratories, or even laboratories that perform testing on human samples. For this reason, the standards in ISO 17025 are universal and general in nature. However, ISO 17025 recognizes that more specific requirements may be needed in order to ensure that specialty labs are competent to perform their jobs. For this reason, Annex B of ISO 17025 provides guidelines that certain specialty labs must use to supplement the general requirements of ISO 17025. In particular, Annex B.4 of ISO 17025 provides as follows:

[I]t may be necessary to develop a separate document of applications to supplement [ISO 17025] for specific types or groups of tests … Such a document should provide only the necessary supplementary information, while maintaining [ISO 17025] as the governing document through reference.

By its terms, Article 5 of the ISL is a supplementary document under Annex B.4 of ISO 17025, for the field of doping control. As required by ISO 17025, the ISL recognizes that any aspect of testing or management not addressed in the ISL is to be governed by ISO 17025. See ISL Rule 5.1. Moreover, all WADA laboratories are required to be accredited by a national accreditation body and periodically audited according to ISO 17025 (see ISL Rules 4.1.1 and 6.4.7.2), and all WADA lab methods and procedures must eventually be included in the scope of these periodic ISO audits. See ISL Rule 4.2.2.

The principle that ISO 17025 is applicable to anti-doping testing is expressly recognized in the decision of the majority arbitrators in the FL case, who ruled that “violations of the ISO 17025 …can be violations of the ISL for purposes of rebutting the initial presumption favouring the Lab that an AAF has been established.” See http://ia341243.us.archive.org/0/items/Floyd_Landis_USADA_Case_Decision_Documents/UsadaAndLandis-FinalAward20-09-07.pdf paragraph 156-157.

In some of my prior posts, I’ve noted that ISL Rule 7.1 contains language that appears to limit the application of ISO 17025 to anti-doping cases. The applicable language of ISL Rule 7.1 is set forth in full below:

The Laboratory is not required to provide any documentation not specifically included in the Laboratory Documentation Package. Therefore, the Laboratory is not required to support an Adverse Analytical Finding by producing, either to the Testing Authority or in response to discovery requests related to the hearing, standard operating procedures, general quality management documents (e.g., ISO compliance documents) or any other documents not specifically required by Technical Document on Laboratory Documentation Packages. References in the International Standard for Laboratories to ISO requirements are for general quality control purposes only and have no applicability to any adjudication of any specific Adverse Analytical Finding. (emphasis added)


Read broadly, the language in italics above would indicate that a lab’s violation of applicable ISO 17025 standards would not give an athlete a potential defense to an AAF found by the lab. However, this interpretation of ISL Rule 7.1 is contradicted by ISL Rule 5.1, by the express language of the FL decision, and by the fact that specific ISO 17025 requirements are cited more than a dozen times in the ISL, in ways that make it clear that ISO 17025 is an integral part of WADA drug testing. (See, for example, ISL Rules 4.1.1, 4.2.1, 4.2.2, 4.3.1, 5.2.6.6, 5.3.1.1, 5.3.2.1, 5.3.3, 5.3.4, 5.3.7.1, 5.3.7.3.4, 5.3.8, 5.3.10, 5.3.11, 5.3.13.1, 5.3.14.1, 5.4.1, 5.4.5.2, 6.2.1, 6.3.1, and 6.4.7). Consequently, the language in italics in Rule 7.1 should be understood as limited to the matters addressed in that rule. In other words, ISL Rule 7.1 should be understood to state that ISO 17025 requirements are not applicable to determine the documents that a lab is required to include in its Laboratory Document Package (LDP), but that ISO 17025 is applicable to other aspects of the lab’s work.

Let’s summarize. It is clear that in order to understand the ISL, we must also understand ISO 17025. ISO 17025 is the master document, while the ISL provides more specific rules to supplement the general rules in ISO 17025. These two documents are complementary documents that must be read together to understand the rules governing anti-doping testing in cycling. Where the ISL is silent, we must look to ISO 17025 for the governing rules. And as the ISL was written to “follow the format of the ISO 17025 document” (see ISL Rule 5.1), we should read the ISL rules within the context of ISO 17025 and within the structure set forth in ISO 17025.



Up to the Introduction; Back to part 2; On to part 4.

6 comments:

dailbob said...

Larry,

Do you happen to know if U.S citizens have a right (i.e., based in codified law) to see all of the evidence against them is U.S (or state) courts (i.e., regular courst outside of WADA)?

I ask this because not being able to request lab SOPs (in particular)robs an individual of the main (maybe only) way to determine if the tests were done properly (since SOPs govern exactly how the test is supposed to be executed). If individuals in regular courts have this right, then it seems this rule would be incongruent U.S jurisprudence. Thoughts?

Larry said...

d-bob -

That's quite a question you've posed there.

Your humble author (and he IS humble, it took him until late last night to figure out why TBV titled his article "Curb Your Anticipation") is struggling hard enough just to understand the WADA rules. I'm not qualified or expert enough to do a comparative study of the WADA rules versus, say, the Federal Rules of Civil Procedure and the Federal Rules of Evidence.

So, take the following with a larger grain of salt than usual.

Let's take a big picture approach. You're asking a question that goes to the concept of "due process". "Due process" is a constitutionally guaranteed right, but it does not mean the same process in every forum and for every dispute. If a defendant is accused of murder, he'll get a lot of due process. If you're looking at an 80 year old fighting denial of a medicare benefit, she'll receive somewhat less due process. If you're looking at an 8 year old facing a day's suspension from school, that's probably worth a tiny amount of due process.

The way you've asked your question makes me think that you want to compare WADA doping proceedings to a criminal proceeding in a U.S. state or federal court. Someone like "m" might argue that this is not a fair comparison. "m" likes to say that the FL case is more like a contract dispute between two private parties, or an employment dispute where an employer wants to fire an employee. So maybe FL is not entitled to the same kind of due process that would be available in a U.S. criminal proceeding. I'm not saying one way or the other, I just want to set up our discussion in the fairest way I can.

So ... from the point of view of a criminal proceeding ... we can say with confidence that no such proceeding could be considered to be fair or just unless the accused is presented with all of the evidence against him, and is given a reasonable opportunity to challenge that evidence (and present evidence in his defense). Which is not the same thing as saying that the defendant has to be presented with every piece of evidence that might be in the possession of the police or the district attorney (there are rules about this too -- the police is not supposed to conceal evidence favorable to the defendant -- but let's not go to that particular place for the moment). However, if the evidence is going to be used in the criminal case, argued by the district attorney, presented to the jury -- then the evidence has to be presented to the defendant and the defendant has to be given the opportunity to challenge the evidence.

That being said ...

If you go back to the first sentence of my article, you'll see that I characterized the FL case as an uncertain set of facts to be judged by a confusing set of rules. It's probably an oversimplification, but that's pretty much the law in a nutshell: you assemble the relevant facts, then you figure out the rules that are applicable to those facts.

There is always going to be a limit to ability of any side in a legal dispute to get access to the factual information in the possession of the other side. There is some point at which the legal system has to say "enough". As you might guess, it's never going to be easy to determine when this point has been reached.

I am not offended by the fact that the WADA rules limit the ability of the athlete to get at information in the possession of the labs. There has to be a limit; the question is where to set the limit. I don't like the limit set by the WADA rules, but at least in a U.S. arbitration, the arbitrators have the discretion to order additional discovery, which is a good thing.

Let's draw a contrast to a DUI case, where the primary evidence against the defendant is from a breathalyzer test administered by a police officer. To be sure, the defendant is going to see the results of the breathalyzer test, and be given the opportunity to challenge the reliability of the test. Will the defendant be able to question the police officer on how the test was performed? Sure. Will the defendant be given the opportunity to review the police department's maintenance records for the breathalyzer (assuming that there IS such a thing as a maintenance record for a breathalyzer)? Maybe. Will the defendant be able to send the breathalyzer to an independent laboratory to see if it is working correctly? Probably not. Will the defendant be able to access a history of how well this breathalyzer has worked in the past? I kind of doubt it. Will the defendant be given access to the police department's history of administering breathalyzer tests (with a comparative breakdown showing the statistics for the individual officer who performed this particular test), complete with a showing of which tests were proven to be false positives? Probably not.

At some point the court will say that the defendant is "fishing", looking into areas where there's no evidence that the area is relevant to the case.

However, in a court, the rule of what is and is not relevant is going to depend on the facts of the case. For example, if the police officer had to administer the breathalyzer test 3 times, because the breathalyzer did not work the first two times, then the issue of the maintenance of the breathalyzer becomes more relevant, and the chances of getting evidence of that maintenance should increase. In contrast, the contents of a WADA lab's document package doesn't change because (say) the quality of the lab's chromatographs is questionable. This is where the arbitrators can step in and say, additional discovery is required to serve the needs of justice. (So remember that we're not just talking about the fairness of the WADA rules but about the willingness of the arbitrators to act fairly within these rules, and then ultimately about how these arbitrators are selected in the first place.)

In any event ... I'm not the right person to talk about how to set up rules of evidence.

Next.

Remember when I set up the distinction between facts and law? You find the facts then apply the law to the facts? There has to be some end to finding facts, some limit placed on the ability of a defendant to fish for facts, and in the case of the WADA rules, we're arguing about where to set the limit. In contrast, NO fair system of law should place a limit on the ability of a party to figure out the rules.

Let's repeat. The ability of the accused to get access to the facts might be limited. The ability of the accused to get access to the rules cannot be limited. Not ever. Not in a criminal case, not when the 8 year old is being suspended from school.

There may be some discretion in how the rules are applied, even some ambiguity in how the rules are written -- it's good to limit those factors if you can, but the existence of those factors can't be helped. However, the rules should be known, or knowable. They should be equally available to both sides.

This is MY primary beef with the WADA rules. The WADA rules point to the ISL, which in many cases requires the lab to develop rules (in the lab's SOP) consistent with criteria set forth in the ISL. The athlete's defense against an AAF depends on his ability to show an ISL departure, which means in most cases that the athlete has to show that the lab departed from its SOP. If the athlete cannot see the lab's SOP, then the athlete does not have access to many of the RULES used to determine the AAF, and many of the RULES that could serve as the basis for showing a departure from the ISL.

That's not fair.

What FL was forced to do in his case was to argue that certain LNDD actions violated the CRITERIA set up in the ISL that the labs were required to use in writing various policies for inclusion into the labs' SOPs. He was forced to consider rules that are one step removed from the rules used to convict him, and the rules that could have exonerated him. You seem to be familiar with the FDA, so you probably know that there are laws passed by Congress governing food and drugs, and voluminous regulations promulgated by the FDA under those laws. You probably know that the FDA's main operating rules are contained in those regulations, and when the FDA needs to determine whether a company is doing something illegal with food and drugs, the first place the FDA looks is to its regulations. Now ... try to imagine an impure food kind of case brought against a defendant by the FDA, where the defendant was not allowed to see the FDA regulations, but instead had to try to figure out the regulations based solely on the food and drug laws passed by Congress. That defendant would be in a very bad way.

Kind of like FL.

(By the way, I think that our friend Judge Hue would have something to say here that would be shorter, written from a perspective that is more confident than mine, and a great deal more pointed!)

daniel m (a/k/a Rant) said...

Dailbob and Larry,

Seems to me, if you want to determine whether tests were performed correctly, you would need to know two things:

1) How the tests are supposed to be performed. That would be the SOPs for the tests, as well as any other supporting documentation, such as information on operating the equipment necessary for the tests (if that information isn't included in the SOPs).

2) A complete record of how the tests were actually performed, including the raw data, the interpretation and any supporting information/documentation or rationales for the interpretation given.

Without both elements, it's very hard (perhaps impossible) to accurately gauge whether the tests were performed correctly.

Since this is one of the main avenues of defense for an athlete accused of a doping violation, if the athlete doesn't have all the information, then how can he/she successfully argue that the testing was not performed correctly (and therefore, the results are not correct, either)?

Sounds like a Catch-22 to me. WADA's rules say that labs don't have to supply the SOPs. But what do does an athlete need in the case?

The SOPs. (I'm not concerned with what time the lab opens and administrative kinds of things. Just the SOPs that relate to the test(s) that are at issue.) Catch-22, eh?

Kind of stacks the deck in the ADAs favor, don't you think?

dailbob said...

Larry,

Thank you for your incredibly thorough, and eye-opening, response. I know you've said all these things before,but seeing them all in one place puts it all together for me.

This case must be as interesting to you, from a legal perspective, as it is to me from a science POV. What a mess!

Larry said...

Mr. Rant -

I was kinda hoping I'd answered the question you're raising when I responded to D-bob. Guess not.

I go back to the distinction I drew for D-bob between facts and law. The athlete has a complete right to all of the applicable rules of law. There's always going to be some limit on the athlete's right to get at the facts.

The "facts" are a bottomless pit. You can spend forever trying to get at the facts. You start with the chromatograms. You're not satisfied with the chromatograms, so you ask for the EDFs. Then you worry that maybe those aren't the right EDFs, or that the EDFs were corrupted, so you want to see evidence that the EDFs are unchanged from the day they were created. Then you worry that maybe the EDFs weren't properly created in the first place, so what you really want is access to whatever's left of the sample. Then maybe you worry that the sample was corrupted, or tampered with. And on and on.

In a case of this complexity, there are hundreds of avenues of factual inquiry that can be pursued forever. No matter what amount of information you have, there's always the NEXT layer of information beckoning, promising to provide you with the key to the Truth.

I don't mean to overdo the lawyer thing, Mr. Rant, but when you ask for a "complete" record of how the tests were actually performed, I have to ask you what you mean by "complete". The "complete" record for performance of scientific tests is a subjective standard. Complete for what purpose? So you can know what steps the lab took? At what level of detail? If I tell you that I woke up this morning, went to work, had lunch and then wrote you this reply, is that a "complete" record of my day so far? Or do you need to know what time I woke up and what I ate for lunch? It depends on why you're asking. If you're asking because you want to know if I was in Atlanta this morning when the First National Bank got robbed, you don't need to know what I had for lunch. If you're my coach at Weight Watchers, that's another thing entirely.

What you (and I, and others) want to know is: did the lab screw up? But the only level of detail sufficient for this purpose is the level of detail necessary to prove that the lab screwed up. It's like FL says: you can't prove a negative. The lab can't prove that it didn't screw up. There's no level of detail high enough to prove that the lab didn't screw up.

And even if there WAS such a thing as a "complete" record, who's to say that the record would be truthful? What would you add to the factual part of the record to verify the facts? And how would you verify the verification?

No, Mr. Rant. There's no way around the hard question by using a word like "complete". The hard question follows from the realization that there's no end to facts, and no ultimate level of detail. You have to decide the case based on a reasonably detailed set of facts (and risk the possibility that if you'd just dug down to the next level of detail, you'd have uncovered the truth). You have to draw a line: this much factual information, and no more.

It's something like the line TBV drew with me. Stop writing the opus, and start putting it out there. The opus is not as good as it would have been if I just had another week to work on it. But at some point, you have to do the best with what you have available and get on with it.

daniel m (a/k/a Rant) said...

Larry,

To answer your question about what I mean by a complete record, try this on for size:

In the few lab settings that I've worked in, we were expected to keep a record of what steps were performed, how we did them, what the results were, what data we collected, and so forth. Usually in some sort of notebook, because back in the days I was doing this kind of work, we didn't have any computerized systems to log into and record such information.

I don't need to know when you woke up, went to lunch or anything like that -- unless there were a large time gap between steps when there shouldn't have been.

So, for my purposes, in the Landis case, I'd want to see everything the lab techs recorded during their work (which should document how the tests were performed, step by step), the raw data, and any reference materials that they used to interpret the data ("my scientific experience" is not valid in this regard) in order to arrive at their conclusions.

Yes, it can be a bottomless pit. But it doesn't have to be. Show us the entire record of what they did, and then we can look at how they did the work and determine whether any errors occurred. This is the same depth of record keeping that the accreditation agencies would be looking for in an audit, from my experience with ISO.

Now, if there are huge gaps in what they recorded, that could have some consequences in how we determine what they did. If they've been diligent (as they should and are required to be) in keeping records of their work, then it will show that they did everything by the book -- or not -- as the case may be.

I do realize that even if we were to get all of the observations, data and SOPs and so forth for the Landis case that not everything in in might be the "truth." But, at least we'd have all the documentation to really understand what they did.

I do realize that at some point you must draw a line in the sand and say, "that's enough." But my frustration is that the way the rules are written, although in theory an athlete can contest a finding, he or she will never really have all of the exculpatory evidence that might help show that the lab deviated from its SOPs.

You've explained things quite well. I'll be looking forward to more of the opus.