Idiots look at Data, Part I. What is good?

UPDATED:

Having explored causes of error in "Integration for Idiots", we presented the argument that crappy collected data yields crappy results that are not reliable.

We take as axiomatic that when there are significantly overlapped peaks of unknown composition that no reliable CIR can be computed by common methods. There are three qualifications in that statement.

1. Significant overlaps.
2. Unknown composition.
3. Common methods.
   

[MORE]




There is, perhaps, debate to be had about what constitutes a significant overlap, but fortunately, we need not address that here. It should be inarguable that when there is a third peak contained fully in the region occupied by two other peaks that there is significant overlap. See figure 17 in Part IV for an example about which we should have no doubt the overlap is significant.

We include common methods only for completeness. There is speculation some methods being researched by Brenna (what his $1.3 million are for) will be able to resolve some cases by working in multiple dimensions at the same time to resolve ambiguity. These methods are not widely deployed, and are certainly not in use by the LNDD.

We will also point out that, by definition, the complete occlusion of a third peak as shown in both figures 15 and 17 of Part IV makes it a peak of unknown composition. It has not even been identified as a peak, much less had its composition identified.

It is scientifically invalid to report CIR results for an impure peak. Whether doing so is invalid per the ISL is a different question we will leave aside for legal minds to consider.

We will in the future of this series look at all the Landis F3 chromatograms, since those are the ones at issue in the case. Before we do so, we'll identify things to be looking for.

• Clean, unambiguous baselines suggesting good chemical separation of the prepared samples.
  
• Significant (a debatable term) baseline (chromatographic) separation of peaks.
• Absence of shoulders suggesting unidentified peaks.
  
• Measurement of nearby peaks to consider their potential for influence.

We are looking for hints of the sorts of interference shown in "Integration for Idiots". If there are none present, then we may be inclined to think the chemical and chromatographic separation is good, and we are unlikely to have unidentified co-elutes and interference. But if we do find hints of interfering data, we are right to be skeptical of the quality of the data and want more evidence of proper identification and purity.

[Updated Digression: Earlier versions of this post must have caused guffaws and spilled coffee over at the offices of our friends at HRO, story at Idiots R Us.]

Before we look at the Landis chromatograms, it seems useful to look at some that do seem to be good, and unambiguous.

The one I'd like to show is in GDC 1362 I(and there are probably more useful ones around it), but I've had a hard time getting a really good copy.

(It seems to be something of a hot potato. USADA fought very hard to keep it from being shown, when Suh attempted to introduce it during cross examination of Catlin. I suspect continuing redaction issues have discouraged Landis from releasing it in original form. It was the cause of a Matt Barnett as William Novak moment. when he went out of his way later in the hearing to effectively identify the athlete whose test this was from. Apparently, proving the redaction wasn't great was more important to USADA than maintaining the confidentiality of the athlete in question, and showing that USADA will burn those who don't play ball.)

Fortunately, commenter M found a reasonable one in WMA's slideshow on page 40, but the resolution isn't great.

Even so, it is not be difficult to spot the difference between this and the chromatograms produced by the LNDD.

Figure 1: UCLA Chromatogram from WMA's slide 40.

So, it does appear possible to generate pretty clear chromatograms in a Testosterone IRMS. It is not the fundamental science of the method that produces garbage. Let's look closer at the payload part of the plot.

Figure 2: Zoom in on Figure 1.

What we don't see in this chromatogram is all of the junk at the beginning of the chromatogram typical in an LNDD result. Nor do we see anything that hints of an adjacent peak, or a shoulder, or a background that might be confused with a hidden peak, with the possible exception of the first true peak at around 1200s which might be bleeding towards the one that follows. There are exactly the four target peaks, and one pulse that looks like a reference.

Clean looking data.

These look like our pure, theoretical examples.

If there is support for the claim they are pure peaks, this is a very reliable looking case. We compliment the scientists and lab technicians at UCLA for producing excellent results, even if Don only wants to call them "pretty good". We understand the IRMS chromatograms produced by Ayotte's group at Montreal are about the same quality.

We can see why USADA didn't much like this being shown. Without this, you might think that what LNDD produces is typical.

In Part II, we'll start looking at the Landis chromatograms.