Integration for Idiots, Part V: What Wolfram Meant
Part V: What the Doktor Meant
Series by Ali and TBV
In Part IV, we worked toward a result that surprised many. The presence of an unexpected peak with an unremarkable value can have significant effects on the computed values of our peaks of interest.
Let's go over what we've learned.
First, that with a single peak we can pretty easily mark the left and right integration marks, and handle a sloping background. For these purposes, peaks that are so far apart they are not visible are single peaks. There were a couple in our examples over at 1600 seconds, but they don't matter.
Second, when peaks get close enough together to interfere, things become very complicated, and all sorts of significant errors can be introduced. The kinds of interference that can occur include:
- Interference between two clearly distinguishable peaks.
- Invisible interference in a pure, undetected co-elute at the same time as a peak of interest.
- Confusing interference causes by an unexpected, undetected peak between close peaks of interests.
Now let us turn to some slides from Dr. Meier-Augenstein's presentation at the hearing, which hasn't received much attention since it was released last Friday. It is all very relevant, but we'd like to highlight some very important points that may not have been sufficiently clear during the hearing, having gotten lost among other issues. (Transcript beginning on PDF page 1137 or so).
WM-A Slide 10, circle added.This points out the fundamental premise: that you know what you are looking at is a single compound of interest. We've seen in our examples above the various kinds of interference that can result if there are things you do not expect in the vicinity of the peak you are trying to measure - either contained in the peak, or close enough to overlap.
WM-A Slide 11We've also shown in our examples above what happens when peaks overlap, and if minor peaks are contained in the peak of interest. We will end up asking questions about what LNDD has assumed, or perhaps presumed, or maybe even deemed to be the invariants of their methodology, and whether they are in fact, true.
But, let us turn our attention to the key point:
Software will not separate your peaks.
In the presence of overlapping peaks, the software, even the automatic software on the IsoPrime 2, cannot do computation that leads to correct results. If peaks of known substances overlap, the results will be wrong. If there are hidden peaks of unknown substances, the results will be wrong.
Basically, if the chemical and chromatographic separation isn't good, your results aren't good either. You can't do anything after the fact with your collected data to correct it.
Let's be completely clear about the relevance of this observation to the alternate B samples and the Stage 17 data reprocessed on the IsoPrime 2. If the conditions of chemical and chromatographic separation lead to consistently overlapped data sets, consistent results from automatic methods can still be incorrect. Consistency just means it repeatedly made the same unresolvable decisions.
WM-A Slide 16After a few examples, the Doctor addresses an alternate formulation of the same problem in slide 16:
Manual Processing Won't Work Either.
Once there are overlaps, of either known or unknowns, the results are not reliable. It's possible the software will pick the similar points and get consistent numbers, or that manual processing will allow you to get values in an "expected" range, but you won't know the truth.
With this background, we encourage you to look, perhaps for the first time, at Meier-Augenstein's presentation. Then, re-read his testimony beginning on PDF page 1131, and see if it doesn't make much more sense.
This concludes Integration for Idiots.
Our next series will be, "Idiots look at the Data". Based on our newfound understanding, do the data we see produced in the Landis case seem good enough to lead to reliable results? Are there unfounded and untested assumptions built into the collection conditions and the data set?
23 comments:
Re: Slide 16
My reading of the Meier testimony is that in only the lower right example where the peak overlap is above the baseline will there be a problem, In the lower left and middle where the overlap is below or at the baseline, the automatic correction will not distort the results.
As to small overlapping peaks causing the carbon ratios to be more negative, we have this testimony by Brenna wrt to slide 13:
"Q. Let me have you explain -- and I may need to invite you back up here -- the concept of when you have two overlapping peaks. Which contributes to the other? Your paper was quoted by Dr. Meier-Augenstein, and I want to make sure we have it right."
"A. (Witness drawing an exhibit.) Those are two peaks that are resolved. There's a baseline. Minus 28, minus 28. Our study, which was an experimental study..................
A: There's two overlapping peaks. Let me get this right. It showed that the direction would be the opposite of what was quoted, and our study was really quite an extensive study in this regard. And so, were there overlapping peaks, particularly in relationship to the 5-alpha, I would have expected it to rise, not
fall, compared to -- as an artifact. If it were an artifact, which I don't believe it is."
And Brenna wrt to the F3 chromatographs:
"In reviewing these chromatograms and the GC chromatograms that go with them and the ion traces that go with them, do you see any issue with peak interference or other problems that would cause these results to be unreliable?
A. I do not."
Transcript @ 1946-9.
PS
Sorry if this has been answered, but are you claiming that any of these potential problems with the peak integration violates a technical standard?
m,
Are you suggesting that Floyd be sanctioned on a technicality ?
Talk about role reversal. How ironic would that be.
Reality check for a moment. The discussion seems to be stating the following: The interference of another peak (which makes it difficult to establish where a relevant peak begins) makes it very difficult (or impossible) to accurately measure a peak. Is the discussion about the processing trying to establish reasonable doubt, or invalidate the results? Also, is there a suggestion that a result that has interferring peaks unable to be tested accurately? What does that say about testing for doping as a whole? I'm confused because sometimes when I'm reading the threads, I feel as though I'm being told that testing doesn't work - ALL testing for ratios of testosterone to epitosterone is far too vague. Am I on the mark, or is there another point trying to be made? Also - does this argument suggest that there was NO adverse finding whatever? Context would help here.
TBV
I am most appreciative of having the M-A power point presentation along with the Final Findings of Fact et al. I was able to see the M-A power point during the May hearing and have been waiting patiently. the Integration for Idiots Series was great as many times the discussions get very focused on legalities.
I recall from the hearing that it is necessary to have good "GC" in order to do the IRMS (Ayotte, Caitlin, Brenna). Does the three ion (or lack there of with LNDD processing) also apply to the GC/MS portion of the tests?
GMR
boston,
They are trying to suggest that the chromatography here in general, or in a number of unspecified examples, was so bad, e.g. poor peak separation, that the peak identification and analysis was unreliable.
This issue was discussed in the Majority Decision, which you can read.
Meier himself says in the quoted journal article that "there is many an application where overlapping peaks simply cannot be avoided." He then discusses various methods to correct for this problem. It seems to me that problems with peak separation will always be a question of degree.
Having talked about general potential problems, I think TBV and Ali should focus on showing specifically how any of these likely biased the F3 results.
As I point out above, Brenna testified that at least the small peak problem referred to by Meier (I think wrt to F3) did not bias the results.
TBV and Ali,
"In the presence of overlapping peaks, the software, even the automatic software on the IsoPrime 2, cannot do computation that leads to correct results. If peaks of known substances overlap, the results will be wrong. If there are hidden peaks of unknown substances, the results will be wrong."
I believe this is incorrect.
As I said before wrt to slide 16, my reading of Meier is that the software can correct for overlapping peaks except where the peak overlap occurs above the baseline.
And if you go back and read the Meier journal article you cite he discusses a number of software algorithms intended to correct for peak overlap.
Moreover "wrong" results here is really a matter of degree - what percent error or bias was introduced. 1% error is a far cry from 50% error. So far you haven't attempted to quantify the alleged error, or to show how that error rendered a specific test result, like the F3, invalid.
Boston 4:13,
You ask: "Also, is there a suggestion that a result that has interferring peaks (is) unable to be tested accurately?"
I am not sure that Ali and TbV would go that far. If so, then, if I understand right, you would just start over and try to get better separation.
But at some point this conversation has to be combined with the conversation about specificity (as mentioned in ISL 5.4.4.2.1). IF the lab had the complete mass spectra data from the GC/MS (which they don't), AND they had kept the chromatographic conditions the same between the GC/MS and the GC/C/IRMS (and they didn't), then these integration problems wouldn't be such a big deal. You would know from the mass spectra data what exactly you had measured, no matter the interference.
Presumably the way it is supposed to work is that you run the sample through the GC/MS and check the mass spec data to see that you have only measured the metabolite of interest, and if there is other stuff in there, then you do it again to get better separation, and thus cleaner peaks. Then you keep the chromatographic conditions the same when you go to the GC/C/IRMS, and you can come out very confident in your results.
Given that they don't have the complete mass spectra data AND they didn't keep the chromatographic conditions the same between the GC/MS and the GC/C/IRMS, they simply had to assume that there was only the metabolite of interest in each one of the peaks they measured. That's why M-A kept asking in different ways "But how did they know?" And thus, these integration issues are critical.
syi
m,
You said: "As I said before wrt to slide 16, my reading of Meier is that the software can correct for overlapping peaks except where the peak overlap occurs above the baseline"
I don't understand what you mean here. When you say baseline, are you referring to the background. In which case, if the peaks overlap at all, it will be above the background. The peaks aren't buried in the background, they sit on top of it.
Furthermore, I'd say that we have quantified the error - explicitly. We've given the true and computed value for the peak o/oo. If a true value of -27 can be "misdiagnosed" as being -32, then you've got a false positive. I'd classify that as wrong.
I think we're all jumping the gun here. M, Boston, even Mike: I don't think the presentation has proceeded far enough yet to allow us to reach any of the "big questions" you guys are discussing.
M, we have not gotten anywhere near a discussion of ISL technical standards.
BLT and M, we're nowhere near reaching any conclusions about how testing was done in the FL case.
Mike, I didn't hear TBV/Ali say anything about how integration problems could be resolved with reference to the mass spectra. Yes, I can see that the mass spectra would help you determine the stuff you were measuring, but I don't see how it would resolve errors in the C13/C12 measurement when peaks overlap. Nor do I see where maintaining consistent chromatographic conditions would enter into their discussion. They weren't talking about comparing results from GC/MS to GC/IRMS, or even about comparing different runs from the same machine. All of their examples were based on a single run on a single machine. I understand what you're saying from the standpoint that the mass spectra and GC/MS - GC/IRMS comparisons would help a lab avoid peak interference, but it doesn't offer any way to help us interpret interfering peaks.
But I do see two questions raised by M and BLT that perhaps could be addressed at this early point. The first is BLT's question about the inability to accurately measure interfering peaks. I think we ARE at a point in this discussion where this question can be asked. Mike reads TBV/Ali to say that if you know that you have interfering peaks in a test result, you should toss the result and start over. TBV and Ali, is that what you're saying? Or in practice, is there a minimum amount of interference that is tolerable, and if so, how can you determine that you're safely within the minimum?
Second, I AM interested in M's questions about the testimony of Dr. Brenna. If Dr. Brenna testified that peak interference doesn't matter, then I guess it's obvious that TBV and Ali disagree with Dr. Brenna ... and that this series as a whole is an explanation of WHY TBV and Ali disagree with Dr. Brenna. But I don't think that Dr. Brenna said that peak interference doesn't matter, and I'm not sure that TBV and Ali are arguing that any peak interference invalidates the test result. So ... it would be interesting to know if TBV and Ali disagree with the testimony of Dr. Brenna, and if so, what is the nature of the disagreement.
Yes, I jumped ahead. Sorry. Ali and TbV are being so patient, I should follow suit, I suppose.
syi
Let me see if I have one basic thing down here: if the actual value is -27 and you get a smaller value due to overlapping peaks (or for some other reason), this is a false positive for doping? Note to none math types:
-32 is a smaller number than -27.
Ali,
Was your -32 wrt to the F3 or just a theoretical result? Brenna's testimony is contrary in any case.
WRT to slide 16. Maybe we are disagreeing about terms.
Meier specifically says that the bottom left and center graphs were OK. Background subtraction works there. Only the bottom right graph will have error.
"Q. Okay. Let's go to the next slide.
9 Let me ask you a question: Does background
10 subtraction, as performed by LNDD, have an
11 impact or a corrective impact on the delta
12 13-carbon values?
13 A. Yes, it can have. If you look at
14 the -- if you look at the top left in
15 conjunction with the schematic on the bottom
16 left, you can see that is basically -- the
17 bottom left shows you very, very schematically
18 two peaks just about baseline separated, that
19 the baseline as such, is stable, and the
20 correction basically -- just as indicated in the
21 blue and the magenta square or that square right
22 angle shows you what the baseline correction
23 takes away, so that's fine.
24 You can even have an extreme case,
25 like shown in the middle, where the baseline slopes. And as long as the peaks are well separated or sufficiently separated -- again,
3 here is the piece on the screen shows you what
4 the baseline correction takes away.
5 If you then go into the very right
6 example, this is a case where you have peak
7 overlap. And what happens if the baseline
8 correction connects -- because, you see, it
9 can't extrapolate across. There is something
10 that's basically above the baseline, where the
11 two peaks join. So what happens is that the
12 baseline correction then connects -- left
13 corner, that's magenta, to the center up. And
14 from there, it connects down to the right hand
15 of the trailing peak.
16 And what you can"
Meier @ 1373-4 transcript.
Sorry I don't have time to pull out all the line numbers.
OOPs
left out end of the quote.
......"And what you can see, and what I
17 tried to show in this schematic, is again
18 reflected on the top. You actually are cutting
19 away now, not only the background, which would
20 be the right angles underneath, but you are
21 cutting away on top of that two triangles that
22 are actually peak information.
23 And on the left hand, you are
24 cutting away more of the carbon 12 rich part,
25 and on the right hand, you are cutting away part of the carbon 13, which means the left-hand peak
2 will again be apparently more enriched than
3 carbon 13, than it actually ought to be if it
4 would be resolved, analyzed, whereas the right-hand peak, again, will appear to be more
6 depleted.
7 Q. Which would give you a higher delta
8 value?
9 A. A higher negative.
10 Q. A higher negative delta value."
Meier @ 1374-5.
wschart, I've asked the question you've asked, and I think you're supposed to ignore the negative. If I've understood correctly, then in the world of IRMS, -32 means a higher C13/C12 ratio than does -27.
m,
The coloured areas in the bottom left and center examples represent the background areas that will be removed before they evaluate the area of each peak. I believe Part II of the series indicated the dangers of not romoving all of the background. Yes, in both those cases there would be no problem because the peaks don't overlap. The bottom right image effectively corresponds to Figs 12 and 13 of Part III and shows how errors can be made when the peaks overlap. The reason being that you don't know anymore where the real end of the first peak is and where the start of the second peak is. You've lost that information. Errors can now easily be made depending on what assumptions you make about how that shape was formed.
Ali
Larry and wschart
The -32 for the 5A androstandial for example is supposed to be subtracted from the carbon ration for the pregnane, so if the pregnane is -27, then the delta becomes 5 which exceeds the 3 delta threshold. But if the true carbon value of the 5A was -27, not -32, then the delta is (-27) - (-27) = 0.
Larry,
I responded to an earlier query of yours today, but it was back in Part III. It addresses some good points which you raised. You'll have to scroll back to pick it up.
The way it works is the more negative the number, the lower the C13/C12 ratio and the more "synthetic" the source material is. In other words the more negative the value appears, the more likely it is that the individual was doping.
The values we've been using as starting values have been -27 because that is typical of the normal (non-doping) values that you would expect. The refernce values have typically been around -26.6. The doping threshold is -3 away from the reference value so anything more negative than -29.6 would be suspect. A measured value more negative than -30.6 would be considered a definite, cast iron fail (because the declared accuracy of LNDD results is +/- 0.8, ie -29.6 - 0.8 = -30.6)
Oh no, I can't add up !
-30.4 (just a typo, honest !)
M, Mike S. - Thanks for adding that context I needed. Larry, very good read of my question - you'd said 'Or in practice, is there a minimum amount of interference that is tolerable, and if so, how can you determine that you're safely within the minimum?' - That's a far better way of phrasing it, many thanks! Yes I am probably causing static on what is otherwise a tight chemical thread, but I need an anchor once in a while!
As to that question about 'interference that is tolerable' I suppose that is what I'm waiting for. At some point in this discussion, I'm realising, all the collected data will STILL come down to interpretation, correct? And it's been suggested by both camps that each has interpreted correctly.
Now I'll be patient again...
At the moment, we're all racing ahead in the looking at date portion of the idiot's guide. However, I still have questions about the integration portion of the idiot's guide. I'll post them all here, for convenience.
1. BLT has asked the ultimate question on the idiot's guide to integration: in practice, is there a minimum amount of interference that is tolerable, and if so, how can you determine that you're safely within the minimum? I should probably leave that question to the end of this post, but it's too good a question to get lost at the end of one of my "epic" posts.
2. You concluded that the issues you raised in part I on integration do not cause problems in the real world. In other words, if we're looking at one single pair of C13-C12 peaks, then you can easily distinguish the C13 from the C12. I'm not sure why this should be. I might have thought this was because the IRMS has separate sensors for the M44 and M45 ions, but as you explained in your last (8:34 AM) post in Part III, having the separate sensors does not solve all of our problems. Back in part I, you also referenced the second graph in Figure 0, but I never understood what that graph showed or how it solved our problems.
3. Here in part V, you've jumped past the problem of background noise as introduced in part II. Did we come to the conclusion that, in the absence of overlapping peaks, we can confidently correct for background noise? Or can we only do this if the background noise is steady (as in figure 6) or sloping in a straight line (as in figure 9)?
4. I still have some major questions about Part III. In part III we start dealing with the question of a second set of C13-C12 peaks that overlap with the first set. I guess I still don't get why this is such an insurmountable problem. As you drawn these peaks, and as they appear to me in the IRMS graphs I've seen in the FL case, the peaks seem to me to be symmetrical, meaning that the left half of the peak seems always to be the mirror image of the right. If that's the case, couldn't you use the half of the peak that's not subject to interference to draw the correct shape of the peak half that IS subject to interference?
The Part IV stuff, in contrast to Parts I - III, seems easy! If there's a hidden peak adding stuff to the peaks we can see, then of course the values are going to go kabloo-ey.
Larry,
I'll have a bash at adressing your questions.
1) I'd imagine that there is a minimum amount, but I'm not sure how you'd quntify it. You could run numerous examples through the spreadsheet and draw some general conclusions, I guess. The main issue is that this is not just a two dimensional problem. We're used to looking at the m44 plot and making observations like "that interfering peak is really small" or "there's hardly any overlap" ... "it can't possibly have much effect". Well, it may seem that the interference is small on the m44 plot, but it could be much larger on the m45 plot. It depends on the C13/C12 ratios for each peak. Off the top of my head, I don't have an answer to your question, I'm afraid.
2) Fig 0 shows the plot of instantaneous m45 to m44 ratio. In other words, at each point in time, the value for m45 plot is divided by the value for the m44 plot. That's the curve that swoops up, then down, then back up. Below it there is a diagram showing the m45 and m44 plots for our peak. They don't really exist on the same plot, they've just been drawn like that to give the reader an understanding of how the m45 arrives before the m44. The top figure which shows instantaneous m45/m44 ratio is usefel because it can be used to see where the m45 starts (C13) and where the m44 (C12) finishes. If you look at the response of the plot, the first thing that happens is it moves upward. In other words the C13/C12 ratio is increasing. That's because the C13 arrived first. You now know where the C13 plot begins. If you look at the graph below and get a feel for how the ratio between the two graphs changes with time, you'll get a feel for why the top graph swoops up, then down. In the simplest case, it identifies your integration limits so that include all of the C13 and C12 contribution. It may also be of some use in characterising the nature of interference in a peak (changing baseline, co-eluting peak, etc). I'm not sure about that though. I think you'd need a lot of experience to make that sort of judgement.
3) If you know your background is a straight line between the start and stop of your peak, then chopping off everything beneath the peak would be fine. However, what if your background has a hump in it which coincides with the peak ? What if the background has a -50 o/oo vaue. That would be the same as having a -50 peak, the size of your hump in your peak of interest.
4) I believe that some investigation has been done into doing exactly what you suggest (by Brenna ?). I don't believe it has been implemented. Just to play devil's advocate, what if the raised area between the peaks is due to a hump in the background and not due to co-elution ?. How would you know ?. If you fitted a curve to where you thought it should be, you'd be including background in your measurement. Look at Fig 6 and you'll see that LNDD figured the raised area must have been due to a hump in the background (up on the 5B and down on the 5a). Look at Fig 9. Clearly a different assessment was made in this case. They've gone down the road you suggest (although very crudely, slicing the peak up). The two cases are obviously different, right ?
Ali, those are terrific answers, thank you. I know it's not easy to write about science in non-technical language (well, RELATIVELY non-technical!). I'll need to take a closer look at the figures, maybe tonight.
I think at some point down the road, we'll need to revisit BLT's question about how good is good enough. That's going to be the main question here. At the moment, you seem to lean in the direction that this is a judgment call to be made by experienced professionals. If that's true, it's not good news for FL and other athletes who challenge the judgments made by the experienced professional at the ADA labs. But it's probably too early to have this discussion. Let's wait to see what we learn as the idiot's series continues ... exclusively here on TBV!
Post a Comment