Setting the bar for accepting positive findings

To the editor:

The editorial by Dr Woolf¹ in the December JFP raises the important issues of how high to set the methodologic bar for accepting positive findings and whether it is fair to set that bar higher for some results than for others. There is no single evidence-based answer to the former question: Every practitioner must set that bar according to personal values and a sound understanding of clinical epidemiology. I submit that the answer to the second half of the question is absolutely yes, for reasons and in circumstances I will describe.

Evidence-based medicine offers the practitioner a number of useful tools. Among the most important is a set of “corrective lenses” for our built-in tendency as human beings to see causal relationships that do not exist. Biases in medical decision making, both clinically and in the conduct of research, are not randomly or uniformly distributed. Rather, we typically have a pronounced optimistic bias²; we have a strong preference for believing that our interventions work and for overestimating how well or how often. It is to prevent fooling ourselves in this predictable manner that we rely on the rigors of statistics, experimental design, and clinical epidemiology.

But a greater hazard still is advocacy bias: Beyond the natural tendency to see what is not there, there is the powerful desire of researchers and “disease advocates” to demonstrate positive results. We are quick to recognize this potential problem in research sponsored by pharmaceutical or device manufacturers, but have generally lived in denial of it in our “pure” academic research. Such denial is unwarranted. Large sums of research grant funding and charitable donations—and the theories or therapies on which researchers have staked their careers—may depend on positive findings.

Could bias account for the findings of mammography trials? Lacking tools that allow us to estimate the probability of Type I error (finding an effect when none actually exists) in the face of advocacy bias, evidence-based practitioners must make an informed guess. We must ask, how much pressure is there to find positive results in this case?

That question, not the minor differences in baseline characteristics cited by Goetzsche and Olsen,³ is the real red flag in considering screening mammography. The pressure to find positive results in mammography trials is very great indeed. Recall the consequences of the 1997 National Institutes of Health panel report⁴ on screening mammography for women aged 40 to 49 years: When the data failed to clearly support the desired conclusion the researchers might as well have been taken out back and shot, and the desired conclusion was substituted by the National Cancer Institute.⁵ A panel of responsible, careful scientists that dared to not find the desired conclusion was subjected to heated and even slanderous personal attacks by advocacy groups, members of the US Senate, and the press. They were even publicly accused of fraud.⁶ Seldom is advocacy bias so nakedly and vehemently evident; the fact that it has been demonstrated so strongly in connection with breast cancer screening inevitably supports the suspicion that it is present in subtler, less conscious forms as well in the conduct of research.

Thus, the bar for accepting positive results should be set higher where there is substantial reason to suspect advocacy bias and lower where advocacy interests are less powerful or approximate equipoise exists. In the case of mammography, the bar must be set high.

Where does that leave the evidence-based family practitioner? An answer (an answer, not the answer) is that the preponderance of evidence favors mammography after the age of 50 years, and an evidence-based practice would recommend it, while bearing in mind that the absolute effect size is small, and there are reasonable grounds for doubt.

Will any clinical trial ever resolve that doubt? Given the small effect size, a very large and expensive trial would be required, and the likelihood of funding such a study is low. Given the passionate advocacy for a positive outcome and the political power of the advocates, the odds that the funds for such a trial would be put in the hands of an unbeliever are very low, and thus the suspicion of advocacy bias will not likely be cleared. We are almost surely stuck with what we have.

Lee A. Green, MD, MPH
University of Michigan
Ann Arbor

REFERENCES

• Woolf SH. Taking critical appraisal to extremes: the need for balance in the evaluation of evidence. J Fam Pract 2000; 49:1081-85.
• Silverman WA. The optimistic bias favoring medical action. Control Clin Trials 1991; 12:557-59.
• Goetzsche PC, Olsen O. Is screening for breast cancer with mammography justifiable? Lancet 2000; 355:129-33.
• National Institutes of Health. National Institutes of Health consensus statement: breast cancer screening for women ages 40-49, January 21-23, 1997. Bethesda, Md: National Cancer Institute; 1997.
• Fletcher SW. Whither scientific deliberation in health policy recommendations? Alice in the Wonderland of breast-cancer screening. N Engl J Med 1997; 336:1180-83.
• Kolata G. Stand on mammograms greeted by outrage. New York Times Jan 28, 1997:C1.