Science-based medicine conference, part 1

This year's "The Amazing Meeting" (TAM7) was preceded by a Science-Based Medicine conference, organized by Steven Novella and the Science-Based Medicine blog; the speakers were all contributors to that blog. This summary is from my hand-written notes--I've tried to quote and summarize accurately, but keep in mind that some of the quotations and bulleted items may not be verbatim. The conference was videotaped and may become available via DVD or online video; keep an eye on the SBM blog for that. Steven Novella has posted a short summary of the SBM conference at the SBM blog.

I am in the process of posting a summary of TAM7 itself, which begins here. (I summarized 2008's TAM6 here.)

Novella's Introduction
The SBM conference began with Novella's presentation, titled "Science-Based Medicine: Science and Pseudoscience in Clinical Decision Making," which distinguished science-based medicine (SBM) from the recently popular "evidence-based medicine" (EBM) and explained the motivation for promoting the SBM concept.

Novella, assistant professor of clinical neurology at Yale University School of Medicine, began with a slide titled "Foundations of Medicine" which described some historical varieties of medicine:

• Superstition/philosophy-based medicine.
• Scientific medicine.
• Evidence-based medicine (EBM).
• Eastern vs. Western medicine.
• Complementary and alternative medicine (CAM).

He made the point that modern scientific medicine, which arose in the western world, is relatively young, though attempts to put it on a more scientific footing go back much farther.

Next, he discussed "What is science," first observing that "scientifically formulated" is a meaningless marketing label, then noting that there is a battle of memes about science between its defenders, who use it as a label connoting "objective truth, quality, and professionalism," and its critics, who use it as a label connoting "arrogance, oppression, and elitism." Marketers also use it "to imply product safety, effectiveness, and overall value."

He argued that we should adopt "common sense standards" which require that treatments "reasonably account for all available evidence," use "valid and internally consistent logic," have been rigorously and methodically investigated and judged with fair and unbiased criteria, and are conducted by practitioners who adhere to "standards of ethics and professionalism."

Evidence-Based Medicine
Next, he looked at evidence-based medicine, a term of art in use for the past couple of decades. EBM begins with the premise that "products and practices that work and are safe are better than those that don't work or are unsafe," an uncontroversial premise. It promotes scientific investigation as the measure of what works and is safe. But, according to Novella (and later, other speakers), EBM has "too much focus on evidence, and not enough on logic and prior probability, and good science must consider both." He argued that EBM made sense at the time it was introduced, because practices were being used largely "because they made sense, not because of supporting evidence." The introduction of EBM effectively "leveled the playing field, but also opened it up to implausible treatments," with bad timing due to the rise of complementary and alternative medicine (CAM).

The standard reference for evidence in the EBM framework is the Cochrane Reviews. The data reported in the Cochrane Reviews includes not only tests of legitimate medical treatments, but of completely implausible research such as chiropractic treatment for migraine. The problem with the framework is that it assumes that everyone is "playing fair," it does not account sufficiently for fraud or publication bias (such as the "file drawer effect"), it ignores prior probability, and it "doesn't adequately consider the big picture of the entire literature." According to Novella, with EBM it is typical to see the quality of studies decline over time, in order to continue to yield positive results for implausible treatments.

He then discussed "The Work of John Ioannidis," who argues that "most published research findings are false." The reasons are that

• The majority of highly-cited initial medical research is later refuted.
• There's a bias towards publishing positive studies.
• There's a bias towards researchers publishing provocative research.
  

And a "low prior probability worsens the effect," i.e., studies of treatments with low prior probability are more likely to be refuted.

Science-Based Medicine
By contrast with EBM, Novella identified the following features to distinguish Science-Based Medicine, SBM:

• It affirms high-quality science as a basis for standard of care in medicine.
• It acknowledges the consilience of science.
• It considered scientific medical plausibility of an intervention when weighing evidence.
• It considers the overall pattern in the literature.

In other words, SBM considers prior probability in a Bayesian sense as part of the evaluation, it looks at whether there is other scientific evidence that casts doubt on the plausibility of a suggested treatment (like violating the laws of physics or including unknown entities and mechanisms), in addition to merely looking at the specific results of controlled trials of the particular treatment. CAM, in particular, is loaded with claims that have extremely low and near-zero prior plausibility, as evidenced by the fact that $1.2 billion of U.S. taxpayer funding to the National Center for Complementary and Alternative Medicine since its founding in 1991 (originally as the Office of Alternative Medicine) has so far yielded zero effective treatments for anything.

He stated that finding anomalies argues for deeper research, and we can't have two inconsistent views that both work. Here, I think he overstated his case, since we have had cases in science where there are mutually inconsistent theories that both work, though we also take the inconsistency as grounds that something is wrong and ultimately needs to be reconciled (e.g., light wave/particle duality, quantum field theory vs. general relativity). Even theories that are wrong at some level can still work for solving certain kinds of problems (e.g., Newtonian physics)--and I'd agree with Novella's claim that scientific medicine is still in its infancy. A point Novella didn't make that I would like to insert here is that when you have two inconsistent views that doesn't mean that only one of them must be wrong--they could both be wrong.

Novella did go on to mention two cases where things that seemed initially implausible or lacking in mechanism have turned out to be correct, the postulation of dark energy in physics, and, more directly relevant to the topic at hand, the use of botulinum as a treatment for migraine. This treatment seemed to him completely implausible even though the evidence of trials suggested its effectiveness, and now a mechanism has been discovered and is understood. (My Google searching on this subject, however, yielded some recent evidence that it is not a good treatment for migraines and is no better than placebo, so this appears to me to still be somewhat controversial.)

Clinical Decision-Making
Novella ended his talk by talking about the process of clinical decision-making and pitfalls that arise as a result of human psychology and limitations. While clinical decision-making "individualizes the best available evidence to a specific patient" and "considers risk vs. benefit in both therapeutic and diagnostic intervention," this evaluation needs to include not acting as an alternative. In some cases, screening for certain diseases causes more harm than not performing the screening test, because conducting the test will yield far more false positives than true positives. (This is an effect discussed in some detail in John Allen Poulos' book, Innumeracy, and is a reason not to do things like mandatory HIV screening as a condition of a marriage license, drug testing of grade school students, and certain kinds of security screening for terrorists--if your baseline prevalence of what you're testing for is very low, your false positive results will swamp your true positive results.)

He briefly discussed the claim that "surgery kills more people than car accidents," noting that it doesn't really compare against the outcomes that would occur without surgery--far more deaths.

He then recounted some examples of pitfalls in the clinical context, such as the human capacity for pattern recognition even when the pattern isn't really there (pareidolia), the tendency to be "unduly influenced by quirky experience" or to "value experience over evidence," "failure to consider alternatives," "over-reliance on non-specific signs and symptoms," and confirmation bias (e.g., the sorts of heuristics and biases discussed in Kahneman and Tversky's classic Judgment Under Uncertainty). He then listed a few logical fallacies, pointed out the confounding factor of the placebo effect, and a couple of statistical effects--regression to the mean and the fact that most diseases are self-limiting.

Q&A
In the Q&A session, someone asked what Novella thought of legislation supporting evidence-based medicine, apparently referring to $1.1 billion in the stimulus package for evidence-based medicine research. Novella said that he thought conceptually it was a good idea but wasn't familiar with the specifics of the legislation. Another question was whether, given the current state of health care and the desire for reform, SBM would be challenged or supported. Novella said that the delivery of healthcare is a separate issue from how we decide what to research or what treatments are appropriate, and that things will either get much better or much worse. If he had also added that things might also stay about the same in overall quality, I'd say he's certainly correct; without it, merely probably correct.

(Part two of my conference summary, on cancer quackery, is here. Part three, on chiropractic, is here. Part four, on evidence-based medicine and homeopathy, is here. Part five, on chronic Lyme disease, is here. Part six, on online health and social media, and the closing Q&A panel, is here.)