Science-based medicine conference, part 4: evidence-based medicine and homeopathy

This is part four of my summary of the Science-Based Medicine conference at TAM7, which will be followed by a summary of TAM7 itself. Part one, Dr. Steven Novella's introduction, is here. Part two, Dr. David Gorski on cancer quackery, is here. Part three, Dr. Harriet Hall on chiropractic, is here.

The next session was Dr. Kimball Atwood, an anesthesiologist who is also board-certified in internal medicine and associate editor of the Scientific Review of Alternative Medicine. He spoke on "Why Evidence-Based Medicine is not yet Science-Based Medicine," or "Do Clinical Studies of Highly Implausible Methods Help or Hinder?"

Dr. Atwood's talk made some points right at the start similar to the critique of evidence-based medicine (EBM) given in Dr. Novella's introduction--that EBM makes the mistake of devaluing what he called "pre-clinical knowledge." Unlike Dr. Novella, however, he also stated that random controlled trials (RCTs) "cannot, by themselves, overturn well-established medical principles." I'm not sure that his talk actually demonstrated that point--as stated, it sounds like "well-established medicalscientific principles" are part of a "hard core" of medical theory (in Imre Lakatos' sense) that cannot be refuted even if found to be in conflict with empirical results, because they are shielded by the addition of auxiliary hypotheses to salvage it. I don't think that was his intent, as surely even "well-established medical principles" should be eligible for revision in light of contrary empirical evidence. Instead, what I think he meant by "pre-clinical knowledge" and "well-established medical principles" is really more like a demand for consilience with the rest of scientific knowledge, adherence to logic and mathematical principles, and having a plausible mechanism (or at least not having a purported mechanism in conflict with other known facts). More on that in the "Q&A" section, below. [UPDATE (July 21, 2009): As Dr. Atwood notes in the comments, I incorrectly transcribed what his slide said. Also note his further discussion on what he means by a "plausible mechanism.]

He began his talk with an overview of EBM--EBM advocates, to quote EBM pioneer David Sackett, "use of the best available evience, especially from patient-centered clinical research." It relies on randomized controlled trials and systematic reviews. Dr. Atwood gave two examples of successes of EBM. First was the standard practice of giving heart attack patients anti-arrhythmia drugs post-myocardial infarction, which EBM trials showed causes excess deaths and was an incorrect practice. Second was the Women's Health Initiative study on hormone replacement theory, which showed that risks exceed benefits for taking estrogen. (I'm no authority, but I am skeptical of this claim based on my understanding of the flaws in that study from conversations with the late Chris Heward, who is co-author on a paper in Fertility and Sterility that challenged the WHI study for methodological flaws which made it unable to detect cardioprotective effects.)

The four steps of "pull" EBM are:

1. formulate an answerable question
2. track down the best evidence
3. critically appraise the evidence
4. individualize, based on clinical expertise and patient concerns

Atwood stated that those in practice the longest perform the worst, by not keeping up to date with their fields.

Cochrane Collaboration
Atwood next turned to the Cochrane Collaboration, the organization that maintains a library of EBT results, classified by type of evidence and evaluated with reviews in the form of evidence statements and recommendations. He gave a few examples, such as a statement about the effect of physical activity on dementia: "[There is] insufficient evidence to determine the effectiveness of ... physical activity programs in managing or improving ... dementia." And a seemingly equivalent statement about the use of homeopathy for the same purpose: "In view of the absence of evience, it is not possible to comment on the use of ... homeopathy in treating dementia." This, according to Dr. Atwood, is B.S. -- Bogus Science. Why don't they just say that homeopathy doesn't work? Because there are no sound clinical trials in the database.

EBT categorizes evidence into three classes. Class I evidence involves randomized controlled trials (RCTs). Class II involves controlled trials without randomization (or several other forms of case-control studies). Class III involves one or more case studies of a treatment without a control group and is considered insufficient for a treatment to be called "evidence-based medicine." Within each class there are further divisions, for example, Class I is broken down further, with systematic reviews of RCTs at the top (1a), followed by individual RCTs (1b), and so forth.

Atwood objected that this devalues pre-clinical knowledge by making "evidence" synonymous with clinical trials, and that therefore EBM "is not based on all of the evidence." He provided a few more examples of complementary and alternative medicine (CAM) treatments from Cochrane, described with terms like "Not enough evidence to ...," "little evidence," and, in the case of laetrile as a cancer treatment, "No studies found that met inclusion criteria." There was, however, a 1982 New England Journal of Medicine clinical trial on 180 patients which yielded a negative result and the conclusion that the treatment is dangerous, but this constituted a case study at the lowest level of evidence in EBM.

Atwood quoted a statement from Edzard Ernst, co-author with Simon Singh on the chiropractic-critical book, Trick or Treatment, to the effect that "a priori plausibility has become less and less important" as a result of EBM (which Ernst advocates).

He then continued with more CAM treatments in Cochrane, such as craniosacral therapy, reflexology, Kirlian photography ("may be more reliable than chance"), and therapeutic touch ("remains controversial").

Homeopathy
Atwood then described the case of homeopathy in order to make his central argument critical of EBM. Samuel Hahnemann invented homeopathy in 1796, on the basis of two principles. The first principle is "similia similibus curantur," or "like cures like," or the "law of similars." This claims that if you find a substance that gives you symptoms similar to an illness, that's the substance you use to cure that illness. Hahnemann read in William Cullen's "A treatise of the materia medica" that cinchona bark could be used to treat malaria (now known to be true because of quinine in the bark). He gave himself a sample of that bark while healthy and observed that he developed symptoms that were similar to malaria. From this single example, he concluded that all medicines produce symptoms in healthy people similar to the symptoms of diseases they effectively treat.

The second principle of homeopathy is the "law of infinitesimals." He reasoned that dosages sufficient to produce overt symptoms were too high, so the substances should be diluted in order to provide an effective treatment, and in fact the more diluted, the more powerful the cure. Homeopathic remedies of 24X (or 12C) are the equivalent of diluting 0.36mL of salt into a volume of water the size of the Atlantic Ocean. Hahnemann most commonly recommended an even greater dilution of 60X (30C). For all intents and purposes, homeopathic remedies of standard dilutions are indistinguishable from the water used to dilute them.

Atwood went on to note that homeopaths do not agree on prescriptions for various maladies.

On top of that, the outcome of all trials to date have been failures. This is a long list of powerful reasons for rejecting homeopathy, but the last one is the only one EBM considers relevant. There is clearly a very low degree of plausibility for homeopathy independently of such trials, and homeopathy is a clear case in point that "some hypotheses are too implausible to spend time on (or spend more time on)."

Prior Probabilities
Atwood offered the following set of broad categories of prior probability and types of treatments that fall into them:

Prior probability of about zero: homeopathy, neurocranial restructuring (putting balloons up your nose and inflating them).

Prior probability significantly lower than (<<) 1: metabolic therapies for cancer, detoxification, chiropractic for any purpose other than back pain Prior probability very low: acupuncture for pain most popular herbal claims (St. John's wort, echinacea) Prior probability moderate to high: massage, relaxation techniques for anxiety reduction and chronic pain Prior probability depends on:

• basic science
• cogency of theory
• previous studies
• source

He then discussed Bayes' Theorem, which says that the odds of a hypothesis being true given certain evidence is equal to the odds of the hypotheses (prior probability) multiplied by the Bayes factor, which is the probability of the evidence given the hypothesis divided by the probability of the evidence given the null hypothesis. If the prior probability is near zero, then much greater evidence is needed in the Bayes factor to result in a probability of the hypothesis being true. (Note that Atwood gave the theorem partly in terms of odds rather than probability for the purposes of his talk. Odds = probability / (1 - probability).)

When Bayes' Theorem is taken into account, the p-value of a statistical result can become much less impressive. For example, with a p-value of .05, which means that a result would be expected to occur by chance 5 times in 100 with a Bayes factor of 2.7, if the prior probability is only 1%, that result only raises the posterior probability to 3%. If the prior probability is 20%, it raises it to only 40%. With a p-value of .01, a result expected to occur by chance only 1 time in 100, and a Bayes factor of 15, a prior probability of 1% is raised to 13%; a prior probability of 20% raised to 78%. Dr. Atwood provided a table with more detail that went up to p=.001 (result expected by chance 1 in 1000 times).

Dr. Atwood advocated that "prior probability ought to be formally considered in EBT," and gave the further example of a "positive" trial for intercessory prayer in the critical care unit (CCU) with an 11% reduction in some harm with a p-value of .04, and noted that if the prior probability was 1%, this still produces less than 6% odds of a genuine effect. A few other similar examples were given involving acupuncture, homeopathy, and distant healing, the final example of which had Edzard Ernst as a study co-author and concluded that it "warrants further study," but which he subsequently backed away from after "some positive trials [were] found to be fraudulent."

Finally, he noted that pre-trial knowledge is not sufficient, but is necessary for a treatment claim.

Q&A
I asked Dr. Atwood if, in his final statement, he was saying that you have to have a plausible mechanism for a treatment in order for a treatment to be justified (a positive requirement), or if he only wanted to impose a negative requirement that the proposed mechanism or method did not contradict known facts from other realms. His initial answer was that he thought those would be equivalent, but I observed that we can discover cause-effect relationships without having any knowledge of the underlying mechanism, such as Mendel's discovery of genetics. At that point, he agreed that he just wanted to require the negative condition. Another audience member then suggested that this might be accomplished by creating a categorization scheme for levels of plausibility that in some way parallels the levels of evidence scheme.

Another questioner asked how to standardize assignment of prior probabilities and address bias, to which Dr. Atwood said that you could just pick neutral prior probabilities, since if you do enough studies the posterior probability of each study becomes the prior probability for the next.

EXTRA: As appropriate for a talk that touched on homeopathy, prior to Dr. Atwood's presentation this excerpt from the fourth episode of season three of "That Mitchell and Webb Look" was shown to the audience:



(Part five of my conference summary, on chronic Lyme disease, is here. Part six, on online health and social media, and the closing Q&A panel, is here.)