Britannica asks Nature for retraction on Wikipedia comparison

Back in December, I wrote about criticisms of Wikipedia in Communications of the ACM and a study published by Nature which found that Wikipedia's coverage of scientific subjects was about as accurate as that of the Encyclopedia Britannica.

Now Britannica has demanded a retraction of the Nature study on the grounds that its "research [is] invalid, its study poorly carried out, and its findings [are] 'so error-laden that it was completely without merit.'" (Inside quote is from Britannica's response, outside quote from Seattle Times coverage.)

Britannica's website has a 20-page PDF (7 pages of response, 13 pages of supporting information in two appendixes) that is a response to the Nature study, titled "Fatally Flawed: Refuting the recent study on encyclopedic accuracy by the journal Nature." This response states that "Nature's research was invalid. As we demonstrate below, almost everything about the journal's investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading. Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of articles Nature examined were not even in the Encyclopedia Britannica."

The initial criticism of the response is that, while the Nature study headline claimed that "Wikipedia comes close to Britannica in terms of the accuracy of its science entries," the actual study showed that Wikipedia had a third more inaccuracies than Britannica.

The next criticism is that as they reviewed the alleged inaccuracies, they "discovered in Nature's work a pattern of sloppiness, indifference to basic scholarly standards, and flagrant errors so numerous they completely invalidated the results." Nature refused to supply the complete reviewer reports comparing Britannica to Wikipedia articles, so Britannica reviewed the truncated reviewer reports that had been posted to the web, along with the articles which were supplied by Nature.

Several of the Britannica articles reviewed were not from the Encyclopedia, but from editions of the Britannica Book of the Year. Britannica notes that "Yearbook authors are often given greater latitude to express personal views than writers of encyclopedia articles." In one instance, a sentence in an article on Steven Wolfram "in which point of view figured significantly" was counted as an inaccuracy. In one case, an article on ethanol, the source of the article was from the Britannica Student Encyclopedia, "a more basic work for younger readers."

A more significant flaw was that in some cases, reviewers criticized articles for omissions when they were only sent excerpts from the articles. The report notes that the reviewer of an article on lipids was sent only a 350-word introduction rather than the full 6,000-word article, which covered the items marked as omissions on the basis of the introduction alone. Similarly, what was delivered to reviewers as articles on kin selection and punctuated equilibrium were actually only sections from a longer article on the theory of evolution, and what was identified as an article on field-effect transistors was a section of the entry on integrated circuits. In another case, an article on aldol reaction was composed of selections taken from two separate Britannica articles, connected together with language apparently authored by Nature's editors.

Another flaw in the Nature study was that Nature did not require reviewers to document their assertions; where they disagreed with articles being reviewed, the reviewers were taken to be authoritative. The Britannica response supplies two examples where the reviewers were incorrect.

Finally, Nature failed to distinguish minor from major errors, treating all as equal even though Wikipedia had more significant issues, and counted as omissions cases where Britannica made editorial judgments to cover specific information in either a different way than the reviewer preferred or in other articles in the encyclopedia.

I think Britannica makes their case--the study shouldn't be relied upon as evidence that Wikipedia's coverage of science is as good as the Encyclopedia Britannica.