Science journals in decline

A major psychology journal will be publishing a paper that purports to offer evidence for extrasensory perception, or ESP. Yup. A study that evidently followed established psychological research methods and was peer reviewed by four "trusted" experts provides "scientific" evidence in support of precognition.

It is unlikely that the publication of this study means the scientific method is broken, though it is a tempting conclusion. Over the past several years, however, scientific journals have been littered with sloppy research, claims that can't be duplicated, conclusions that defy scientific laws, like the ESP paper, and outright fraud.

On Thursday, the prestigious British journal BMJ made public the results of an investigation that showed the author of a well-known paper that linked autism to vaccinations had falsified his results. Even if fraud hadn't been an issue, the paper, originally published in the Lancet in 1998, was flimsy to begin with. It was based on interviews with a mere 12 subjects.

Not quite a sham, but still questionable, is a study released by NASA in November that claimed the discovery of a species where phosphorus was replaced by arsenic in its biological makeup, unlike any other species on the planet. It was supposed to be a key piece of evidence that extraterrestrial life existed.

Later it was pointed out that the scientists had been feeding the micro-organism salts that had been contaminated with phosphates, so there was no way of knowing whether the species could actually survive without phosphorus as had been claimed. The study has been panned as "flim-flam" and one biologist said it "should not have been published."

An investigation by Postmedia revealed in the fall that four papers authored by a Queen's University engineer had been retracted by the Annals of the New York Academy of Sciences because the professor was found to have recycled previously published research from earlier in his career.

Closer to home, a University of Manitoba scientist resigned after a paper he published in the journal Nature was found to contain doctored results in 2008.

There is a litany of cases involving either fraudulent research or research of questionable quality being published in reputable publications in recent years. Some of it is quite harmful, like the autism study that prompted thousands of parents to forgo vaccinations for their children. Others are simply an embarrassment for journal editors. All contribute to public ignorance.

No one is accusing the author of the ESP study, Daryl J. Bem, a Cornell University psychologist, of committing a fraud. He is a genuinely respected and prominent researcher. But he has been attacked on the grounds that his statistical analysis was faulty and not sufficiently scientific. In particular, critics complain he did not properly account for the possibility that his hypothesis might not be true.

At least two rebuttals have already been penned, and one is to appear in the same issue of the Journal of Personality and Social Psychology later this year. Three attempts at duplicating the results have failed.

Such concerns are not limited to high-profile retractions or wacky claims. A recent article in the New Yorker by Jonah Lehrer investigated how a growing body of research cannot be replicated despite being initially praised as groundbreaking.

Lehrer tracked what's known as the "decline effect" in several fields, including publications about second-generation antipsychotics, whether certain female insects are attracted to males based on symmetry, and whether diseases affect men and women differently.

In each case, the story was the same. At first, the research was easily replicated, but over time the confirmation rate of accepted theories declined by, in some instances, as much as 80 per cent, calling into question, if not disproving, the original hypothesis.

It is not the fact that research is sometimes disproved that is problematic, but rather the scope, and the reasons behind it.

Lehrer considered several contributing factors. He looked at whether journals are biased towards positive results, while ignoring disproving research. Human error also likely plays a role, as does the tendency for some scientists, either consciously or unconsciously, to select data that confirms a given hypothesis.

One of the scientists he interviewed has gone so far as to claim that the majority of scientific articles currently in circulation are simply untrue.

Science still remains the most reliable way of acquiring new knowledge, but these objections show just how difficult discovering that knowledge can be.

Carson Jerema is editor

of Maclean's On Campus

Republished from the Winnipeg Free Press print edition January 10, 2011 A11