Analyzing the studies

Enlarge Image

Enlarge Image

What works on a lab rat may not work on humans. (THE ASSOCIATED PRESS)

Enlarge Image

Vitamin Y is good for you and can add years to your life, reads a news story reporting the latest, head-turning research to appear in a reputable medical journal.

So you head to the drugstore as fast as you can to load up on the miraculous vitamin.

A few weeks later, you come across news about another study claiming the same vitamin Y may be linked to early death.

What should you believe? Should you trash your recent purchase? Are most journalists reporting about so-called junk science? Why are you being bombarded with conflicting health news?

"I'm not concerned that the information is out there ... because that's the nature of scientific research," says Dr. Matthew Stanbrook, a Toronto-based lung specialist and researcher. "It's how you distill what's important from it."

Maybe you consider yourself news-savvy and disregard studies funded by corporations. Or perhaps you ignore studies that appear to be too small to matter.

You might be surprised to find out what makes health studies credible is not always so black and white.

Stanbrook, who is also deputy editor of the Canadian Medical Association Journal, a century-old publication that's considered one of the most reputable in the country, spoke to the Free Press about how to decipher the latest headline-making science.

Here's his advice:

LOOK FOR STUDIES THAT APPEAR IN REPUTABLE, PEER-REVIEWED JOURNALS

Prominent journals are usually peer-reviewed, meaning every study they publish has been examined and approved by several experts in the field. "That's some sign that there has been some attempt from the scientific community to vet what's in there," says Stanbrook. He notes, however, there's a "wide spectrum" of quality even among peer-reviewed journals, so always approach studies with a skepticism.

UNDERSTAND THE PROCESS SCIENTISTS GO THROUGH BEFORE THEIR WORK IS PUBLISHED

Scientists want to see their work published in journals, where it garners the most attention and perceived credibility. So they submit their research papers to journal editors who, if they are interested in the study, then send it to experts in the field for further criticism. This back and forth process between the authors and the expert critics often goes a few rounds. "At the end, if the authors seem to satisfy all the criticisms being raised, the paper gets published, often with a lot of improvement from the initial submitted version."

DON'T ASSUME THAT SCIENCE EQUALS DEFINITIVE FACTS

"Not all scientifically valid findings turn out to be true. They can be contradicted by future studies, for example," says Stanbrook, noting that's the nature of all science. That means you probably shouldn't take mega-doses of the latest supplement to make the headlines without consulting your physician.

DON'T DISREGARD STUDIES FUNDED BY CORPORATIONS, BUT ALWAYS BE SKEPTICAL

"A good scientist is skeptical of everything they read. And a good physician and a good member of the public should be skeptical of new things they hear about in science. That's part of science," says Stanbrook. "That doesn't mean we should dismiss studies because they're funded by companies that have an interest in selling their drug." He says that all scientists are likely to have some degree of bias; even scientist/academics who want to further their point of view. He says industry is unlikely to engage in "outright fraud" but may tend to leave information out of their studies. "It's not that they lied about what they did. It's what they don't tell you."

KNOW THAT SIZE AND LENGTH MATTERS

In general, the larger the study and the longer it is, the better. "But there are nuances around that," explains Stanbrook. He says too few patients in a study can leave questions about whether the study's outcome was accidental or not, or whether it affects men and women the same way. At the same time, studies that are too big also pose problems. (When you study population databases of 12 million people, there are many variables between the people, making coming up with a conclusion difficult.) What number makes a study too small or too large? That depends on what the study is trying to find out, says Stanbrook.

QUESTION A STUDY'S SCIENTIFIC METHOD

Experts consider randomized, double-blind, placebo-controlled studies among the best in the business. (That's when neither the researchers nor the patients know what treatment they are receiving. There's also a control group which is getting a placebo disguised as treatment/medicine). Problems with these types of trials? The patients in the experiment often don't represent a realistic cross section of the types of patients physicians tend to see. Also, they are expensive and take a lot of time to complete, says Stanbrook.

A randomized crossover trial (like the kind the Free Press wrote about last week) allows all groups to get the treatment before they are given a resting period to "wash out" the effects of the last treatment from their systems. The benefits? It allows researchers to conduct a fairly reliable study with smaller numbers and less money. The downside? Stanbrook says the effects of each treatment may not "wash out," thereby skewing results.

Keep in mind there are numerous scientific methods, each with pros and cons.

BE AWARE WHAT WORKS FOR LAB RATS MIGHT NOT WORK ON PEOPLE

"People are not rats," says Stanbrook, although he notes that lots of promising research starts with animal studies. That's because "you can do the kinds of things with animals you can't do with humans" such as gene manipulation. At the same time, "the fact you can wipe out all cancers in rats with this treatment doesn't necessarily meat you'll be doing it to people the next week. It's just a hopeful development," he says.

REALIZE THAT REPETITION IS GOOD

Look for other studies that prove the same thing as the study-of-the-day, says Stanbrook, noting that repetition in science is the ultimate proof a finding is legitimate.

GO TO THE SOURCE

Rather than relying on a short television news segment for your information about the latest science, try reading the study itself, says Stanbrook, noting that his journal's website is getting more traffic than ever from the public rather than just doctors. He says that although studies aren't written in plain language, the editorial comment that goes with it often is and can give the public a good idea about details a news story may have missed.

Follow Shamona on Twitter: @ShamonaHarnett

Have an interesting idea you'd like Shamona to write about? Contact her at shamona.harnett@freepress.mb.ca

Republished from the Winnipeg Free Press print edition July 23, 2012 D1