Winnipeg Free Press - PRINT EDITION

How CBC and others torque ratings

  • Print

CALGARY -- The CBC's Fifth Estate recently broadcast Rate My Hospital, a "sweeping cross-country investigation into Canada's hospitals.

"Manufacturing scaremongering headlines on junk-science hospital ratings" however, would have been a better description. The last time there was an "investigation" such as this, they found Galileo guilty.

If the yellow journalism wasn't enough, interviews of health-system administrators trying to explain their poor ratings only added to the sleaziness. It's like asking confidence game victims to explain how they were conned before they realized they have been.

Yorkton Regional Health Centre spokeswoman Sharon Tropin did an admirable job explaining a D rating, criticizing the rating tool for having only five indicators. That's a good point.

Here's a better one. The CBC rankings are fairy tale products of an elaborate con game. Statistical techniques are misused to manufacture phony findings while giving the (false) impression of scientific rigor.

The Fifth Estate's website indicates the statistical methodology was developed by biostatisticians noted for their work in evaluating health system performance. Their credentials are impressive.

But science doesn't work by credentials. It works by data, and evidence produced through rational, logical, analysis. Evidently, researchers substituted extensive data torture for rationality and logic, the better to hear what was wanted, I guess.

Here's how this data torture took place:

Hospital report card grades were determined by measuring the number of standard deviations a hospital is from the average for all hospitals. (Standard Deviation is a measure of the variability in the data.)

For example, a grade of A is given to a hospital whose standardized performance score is one or more standard deviations from the average. A grade of B is given if the hospital score is between 0.5 and 0.999 standard deviations better than the average, and so forth all the way to D. There are three problems with this.

First, the expected operating range of any process is +/- 3 standard deviations. Using +/- 1 standard deviation means the CBC had to narrow the definition of expected performance because the data were otherwise unable to produce any findings. In other words, the data supported no conclusions of scientific or practical importance so statistical technique was manipulated to manufacture some.

Second, combined with the system of grades, this data torture transforms measured ratings into ordinal rankings, and ranking is a way of lying with statistics. If your data proves nothing, ranking makes it look as though it did.

The CBC's report card proves nothing more than that, in a ranking of hospitals (or anything else), some will be at the top, some at the bottom, and about half will be below average. I wonder how much the CBC paid to validate Grade 10 arithmetic?

Third, and most importantly, the method rests on a logical fallacy. The report card standard deviations are the probabilities of obtaining a performance rating assuming hospitals are the cause. But the CBC conclusions concern the probability the hospital is the cause given the ratings.

It's a statistical sleight of hand called the fallacy of the transposed conditional, and yes, it's confusing. But that's what makes this con so effective.

Think of it this way. The CBC measured the probability that a person would die after jumping off Toronto's CN Tower. It then used these same probabilities to conclude that a man found dead in Saskatchewan must have jumped from the Toronto landmark.

Admittedly, the CBC is not alone in using these confidence game techniques. They are ubiquitous in reporting from provincial health systems, quality councils, health ministries, and the Canadian Institute for Healthcare Information. But shouldn't the media be exposing scientifically dishonest and incompetent practices rather than promoting them?

The health care system in Canada has enough problems. It doesn't need hacks manufacturing new ones.

 

Troy Media columnist Robert Gerst is a partner in charge of operational excellence and research & statistical methods at Converge Consulting Group Inc. He is author of The Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement and numerous articles in publications.

 

--Troy Media

Republished from the Winnipeg Free Press print edition April 18, 2013 A15

Fact Check

Fact Check

Have you found an error, or know of something we’ve missed in one of our stories?
Please use the form below and let us know.

* Required
  • Please post the headline of the story or the title of the video with the error.

  • Please post exactly what was wrong with the story.

  • Please indicate your source for the correct information.

  • Yes

    No

  • This will only be used to contact you if we have a question about your submission, it will not be used to identify you or be published.

  • Cancel

Having problems with the form?

Contact Us Directly
  • Print

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

Have Your Say

New to commenting? Check out our Frequently Asked Questions.

Have Your Say

Comments are open to Winnipeg Free Press print or e-edition subscribers only. why?

Have Your Say

Comments are open to Winnipeg Free Press Subscribers only. why?

The Winnipeg Free Press does not necessarily endorse any of the views posted. By submitting your comment, you agree to our Terms and Conditions. These terms were revised effective April 16, 2010.

letters

Make text: Larger | Smaller

LATEST VIDEO

Peguis Chief Hudson comments on toddler's death upgrade to homicide investigation

View more like this

Photo Store Gallery

  • A young gosling flaps his wings after taking a bath in the duck pond at St Vital Park Tuesday morning- - Day 21– June 12, 2012   (JOE BRYKSA / WINNIPEG FREE PRESS)
  • MIKE APORIUS/WINNIPEG FREE PRESS STANDUP - pretty sunflower in field off HWY 206 near Bird's Hill Park Thursday August 09/2007

View More Gallery Photos

Poll

What should the city do with the 102-year-old Arlington Street bridge?

View Results

View Related Story

Ads by Google