CALGARY -- The CBC's Fifth Estate recently broadcast Rate My Hospital, a "sweeping cross-country investigation into Canada's hospitals.
"Manufacturing scaremongering headlines on junk-science hospital ratings" however, would have been a better description. The last time there was an "investigation" such as this, they found Galileo guilty.
If the yellow journalism wasn't enough, interviews of health-system administrators trying to explain their poor ratings only added to the sleaziness. It's like asking confidence game victims to explain how they were conned before they realized they have been.
Yorkton Regional Health Centre spokeswoman Sharon Tropin did an admirable job explaining a D rating, criticizing the rating tool for having only five indicators. That's a good point.
Here's a better one. The CBC rankings are fairy tale products of an elaborate con game. Statistical techniques are misused to manufacture phony findings while giving the (false) impression of scientific rigor.
The Fifth Estate's website indicates the statistical methodology was developed by biostatisticians noted for their work in evaluating health system performance. Their credentials are impressive.
But science doesn't work by credentials. It works by data, and evidence produced through rational, logical, analysis. Evidently, researchers substituted extensive data torture for rationality and logic, the better to hear what was wanted, I guess.
Here's how this data torture took place:
Hospital report card grades were determined by measuring the number of standard deviations a hospital is from the average for all hospitals. (Standard Deviation is a measure of the variability in the data.)
For example, a grade of A is given to a hospital whose standardized performance score is one or more standard deviations from the average. A grade of B is given if the hospital score is between 0.5 and 0.999 standard deviations better than the average, and so forth all the way to D. There are three problems with this.
First, the expected operating range of any process is +/- 3 standard deviations. Using +/- 1 standard deviation means the CBC had to narrow the definition of expected performance because the data were otherwise unable to produce any findings. In other words, the data supported no conclusions of scientific or practical importance so statistical technique was manipulated to manufacture some.
Second, combined with the system of grades, this data torture transforms measured ratings into ordinal rankings, and ranking is a way of lying with statistics. If your data proves nothing, ranking makes it look as though it did.
The CBC's report card proves nothing more than that, in a ranking of hospitals (or anything else), some will be at the top, some at the bottom, and about half will be below average. I wonder how much the CBC paid to validate Grade 10 arithmetic?
Third, and most importantly, the method rests on a logical fallacy. The report card standard deviations are the probabilities of obtaining a performance rating assuming hospitals are the cause. But the CBC conclusions concern the probability the hospital is the cause given the ratings.
It's a statistical sleight of hand called the fallacy of the transposed conditional, and yes, it's confusing. But that's what makes this con so effective.
Think of it this way. The CBC measured the probability that a person would die after jumping off Toronto's CN Tower. It then used these same probabilities to conclude that a man found dead in Saskatchewan must have jumped from the Toronto landmark.
Admittedly, the CBC is not alone in using these confidence game techniques. They are ubiquitous in reporting from provincial health systems, quality councils, health ministries, and the Canadian Institute for Healthcare Information. But shouldn't the media be exposing scientifically dishonest and incompetent practices rather than promoting them?
The health care system in Canada has enough problems. It doesn't need hacks manufacturing new ones.
Troy Media columnist Robert Gerst is a partner in charge of operational excellence and research & statistical methods at Converge Consulting Group Inc. He is author of The Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement and numerous articles in publications.