Winnipeg Free Press - ONLINE EDITION

Measuring performance shouldn’t be child’s play

  • Print

CALGARY — Timing is everything.

I had just accepted an invitation to speak on performance measurement and reporting at the International Deming Research Seminar at Fordham University in New York when Alberta Health Services (AHS) released its second quarter (Q2) performance report. When giving technical presentations on stuff like performance reporting, recent real world examples can really help.

The AHS Q2 performance report details results for over 50 performance measures — everything from knee replacement wait times to immunization rates to staff absenteeism. It’s worth checking out. AHS calls it a "thorough analysis of our performance" but it’s really just 90 pages of childish nonsense.

Nowhere is this more apparent than in the report’s ‘Performance Dashboard’. It compares Q2 results for all 50+ performance measures with results from the previous quarter, the previous year and a performance target. For example, the Q2 wait time for radiation therapy (first consult) of 5.0 weeks is compared to the 4.4 weeks of the previous quarter. The report concludes that it’s now taking longer to see the radiologist.

If the sophisticated mathematics and interpretations of this are beyond you, relax. AHS provides symbols to help, in this instance, adding a little red circle to the dashboard indicating that the longer wait time is ‘bad’. If our radiologists don’t know that 5.0 weeks is longer than 4.4 weeks, maybe it’s time for new radiologists.

The report also compares the Q2 result of 5.0 weeks with last year’s Q2 result of 6.0 weeks. Because the current performance is better than the previous years’, a little green square for ‘good’ is placed on the dashboard. This apparent tie is broken by another red circle because the 5.0 weeks is still above the ‘acceptable target’ wait time of 3.0 weeks. Acceptable to whom isn’t specified.

All this creates a mishmash of ‘bad’ red circles and’ good’ green squares with some ‘we have no idea’ yellow triangles thrown in for good measure. Collectively, all these numbers and colourful symbols demonstrate nothing more about performance than an understanding of elementary school arithmetic.

This should help my New York presentation because these two-numbers-at-time comparisons always get big laughs at conferences. There’s even a long standing joke among statisticians, scientists and engineers describing the inanity of this type of performance reporting. Question, What do you call any two data points? Answer: An executive trend! Well, like I say, it’s big with statisticians.

But while good for laughs, the juvenile approach to reporting can have serious consequences, not the least of which is misleading the public on the state of health-care system performance. Here’s how that happens.

The whole point of performance measurement is identifying signals in amongst the noise. Most kindergarten students can tell you that 5.0 weeks is less than 6.0 weeks. That’s hardly insightful. What managers and the public need to know is whether this drop in wait time is important. Is it the result of a significant or important change in the system itself — a signal? Or is it simply the result of a bunch of little things coming together at different times and in different ways producing small, random, fluctuations in year to year results — or noise? (Nick Silver’s aptly titled recent bestseller, The Signal and the Noise, discusses this at length and is worth a read.) Performance measurement that can’t tell the difference is useless.

Which brings us back to the AHS performance scorecard. It can’t distinguish signals from random noise, and because of this, treats every difference between the numbers as a signal. Meaningless noise is sold to the public as important findings. Kindergarten arithmetic is spun as thorough performance analysis, giving the mistaken impression that system leadership has a handle on things.

In fairness to AHS, it’s hardly unique in reporting performance in this way. Government reporting on system, process or program performance is often of similar quality. Analysis of municipal performance in the Ontario Municipal Benchmarking Initiative, and of hospital performance at the Canadian Institute for Health Information, are two examples coming immediately to mind.

If this is the best performance measurement and reporting government can deliver then two things are true. One, we have no idea how well government is performing, and two, performance isn’t going to get better anytime soon.

 

Robert Gerst is a partner in charge of operational excellence and research and statistical methods at Converge Consulting Group Inc. He is author of The Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement and numerous articles in peer-reviewed publications.

 

—Troy Media

Fact Check

Fact Check

Have you found an error, or know of something we’ve missed in one of our stories?
Please use the form below and let us know.

* Required
  • Please post the headline of the story or the title of the video with the error.

  • Please post exactly what was wrong with the story.

  • Please indicate your source for the correct information.

  • Yes

    No

  • This will only be used to contact you if we have a question about your submission, it will not be used to identify you or be published.

  • Cancel

Having problems with the form?

Contact Us Directly
  • Print

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

Have Your Say

New to commenting? Check out our Frequently Asked Questions.

Have Your Say

Comments are open to Winnipeg Free Press print or e-edition subscribers only. why?

Have Your Say

Comments are open to Winnipeg Free Press Subscribers only. why?

The Winnipeg Free Press does not necessarily endorse any of the views posted. By submitting your comment, you agree to our Terms and Conditions. These terms were revised effective April 16, 2010.

letters

Make text: Larger | Smaller

LATEST VIDEO

O'Shea says the team is going to stick to the plan after first loss

View more like this

Photo Store Gallery

  • MIKE.DEAL@FREEPRESS.MB.CA 100615 - Tuesday, June 15th, 2010 The Mane Attraction - Lions are back at the Assiniboine Park Zoo. Xerxes a 3-year-old male African Lion rests in the shade of a tree in his new enclosure at the old Giant Panda building.  MIKE DEAL / WINNIPEG FREE PRESS
  • JOE BRYKSA/WINNIPEG FREE PRESS Local- A large osprey lands in it's nest in a hydro pole on Hyw 59  near the Hillside Beach turnoff turn off. Osprey a large narrow winged hawk which can have a wingspan of over 54 inches are making a incredible recovery since pesticide use of the 1950's and  1960's- For the last two decades these fish hawks have been reappearing in the Lake Winnipeg area- Aug 03, 2005

View More Gallery Photos

Poll

Which of Manitoba's new landlord-tenant rules are you looking forward to most?

View Results

View Related Story

Ads by Google