Winnipeg Free Press - ONLINE EDITION

Measuring performance shouldn’t be child’s play

  • Print

CALGARY — Timing is everything.

I had just accepted an invitation to speak on performance measurement and reporting at the International Deming Research Seminar at Fordham University in New York when Alberta Health Services (AHS) released its second quarter (Q2) performance report. When giving technical presentations on stuff like performance reporting, recent real world examples can really help.

The AHS Q2 performance report details results for over 50 performance measures — everything from knee replacement wait times to immunization rates to staff absenteeism. It’s worth checking out. AHS calls it a "thorough analysis of our performance" but it’s really just 90 pages of childish nonsense.

Nowhere is this more apparent than in the report’s ‘Performance Dashboard’. It compares Q2 results for all 50+ performance measures with results from the previous quarter, the previous year and a performance target. For example, the Q2 wait time for radiation therapy (first consult) of 5.0 weeks is compared to the 4.4 weeks of the previous quarter. The report concludes that it’s now taking longer to see the radiologist.

If the sophisticated mathematics and interpretations of this are beyond you, relax. AHS provides symbols to help, in this instance, adding a little red circle to the dashboard indicating that the longer wait time is ‘bad’. If our radiologists don’t know that 5.0 weeks is longer than 4.4 weeks, maybe it’s time for new radiologists.

The report also compares the Q2 result of 5.0 weeks with last year’s Q2 result of 6.0 weeks. Because the current performance is better than the previous years’, a little green square for ‘good’ is placed on the dashboard. This apparent tie is broken by another red circle because the 5.0 weeks is still above the ‘acceptable target’ wait time of 3.0 weeks. Acceptable to whom isn’t specified.

All this creates a mishmash of ‘bad’ red circles and’ good’ green squares with some ‘we have no idea’ yellow triangles thrown in for good measure. Collectively, all these numbers and colourful symbols demonstrate nothing more about performance than an understanding of elementary school arithmetic.

This should help my New York presentation because these two-numbers-at-time comparisons always get big laughs at conferences. There’s even a long standing joke among statisticians, scientists and engineers describing the inanity of this type of performance reporting. Question, What do you call any two data points? Answer: An executive trend! Well, like I say, it’s big with statisticians.

But while good for laughs, the juvenile approach to reporting can have serious consequences, not the least of which is misleading the public on the state of health-care system performance. Here’s how that happens.

The whole point of performance measurement is identifying signals in amongst the noise. Most kindergarten students can tell you that 5.0 weeks is less than 6.0 weeks. That’s hardly insightful. What managers and the public need to know is whether this drop in wait time is important. Is it the result of a significant or important change in the system itself — a signal? Or is it simply the result of a bunch of little things coming together at different times and in different ways producing small, random, fluctuations in year to year results — or noise? (Nick Silver’s aptly titled recent bestseller, The Signal and the Noise, discusses this at length and is worth a read.) Performance measurement that can’t tell the difference is useless.

Which brings us back to the AHS performance scorecard. It can’t distinguish signals from random noise, and because of this, treats every difference between the numbers as a signal. Meaningless noise is sold to the public as important findings. Kindergarten arithmetic is spun as thorough performance analysis, giving the mistaken impression that system leadership has a handle on things.

In fairness to AHS, it’s hardly unique in reporting performance in this way. Government reporting on system, process or program performance is often of similar quality. Analysis of municipal performance in the Ontario Municipal Benchmarking Initiative, and of hospital performance at the Canadian Institute for Health Information, are two examples coming immediately to mind.

If this is the best performance measurement and reporting government can deliver then two things are true. One, we have no idea how well government is performing, and two, performance isn’t going to get better anytime soon.

 

Robert Gerst is a partner in charge of operational excellence and research and statistical methods at Converge Consulting Group Inc. He is author of The Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement and numerous articles in peer-reviewed publications.

 

—Troy Media

Fact Check

Fact Check

Have you found an error, or know of something we’ve missed in one of our stories?
Please use the form below and let us know.

* Required
  • Please post the headline of the story or the title of the video with the error.

  • Please post exactly what was wrong with the story.

  • Please indicate your source for the correct information.

  • Yes

    No

  • This will only be used to contact you if we have a question about your submission, it will not be used to identify you or be published.

  • Cancel

Having problems with the form?

Contact Us Directly
  • Print

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

You can comment on most stories on winnipegfreepress.com. You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

Have Your Say

New to commenting? Check out our Frequently Asked Questions.

Have Your Say

Comments are open to Winnipeg Free Press print or e-edition subscribers only. why?

Have Your Say

Comments are open to Winnipeg Free Press Subscribers only. why?

The Winnipeg Free Press does not necessarily endorse any of the views posted. By submitting your comment, you agree to our Terms and Conditions. These terms were revised effective April 16, 2010.

letters

Make text: Larger | Smaller

LATEST VIDEO

Exciting changes expected for Saturday's Santa Claus parade

View more like this

Photo Store Gallery

  • Two baby tigers were unveiled at the Assiniboine Park Zoo this morning, October 3rd, 2011. (TREVOR HAGAN/WINNIPEG FREE PRESS)
  • A  young goose stuffed with bread from  St Vital park passers-by takes a nap in the shade Thursday near lunch  –see Bryksa’s 30 day goose challenge Day 29-June 28, 2012   (JOE BRYKSA / WINNIPEG FREE PRESS)

View More Gallery Photos

Poll

Would you visit Dalnavert Museum if it reopened?

View Results

Ads by Google