Winnipeg Free Press - PRINT EDITION

StatCan survey data worthless

  • Print

CALGARY -- There is much talk today of "big data." As our world goes digital, companies, governments, not-for-profits, and organizations of all sorts obtain vast quantities of data about, well, virtually everything.

But is the data any good? So far, the talk has been about quantity, but not quality. That may be because there is so little quality to speak of.

Take, for example, the first data releases from the National Household Survey of Statistics Canada. Released earlier this month, the quality of the results has come under criticism because the voluntary NHS survey replaced the compulsory long-form census questionnaire. In effect, this replaced a random sample with a non-random sample. Non-random samples have their place, but making conclusions about the population isn't one of them.

As a result, no conclusions about the Canadian population can be drawn from the NHS. Since making these types of conclusions is the whole point of a census, the NHS data is worthless. (This is also true for any survey where participation is voluntary, including citizen, customer and employee satisfaction surveys.)

This is why, in resigning as the head of Statistics Canada, Munir Sheikh wrote in an open letter to the prime minister:

"I want to take this opportunity to comment on a technical statistical issue which has become the subject of media discussion... the question of whether a voluntary survey can become a substitute for a mandatory census... It cannot."

Later, Statistics Canada's high-profile chief economic analyst, Phil Cross, also resigned, citing the same concerns. Cross is currently research co-ordinator with the MacDonald-Laurier Institute.

That's two people resigning over a matter of principle. Refusing to compromise scientific integrity for personal or political gain, are admirable actions. Arguably, a level of integrity is all too rare in the Canadian public service and now, totally absent at Statistics Canada.

The NHS replaces sound scientific sampling and data collection with meaningless motherhood pronouncements. Saying: "The Agency is aware of the risks and associated adverse effects on data quality and is currently adapting its data collection and other procedures to mitigate the impact of these risks" is not science; it's spin.

No amount of public relations nonsense from the Communications Office of Statistics Canada can produce a reliable statistical inference from a discretionary sample.

Nor can mailing more surveys. Statistics Canada's claim that; "To promote data accuracy, this voluntary survey will be sent to a larger cross-section of households than the old long-form census," is not evidence of risk mitigation but of statistical incompetence. Sample size doesn't compensate for sampling bias. Quantity can't replace quality. Statistics Canada's mitigation strategy is nothing more than piling it higher and deeper in the hopes nobody will notice that it's all the same BS.

The news releases accompanying the initial release of the NHS results encourage this confusion between quantity and quality. For example, Statistics Canada claimed a high quality of results for the NHS at a national level, but cautioned that the numbers were less reliable for smaller population centres because of low response rates. This is the reason given for withholding the results of one quarter of Canadian municipalities.

But the truth is, the results at a national level are no more reliable than results for any one of the over 1,000 municipalities that had their results withheld. This is because reliability cannot be measured when the sample isn't random and voluntary surveys aren't, by definition, random (as Sheikh makes clear). In publishing results for larger population areas then, Statistics Canada is claiming reliability where none exists and perpetrating what amounts to a scientific fraud on the Canadian public.

Troy Media columnist Robert Gerst is a partner in charge of operational excellence and research & statistical methods at Converge Consulting Group Inc. He is author of The Performance Improvement Toolkit: The Guide to Knowledge-Based Improvement and numerous articles in peer-reviewed publications.

Republished from the Winnipeg Free Press print edition May 18, 2013 J11

Fact Check

Fact Check

Have you found an error, or know of something we’ve missed in one of our stories?
Please use the form below and let us know.

* Required
  • Please post the headline of the story or the title of the video with the error.

  • Please post exactly what was wrong with the story.

  • Please indicate your source for the correct information.

  • Yes


  • This will only be used to contact you if we have a question about your submission, it will not be used to identify you or be published.

  • Cancel

Having problems with the form?

Contact Us Directly
  • Print

You can comment on most stories on You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

You can comment on most stories on You can also agree or disagree with other comments. All you need to do is be a Winnipeg Free Press print or e-edition subscriber to join the conversation and give your feedback.

Have Your Say

New to commenting? Check out our Frequently Asked Questions.

Have Your Say

Comments are open to Winnipeg Free Press print or e-edition subscribers only. why?

Have Your Say

Comments are open to Winnipeg Free Press Subscribers only. why?

The Winnipeg Free Press does not necessarily endorse any of the views posted. By submitting your comment, you agree to our Terms and Conditions. These terms were revised effective April 16, 2010.


Make text: Larger | Smaller


Stephen Harper announces increased support for Canadian child protection agencies

View more like this

Photo Store Gallery

  • A one day old piglet glances up from his morning feeding at Cedar Lane Farm near Altona.    Standup photo Ruth Bonneville Winnipeg Free Press
  • PHIL.HOSSACK@FREEPRESS.MB.CA 100527-Winnipeg Free Press THe Provencher Foot Bridge is lit up

View More Gallery Photos


Did you watch the Bruce Jenner interview?

View Results

Ads by Google