July 15, 2020

22° C, Partly cloudy

Full Forecast

Close this


Advertise With Us


Trends and trickery: Read poll results carefully

Liberal Leader Justin Trudeau walks to the campaign plane at the airport in Toronto, Monday September 21, 2015.


Liberal Leader Justin Trudeau walks to the campaign plane at the airport in Toronto, Monday September 21, 2015.

Hey there, time traveller!
This article was published 21/9/2015 (1758 days ago), so information in it may no longer be current.

For better or worse, public opinion polling has always played an important role in Canadian elections. In the current federal campaign, it seems the worse is outweighing the better.

For proof, you need look no further than a decision by CTV News to publish a poll on voting intentions in the Quebec riding of Papineau, currently represented by Liberal Leader Justin Trudeau. In a Sept. 17 story, CTV reported Trudeau was "in trouble" in his home riding, running 11 points behind NDP candidate Anne Legace Dowson.

Tucked into the CTV story was a note that the poll, conducted by Quebec polling firm CROP, was commissioned by the NDP. That is a big red flag. So big, in fact, that it makes you wonder why CTV reported the results at all. Or, if they felt it was too good a story to pass up, why its story was not more circumspect about the accuracy of the poll.

However, that was not the only concern with the CROP poll. Once the results were made public, other polling companies reviewed the methodology – sample size, margin of error – and concluded that the sample was too small and the weighting was heavily skewed in favour of people who had voted NDP in the past.

A second poll was conducted in Papineau two days later, this one commissioned by a media outlet and featuring a sample twice as big as CROP. The results were not all that surprising: Trudeau was running ahead of his NDP challenger.

Could Trudeau lose his riding? It’s always a possibility, but there is nothing in the national polling trends that would suggest Trudeau was vulnerable in his home riding. That alone should have caused CTV to pass on the poll altogether, or at least warned readers the poll could be less than a fair picture of the riding.

Pollsters fighting amongst themselves

The CROP poll, and CTV’s inexcusable decision to publish it at face value, have kicked off the latest in a series of debates about the risk-reward equation for polling during an election campaign. The debate has become so heated that pollsters are fighting among themselves, with some firms accusing competitors of tarnishing the reputation of the entire industry.

There are really two issues that must be separated in this debate.

First is the issue of campaign trickery engineered by parties. Whatever you think about CROP and its intentions, it merely conducted work ordered by the NDP, and it was the party that decided it should be "leaked" to a news organization.

Trickery like this is hardly new. Parties have always tried to tempt political reporters with nuggets of inside party data in the hopes it would help forge the narrative of news reports and commentary. This data can include internal polling, voter identification results, even the number of lawn signs a particular candidate has in a particular riding.

Although we rarely report on the data itself, it can be tempting to use it as the context for making predictions on the outcome of individual ridings. As a rule of thumb, political reporters should never try to pick a front-runner or an underdog when profiling any riding or reporting on any incident during the campaign.

The trickery is not, however, limited to parties and their relationship with journalists.

For example, it is not unusual for a local campaign to perform a phone blitz where voters are told that there are new polling results showing that its candidate is pulling ahead and destined to win the election. This is another attempt to sway voter intentions with data that are either weak or fictional.

Value of polling misunderstood

The second important issue in this debate is a profound public misunderstanding about the true value of polling data.

The actual numbers revealed in any one poll are not as important as the trend. If a party has been steadily increasing its vote over the past few weeks, that is way more important as a data point than the actual percentage of support in any one poll. Momentum is key in any election campaign and the party that can maintain it usually triumphs.

The other important point is that the firms and commentators that aggregate polling results tend to produce results that are far more valuable than those doing single polls. Nanos Research is the leader in this regard, with surveys based on a rolling three-day survey of 1,200 respondents. Each day, data from the previous three days is used to tweak the numbers, giving Nanos a more accurate and responsive baseline of information.

Eric Grenier of threehundredeight.com reports on the trends revealed in the cumulative results of all polls released in a previous week. Again, Grenier’s work helps experts and lay people alike see the trend lines, which often reveals the real narrative of the election.

The CROP incident is not an indictment against an entire industry. It certainly is unflattering for the NDP and for CTV in particular, but it should not undermine the value of properly conducted, properly reported polling.

As this campaign is revealing, that is easier said than done.


Dan Lett

Dan Lett

Born and raised in and around Toronto, Dan Lett came to Winnipeg in 1986, less than a year out of journalism school with a lifelong dream to be a newspaper reporter.

Read full biography


Advertise With Us

The Winnipeg Free Press invites you to share your opinion on this story in a letter to the editor. A selection of letters to the editor are published daily.

To submit a letter:
• fill out the form on this page, or
• email letters@freepress.mb.ca, or
• mail Letters to the Editor, 1355 Mountain Avenue, Winnipeg, Manitoba, R2X 3B6.

Letters must include the writer’s full name, address, and a daytime phone number. Letters are edited for length and clarity.


Advertise With Us