Trends and trickery: Read poll results carefully

Hey there, time traveller!
This article was published 21/09/2015 (3683 days ago), so information in it may no longer be current.

For better or worse, public opinion polling has always played an important role in Canadian elections. In the current federal campaign, it seems the worse is outweighing the better.

For proof, you need look no further than a decision by CTV News to publish a poll on voting intentions in the Quebec riding of Papineau, currently represented by Liberal Leader Justin Trudeau. In a Sept. 17 story, CTV reported Trudeau was “in trouble” in his home riding, running 11 points behind NDP candidate Anne Legace Dowson.

Tucked into the CTV story was a note that the poll, conducted by Quebec polling firm CROP, was commissioned by the NDP. That is a big red flag. So big, in fact, that it makes you wonder why CTV reported the results at all. Or, if they felt it was too good a story to pass up, why its story was not more circumspect about the accuracy of the poll.

However, that was not the only concern with the CROP poll. Once the results were made public, other polling companies reviewed the methodology – sample size, margin of error – and concluded that the sample was too small and the weighting was heavily skewed in favour of people who had voted NDP in the past.

A second poll was conducted in Papineau two days later, this one commissioned by a media outlet and featuring a sample twice as big as CROP. The results were not all that surprising: Trudeau was running ahead of his NDP challenger.

Could Trudeau lose his riding? It’s always a possibility, but there is nothing in the national polling trends that would suggest Trudeau was vulnerable in his home riding. That alone should have caused CTV to pass on the poll altogether, or at least warned readers the poll could be less than a fair picture of the riding.

Pollsters fighting amongst themselves

The CROP poll, and CTV’s inexcusable decision to publish it at face value, have kicked off the latest in a series of debates about the risk-reward equation for polling during an election campaign. The debate has become so heated that pollsters are fighting among themselves, with some firms accusing competitors of tarnishing the reputation of the entire industry.

There are really two issues that must be separated in this debate.

First is the issue of campaign trickery engineered by parties. Whatever you think about CROP and its intentions, it merely conducted work ordered by the NDP, and it was the party that decided it should be “leaked” to a news organization.

Trickery like this is hardly new. Parties have always tried to tempt political reporters with nuggets of inside party data in the hopes it would help forge the narrative of news reports and commentary. This data can include internal polling, voter identification results, even the number of lawn signs a particular candidate has in a particular riding.

Although we rarely report on the data itself, it can be tempting to use it as the context for making predictions on the outcome of individual ridings. As a rule of thumb, political reporters should never try to pick a front-runner or an underdog when profiling any riding or reporting on any incident during the campaign.

The trickery is not, however, limited to parties and their relationship with journalists.

For example, it is not unusual for a local campaign to perform a phone blitz where voters are told that there are new polling results showing that its candidate is pulling ahead and destined to win the election. This is another attempt to sway voter intentions with data that are either weak or fictional.

Value of polling misunderstood

The second important issue in this debate is a profound public misunderstanding about the true value of polling data.

The actual numbers revealed in any one poll are not as important as the trend. If a party has been steadily increasing its vote over the past few weeks, that is way more important as a data point than the actual percentage of support in any one poll. Momentum is key in any election campaign and the party that can maintain it usually triumphs.

Winnipeg Free Press | Newsletter

Dan Lett | Not for Attribution

Tuesdays

A weekly look at politics close to home and around the world.

Sign up for Dan Lett | Not for Attribution

Sign Up

I agree to the Terms and Conditions, Cookie and Privacy Policies, and CASL agreement.

The other important point is that the firms and commentators that aggregate polling results tend to produce results that are far more valuable than those doing single polls. Nanos Research is the leader in this regard, with surveys based on a rolling three-day survey of 1,200 respondents. Each day, data from the previous three days is used to tweak the numbers, giving Nanos a more accurate and responsive baseline of information.

Eric Grenier of threehundredeight.com reports on the trends revealed in the cumulative results of all polls released in a previous week. Again, Grenier’s work helps experts and lay people alike see the trend lines, which often reveals the real narrative of the election.

The CROP incident is not an indictment against an entire industry. It certainly is unflattering for the NDP and for CTV in particular, but it should not undermine the value of properly conducted, properly reported polling.

As this campaign is revealing, that is easier said than done.

dan.lett@freepress.mb.ca

Dan Lett
Columnist

Dan Lett is a columnist for the Free Press, providing opinion and commentary on politics in Winnipeg and beyond. Born and raised in Toronto, Dan joined the Free Press in 1986.  Read more about Dan.

Dan’s columns are built on facts and reactions, but offer his personal views through arguments and analysis. The Free Press’ editing team reviews Dan’s columns before they are posted online or published in print — part of the our tradition, since 1872, of producing reliable independent journalism. Read more about Free Press’s history and mandate, and learn how our newsroom operates.