Forum poll highlights need for skepticism

Hey there, time traveller!
This article was published 28/11/2013 (4477 days ago), so information in it may no longer be current.

In the endless rumble over public opinion survey accuracy and elections, Lorne Bozinoff has become the latest punching bag.

Bozinoff is president of Forum Research, the Toronto polling company that failed spectacularly in its bid to predict the outcome of the federal byelection in Brandon-Souris.

In a survey last Sunday, Forum tracked Liberal Rolf Dinsdale with a 29-point lead over Tory Larry Maguire. When it was released to the media, the Brandon-Souris result seemed incredulous to just about everyone, including Liberals.

In the past 60 years, the Liberals have won that seat only once, and that win in 1993 required both a complete collapse by the Progressive Conservatives and a split of right-of-centre support with the Reform party.

On byelection day, the Liberal vote did surge, just as Forum had predicted, but so did the Tory vote. Maguire won, if only by a few hundred votes. The Forum result was so wrong Brandon-Souris is destined to be mentioned in the same breath as the worst election polling results of all time.

So, what happened?

Bozinoff said Forum is still studying the results to see where things went wrong. However, he said it’s pretty clear his firm — which employs Interactive Voice Response technology to gather its data — did not accurately identify those people who were actually going to vote.

This is a key issue for all pollsters. You may get 100 people to say they are going to vote for a particular candidate, but the data don’t mean much unless you can predict with reasonable accuracy whether they will show up to mark a ballot.

However, there are other factors Bozinoff believes were outside of Forum’s control. These include an incredible effort by the Tories to get out their vote Monday. That last-ditch effort to squeeze every Tory vote out of the riding, motivated perhaps by the poll’s huge Liberal lead, was not picked up in polling data, Bozinoff added.

Bozinoff is still defending his methodology despite claims from Brandon-Souris voters they were called as many as six times by Forum’s automated IVR system. Bozinoff said it was possible an individual could be called in each of the four surveys conducted during the campaign. However, he said it was “impossible” for someone to receive six calls, or more than one call for any individual survey.

“I can tell you that didn’t happen,” Bozinoff said of the multiple-call allegation. “Look, we were the only firm that actually polled on this. People can say whatever they want but we were the only ones out there doing it.”

While Bozinoff is searching for answers, his critics and competitors are sounding off. Darrell Bricker, CEO of Ipsos Public Affairs, was all over Twitter Wednesday assailing both Forum and the news organizations that published the outrageous results.

Bricker said there are best practices available to any polling company to avoid the problems Forum experienced. Failures of this magnitude cannot be explained by election-day ground game, electorate volatility or weather, Bricker said.

This was a badly constructed, badly executed poll, he added. The fact they got the other three races right does not erase the problems with Brandon-Souris. “(Forum) will tell you that they got most of the results right, but that isn’t the real test here. Getting it right every once in a while is not the test.”

Bricker said the truth could be revealed if Forum, like some other polling companies, published its raw and weighted data.

Bozinoff said he won’t do that.

Still, the pollster was not the only target of criticism in the wake of the byelection results. News organizations that published the its poll, including the Free Press, were soundly thrashed.

For the record, the Free Press did publish the last poll with the 29-point lead but it was in a story questioning the methodology and quoting sources suggesting the result was not believable.

Bricker said it is acceptable to publish the results of a rogue poll as long as the news organization takes the time to include the proper background and context so readers know the results are suspect. Unfortunately, he said, not enough news organizations are doing that.

We know Forum’s poll was flawed, but we still don’t know why. And while we can’t necessarily ensure bad polls won’t happen in future, those of us in the news business can ensure all polls get the same treatment: healthy, respectful skepticism.

dan.lett@freepress.mb.ca