The wobbly ‘science’ of predicting riding outcomes

Hey there, time traveller!
This article was published 17/10/2015 (3671 days ago), so information in it may no longer be current.

Someone once quipped that if you give a thousand monkeys a thousand typewriters, eventually they will produce Shakespeare.

By that logic, if you give the public a thousand (or even a hundred) polls and put them all together, will it produce a perfectly predictive snapshot of public opinion?

Public opinion polls have proliferated in recent years in Canada. In the 2015 federal election, we seem to have reached “peak polling.” Not only are more “horse race” public opinion surveys being done than before, but there is a proliferation of people aggregating the results of these soundings and using them to predict, right down to the riding level, probable voting outcomes in Monday’s election.

In the glory days of polling, pollsters were treated as omniscient soothsayers. Now those who aggregate polls and project seat outcomes are being placed on the same pedestal by a credulous public and news media, many of whom lack a thorough understanding of the limitations of their methodology.

The theory behind poll aggregation is sound. A single poll is subject to a statistical margin of error and may have both obvious and latent biases related to how the questions were worded, the methodology that was used to conduct the survey, etc. Aggregating multiple polls, in theory, is supposed to smooth out the distortions of a single poll and create an average that is more indicative of reality. Nate Silver used this method in the United States to great effect to accurately predict the outcome of the 2012 presidential election. He weighted polls by the research firm’s previous track record and used a formula that mitigated the “house effects” — that is, the biases of each polling firm that emerged over time.

The best-known poll aggregator in Canada today is âric Grenier, who founded the website www.threehundredeight.com (there were 308 seats in the House of Commons prior to this election; Silver’s site is fivethirtyeight.com — the number of electoral college votes in the U.S.). Grenier also runs the CBC’s poll tracker in this election.

The Globe and Mail has an election-forecasting model run by political scientist Paul Fairie, which uses a Monte Carlo simulation — a statistical method that uses polling data and random number generators to predict the probability of certain election outcomes. The Laurier Institute for the Study of Public Opinion and Policy at Wilfrid Laurier University has also used public opinion surveys to predict how many seats each party will win in recent elections.

All of these aggregators use multiple polls to project the likely number of seats each party will win. The Laurier institute and the Globe and Mail predict the number of seats each party will win in each region without picking which specific seats will be won and lost by different parties. Grenier, meanwhile, provides the likelihood of specific seats being won by specific parties.

Although this is interesting to those who want to know how a specific riding will go on election day, it is also problematic because there is a very shaky basis for claiming these seats will be won or lost by particular parties.

Seat projections use what is known as a “regional swing” model to predict the likelihood of a certain party winning a seat that is based both on current regional polling and the results of the last election. To use a simple example, let’s say Party X and Party Y each had 50 per cent of the vote in the last election in a certain region. In a specific riding in this region, Party X received 60 per cent of the vote and Party Y received 40 per cent. Let’s say a poll comes out showing Party Y leads Party X in the region by a 60-40 ratio. By that logic, Party X is down 10 points compared with the last election and Party Y is up 10 points. Based on that, the swing model would extrapolate that Party X and Party Y are tied at 50 per cent each in the specific riding.

The regional swing model was first developed in the UK, where historically it worked well because there were really only two main parties (Labour and Conservative) contesting the vast majority of seats in broadly homogenous regions. When more parties are introduced to the mix — as happened in the UK’s last general election, with the UK Independence Party and the Scottish Independence Party surging in support — matters become more complicated.

Canada’s regional diversity and the abundance of three- and four-party contests make the regional swing model even more complex and problematic. Quebec, for example, includes three diverse regions. In metropolitan Montreal, it’s primarily a battle between the Liberals and the NDP, with the Bloc Québécois also strongly in the mix in some predominantly francophone ridings. Quebec City and the communities between the St. Lawrence River and the U.S. border are three-way battles between the Bloc, Conservatives and NDP, with the Liberals basically a non-factor. The rest of francophone, rural Quebec is mostly a straight NDP-Bloc fight. How do a few surveys of 1,200 Canadians and only 400 Quebecers capture this diversity, particularly when the margin of error is so high?

This is an even greater problem in areas with smaller populations, such as Manitoba and Saskatchewan, which are usually lumped together in national surveys. National surveys that are used to create regional swing models for individual seats are often based on samples of less than 100 voters (with a margin of error of more than 10 per cent) for these two Prairie provinces.

These aggregations might yield stronger seat projections if they included larger regional samples of 800 to 1,000 respondents, such as those produced by Probe Research in Manitoba and by other regional firms in Saskatchewan, B.C., Quebec and Atlantic Canada. But even then, provincewide polls of that size do not allow us to capture what is happening right down at the riding level.

A country with such diversity and with so many parties and ridings makes the model more difficult. Part of the reason Nate Silver was so successful in predicting the outcome of the U.S. presidential election was because the dynamics were much more favourable to his methods. He only had to worry about two parties, not three or four or five. And because the presidential college is a winner-take-all system for each of the 50 states, he had many, many more polls with larger sample sizes (600 or more respondents) for individual states to build his model upon.

This is by no means meant to denigrate the work of Canadian analysts and scholars of public opinion. However, it is important for consumers of this information — the media and citizens — to understand the limitations of this methodology and to refrain from trying to use their prognostications to predict, with absolute certainty, the specific outcomes of local races.

This is particularly important when there is a clamour like never before to vote strategically. If that choice is based on misinterpreted or false information, it will not only lead to undesirable outcomes, but it also weakens the value of your vote.

Curtis Brown is the vice-president of Probe Research, a Winnipeg-based, market research firm. His views are his own.

curtis@probe-research.com Twitter: @curtisatprobe