Polls predicted a Yes, some very accurately, but were treated with caution
2018.05.28Following the historical result at the weekend’s 8th Amendment referendum, RED C have done our normal compilation of polls to evaluate how accurate they were in relation to the final result.
The first point to make, is that the argument that no one expected the result that came about, should probably be put to bed. Two polling companies had polls on the final weekend before the referendum, that actually predicted the result with extraordinary accuracy.
The most accurate poll was RED C’s final poll published in the Sunday Business Post the Sunday before the referendum. This poll was very accurate, with an error of just 1.1% vs. the final result. In fact, it was actually more accurate than both the Exit polls, which should be far more accurate as they don’t have the issues of undecided voters and are based on much larger sample sizes.
It should not have been a surprise either, given that the two previous polls published by RED C in March and April also showed a similar story, and in fact both were within the margin of error of the final result.
The second pollster with an accurate pre-referendum poll was Behaviour and Attitudes. Their poll was published in The Sunday Times, and had an error of 2%, which is also within the margin of error for a poll of this nature.
The rest of the polls conducted by other companies were not quite as accurate in their prediction. But the margin of error was no worse than had been seen at the last referendum, and all at least predicted a Yes result by at least 57% or more.
The problem was perhaps that many didn’t really believe that the polls from RED C and B&A could be accurate. “How could the result be even greater than for the Marriage Referendum?” was one comment.
To be honest, this is not really a surprise. Over recent years polling has become much more difficult task, as reaching a truly representative sample becomes more difficult. Polls also haven’t had a great track record in referendums or elections. In fact, an analysis of referendum polling prior to this weekend, has consistently showed that polls have over emphasised the Yes vote in nearly every referendum. At the Marriage Equality referendum, polls over predicted Yes support by as much as 7-10%.
As a result, much work has been done to try and understand this, and there have been papers written by people far more academic than I, about the Status Quo and Shy undecided voter. It is natural that, as an industry we look at the errors in previous elections and referenda and try to better understand why these have occurred. This is why many, including me, predicted that the race may well be tighter than our final poll suggested.
This feeling that the result would tighten, was further exacerbated by our Wisdom of Crowds approach. This was very successfully used at the Marriage Equality referendum and predicted the result with 100% accuracy. As such, deploying it again at this referendum made perfect sense, in order to try and help us to understand underlying shy or undecided voters. The Wisdom of Crowds predicted a 56% to 44% result, which also pointed to a possibly tighter race than we were seeing in our stated vote intention polls. However, it appears the Crowds were not so wise this time around!
At this stage it’s difficult to understand why, but perhaps the Crowd was simply reflecting the opinion among the population as a whole, that the race might be tighter than expected. It is also the case that having to provide equal airtime to both sides in a referendum, may well also have heightened the public’s views of a stronger No vote. No one watching the Claire Byrne debate would think that the No’s were only 34% of the population.
So why were some of the stated pre-referendum polls so accurate this time around? The first thing to take on board is that no election or referendum is exactly the same as the last. This makes accurate polling extremely difficult. Just ask all the UK pollsters that adjusted their techniques to take account of findings from the polling enquiry following the Brexit referendum, and ended up getting the last General Election result in the UK horribly wrong as a result of implementing those changes.
Having said that RED C have made a number of changes to our polls since the last general election, in order to try and provide a more accurate sample. These include changing the quota and weighting matrix we use, adding an education weight, spreading out fieldwork to reach all groups, and ensuring a minimum landline sample in our polls. Turnout could also have helped, as it appears in this referendum, that those saying in polls they would turn out, appear to have done so. Whereas often they say they will vote, but actually don’t manage it.
A word about the Citizens Assembly. If there is one thing that has come out of this referendum, it is the impact that the assembly had on the process, and how the assembly led the way by truly representing the population. The difficulty of recruiting members to the assembly that broadly represented the population was no easy task. RED C as the chosen recruitment company, acknowledged from the start that trying to get 100 people to be truly representative would be very difficult. In fact, the members of the Assembly voted to repeal in the 8th Amendment in very similar proportions to the electorate as a whole. I am proud that the procedures and processes we put in place, using tried and tested professional market research approaches in an effort to try and get a broadly representative group of members, has been now shown to have been so successful.
Finally, a shout out to the two companies that conducted Exit Polls. Getting the result right should be easier at an exit poll, as the sample is much greater and people have certainly made up their minds after the leave the polling booth. Having said that the task of completing 4000 interviews in one day and ensuing these are representative of all voters is not easy. The Irish Times exit poll conducted by Ipsos MRBI was produced astonishingly quickly, with great accuracy, a first for the industry. The RTE Exit poll, conducted by Behaviour and Attitudes, was released a little later, but the extra information included in the poll, was extremely useful to better understand voters and how they made their decisions.
All in all, not a bad reflection on the polling industry in Ireland. Roll on the next General Election!
