Gallup Revamps Polling Apparatus

Yesterday, Gallup unveiled a report detailing the reasons why their polling efforts failed to predict an Obama victory in 2012.  The report offered four main reasons why their survey efforts did not predict Obama’s victory.

The final poll released by Gallup had Romney leading Obama by one point.  Frank Newport, Gallup’s editor in chief, explained there was an industry-wide problem among pollsters “Something was going on that affected the entire industry,” Newport said. “That’s what prompted our commitment here at Gallup.”

1. Likely voter screen: Gallup uses a screen to determine who is likely to cast a ballot. Establishing the parameters of a likely voter universe is a difficult enterprise.  It is about predicting the right mix of different types of voters.  This ranges from those who vote in every single election to those who are new registrants and will be voting for the first time and everyone in between those two extremes.

Turn out efforts by campaigns, along with public sentiment and enthusiasm effects these ratios. Aggressive turnout/registration efforts will get more voters to the polls who usually do not vote. Determining what percentage of these people ought to be included in a survey requires an expertise that goes beyond mere number crunching.  In the case of the Obama campaign, there was a very aggressive, savvy turnout effort.

Reaching these folks is another matter.  Modern technology allows people to screen their phone calls and avoid unwanted or unexpected contact.  Many people have eliminated their landlines entirely.  Reaching people by cell phone is also difficult (and more costly) because cell phone numbers are easily and quickly changed and databases of such information is not up to date.

“The report found the question seemingly most responsible for tilting their poll too far toward Romney was asking respondents how much thought they were giving to the election. Obama led by 3 percentage points among all voters, but that swung 4 points toward Romney among voters identified by Gallup as likely to cast ballots.”

This indicates that Obama was turning out people who were less politically engaged, which was smart campaigning for his situation. Enthusiasm for Obama was way down from the 2008 election.  We now know that Obama received 10 million less votes in 2012 than he did in 2008. Where else can you go to find more support? To voters who are less politically engaged or inclined.  It’s easier to get people to go out and vote for the first time based on a well-tuned predictive model than it is to convince those who are already disenchanted with your candidate. Obama’s method was to locate possible voters based on the characteristics shared by committed Obama voters. 

Not foreseeing this campaign strategy, compromised the ability of Gallup’s likely voter screen to include enough of folks on the lower end of the political engagement spectrum.

2. Residency of respondents:  Gallup found that they conducted too many interviews with the central and mountain time zones, resulting in less surveying of voters on the east and west coasts where Obama had strong support. A sample frame is created to mirror the latest Census figures on regional population and other factors.  If Obama had more support percentage wise on the coasts, that would not be reflected by simply sampling based on demographic information from the Census.

3. Race/Ethnicity:  Gallup reported that they were weighting their samples to a universe with more whites and fewer non-whites ending up with a “a disproportionate number of respondents reporting they were multiracial.”  They have adopted a new method allowing people to select up to five choices and then using four Census categories instead of two to identify race.

But race is a tricky issue for several reasons. It’s subjective in many ways, based on the respondent’s self-consideration. Some people who were born in the US to parents who are from Mexico, for instance, may not identify as Hispanic. We have a growing population of multi-racial people, how to decide which category these people fall into? Add to this, racial categories used differ from polling company to polling company.  If you read several different media polls, you are very likely going to see different percentages on ethnic demographic.

4. Landline sample frame: In 2012, Gallup used a listed sample, meaning they used a list of people identified with a specific number, causing a “older and more Republican” sample. They have since returned to RDD, or random digit dial to bridge this technology divide.

Gallup was also able to learn that some aspects of their operation did not contribute to 2012 inaccuracies. The name “Gallup” did not turn people away from picking up their phone to participate in a survey, known has the “house effect.” They also found that trying to increase the contact they made with those who did not answer their phone, did not make a difference.

Other issues affect the survey research business in general so it is not clear Gallup’s adjustments will help. More and more people are using cell phones as their primary phone lines and those are more costly for firms to contact and more costly to purchase a sample. Response rates are also getting lower, with Gallup saying they had only a 10% completion rate among those who they contacted.

But Gallup is optimistic, “When the next presidential election rolls around,” Newport said, “we think we’ll certainly be in a position at the accurate end of the spectrum.”