What Polling Data Fails to Reveal About American Opinion

In April 2021, political scientists Micah English and Joshua Kalla released a paper upon which liberal punditry pounced. Here was evidence, burnished with academic credentials, confirming a favored thesis among writers who make their living discussing campaigns despite never running them: Democrats are putting themselves in electoral peril if they talk about race. Then another scholar, Nathan P. Kalmoe, released results in July 2021 wherein race-forward messaging was more persuasive and mobilizing than color-blind alternatives. Two weeks later, English and Kalla put out a retest – under an altered design with opposition rebuttals – that failed to replicate their original results. Race-explicit and race-neutral messaging proved statistically indistinguishable in persuading voters to back progressive policies. (Full disclosure, although data gathering and analysis were done entirely by these scholars, I crafted race-forward messaging tested in the latter two studies.)

All three of these studies are the work of respected scholars, use randomized controlled trials (RCT), boast similar sample sizes within each experimental condition, and employ partisan cues. Yet, where English and Kalla’s first piece garnered extensive media coverage and social media chatter, both their retest and Kalmoe’s paper met with near silence.

As part of the aftermath of the 2020 election and in full speculative swing about what Democrats ought to do in 2022, we’ve seen a flurry of purportedly data-driven declarations that Democrats risk alienating essential swing voters if they take up the torch for racial justice. The “Do Popular Things” pundits often deliver these pronouncements with resignation. They wish that addressing the injustice of policing or supporting immigrant rights didn’t pave the way to Democratic demise; but we must face facts.

Yet, this tale of three studies demonstrates that humans of every pedigree are adept at privileging evidence to support our existing beliefs. And many liberal commentators and Democratic operatives don’t merely ignore or discount contrary evidence. They overestimate that the surveys they choose to heed can reliably predict what works in real-world campaigns for progressive policies and candidates.  

Public opinion is not fixed. And it’s not merely that individuals can shift views on political issues over time; it is precisely the function of political campaigns to move them. We may be able to use polling to determine where people are, but it cannot ever reveal the destination to which we seek to move them. This must come from the policies we wish to see passed, the materials changes we need to deliver, the world we seek to create. The job of a good message isn’t to say what is already popular; it is to make popular what we need said.

If organizers believe, for example, in significantly increasing the minimum wage, they “Fight for $15,” even though polling at the time this campaign launched showed voters marginally tolerating $12 per hour. Then this idea, once deemed improbable and even absurd, became a benchmark for ballot initiatives across the country. Because leaders refused to accept that where voters report they are at a given moment necessarily dictates where they’re capable of going nor indicates how to move them, $150 billion has now gone to 26 million working people.

The same can be said for civil rights, marriage equality, and “ban the box” initiatives, to name a few. And so too for rightwing efforts. When, for example, Republicans first introduced “personalizing” (read: dismantling) Social Security, it polled abysmally. So, the GOP introduced it anyway, again and again. They remain unbothered that this hasn’t proven a winning electoral argument. Because their task is to reshape public perception and now even progressive, high information voters believe that Social Security is “insolvent;” when, in reality, we need only lift the cap.

Granted, these examples occupy longer-term time horizons. But in electoral contests, what testing with a captive audience fails to consider is that if your words don’t spread, they don’t work. By definition, a message that your target audience doesn’t hear cannot persuade them. Messages spread person to person, especially coming from known sources, are far more effective than broadcast communication we script and pay to put out. You can turn a paragraph from a survey into a digital ad or political speech and contend with the fact that most people don’t watch those; but you can’t make the committed base popularize the words and ideas if they do not align with their beliefs.

The right has long understood this. They throw rhetorical red meat to keep their base activated as voters and engaged as messengers to convert the conflicted. They unleash the power of repetition that makes even the most extreme messages seem familiar, offering cognitive ease that has proven to be persuasive.

In short, while in-channel testing with a captive audience is a valuable tool we readily utilize, it tells us a limited part of the story. It can show which frames are popular at present but not which ones live up to our values and build public will for the world we want. It can indicate how people react to our messages when forced to pay attention but not what they’re capable of believing if their friends, neighbors, and family were to repeat them over and again. And it can tell us what potential voters think will make them take action, but not what will actually inspire our organizers to call, text, and drive people to do it.