The problem with online polling

Nate Cohn is out with a great article about the challenges of online polling that is well worth the read. He brought up some fun memories about the Xbox poll of 2012, and the rush by everyone test out Google Consumer surveys (and the resulting failures).

Cohn focuses on three main problems: declining quality of data on the panels and the decline of quality of the pollsters using them. Cohn recounts an unbelievable line that one of the panel survey ‘pollsters’ (SoCal Data) saying their Michigan survey only cost $475. The hard phone costs on a 600n statewide survey might easily be $15,000 so obviously if you could cut 97% of your costs and conduct an accurate survey people would do that.

It just so happens that SoCal used pollfish for their recruitment and collection, which is a platform I have actually tried to make work in about a dozen different ways for mixed mode collection. (Mixed mode is where you use a variety of methods from phones, text-to-web, online, etc to collect interviews.) The problem with almost any of these is that the people who respond to them are too young, too educated, too rich, and too white. That’s where mixed mode can help with then using more expensive methods such as phones to fill in the groups you miss. The problem largely is the inability to match back to a file. This is a much longer discussion but the holy grail of accurate and affordable research especially in small geographies is far away (it not getting further away.)

Total side note: one other issue with mixed mode (including the increasingly popular combination of text-to-web with live calls) is the way people answer questions asked to them verbally is different than reading it on a screen. On a phone survey you don’t read the option undecided but you record it if the voter volunteers it, that’s not possible on written surveys. This results in slight changes (largely increases in undecided voters).

The decline of polling in small geographies

Which is a great segue for an issue that is glaring right now which is even though budgets for state legislative races and congressional have exploded, the accessibility of surveys has fallen due to low response rates. In many state legislative races you literally have to call or text every single voter to get enough interviews, and in districts under 100,000 people (let alone voters!!) you can end up in a situation where you can’t even get a 400 respondent sample.

The only answer here is to raise response rates. One novel approach I saw this cycle was a Democratic firm offering $5 to respondents to fill the survey out. I’ve never tried it but would be fascinating to see if it worked.

Don’t mess with Nate Silver

For fun check out Nate Silver venting his righteous wrath on the clown Allan Lichtman who invented the fake “keys” analysis of who will win Presidential races.