The Math on Michigan - Considering the Effects of $64M in Campaign Spending
By Tavern Founder & CEO Matt Lackey
The polling for the Michigan Senate Democratic Primary, which Abdul El-Sayed won by less than one point on August 4, missed by a lot. Polling has been getting progressively harder and weirder for the last decade, so folks reflexively blame measurement. Despite that, I don’t think Michigan is another case of large scale measurement error, at least for us.
For the purpose of this analysis, I’m only going to talk about our polls at Tavern and will leave other researchers to discuss their own work. And I’ll note that though Tavern is a full scale research, ad making, and placement firm, we only did two independent polls in this race. Our polls were independent and we did not work for any of the campaigns or outside groups supporting them in this race.
At the beginning of July, we published a poll that had El-Sayed at 41% and Stevens at 38%. Then later that month, on July 26 and 27, we went back into the field and found El-Sayed at 48%, and Stevens at 37%. There’s uncertainty around both of these polls: sampling error is a huge deal and has gotten much harder since no one can construct a random sample anymore; no one votes for “undecided”, (we had 16% and 14% undecided respectively;) polls measure the present rather than forecast the future, and, most importantly, there was a lot of campaigning left to do after both polls.
That last point is the one I want to linger on. Our primary polling has been extremely accurate this cycle, especially for races where we publish the day before election day. So it’s worth asking if our instruments were precise and well-calibrated but reality moved.
According to AdImpact, supporters of Stevens spent just under $64M and El-Sayed supporters spent $7.3M. Political advertising has an impact on who voters support. This is especially true in primaries where voters aren’t anchoring on partisanship. And it's even true when a candidate successfully communicates around an opponents' advertising campaign. Instead of speculating on the effect of that advertising, we can look at past research to get a rough estimate of how many votes it moved.
Sides, Vavreck, and Warshaw—publicly engaged political scientists who are trusted as honest brokers by both political practitioners and academics— have an incredibly thorough paper from 2021 on the effects of television advertising. They limit their analysis to general elections, and it’s harder to move votes in a general election than in a primary, so their estimates should serve as a lower bound. This paper finds that, on average, moving a vote in the Senate with TV advertising costs ~$200.
So now let's do some math. If you project our poll from the end of July onto the electorate (multiplying the two-way by the number of voters,) you get this:
The actual margin between El-Sayed and Stevens was 15K votes as of writing this, putting us at a difference of 171K. We get this 171K difference by subtracting the observed margin (15k votes) from our projected margin (186k votes.) This difference tells us how many votes would have needed to change for our July 27th poll to be accurate on election day
Using our estimate from the linked paper, how much money would it take to persuade 171K voters? Roughly $34M according to the best data we have:
We are looking to change 171K voters opinions
Our cost per vote estimate is $200
$200 * 171K = $34M
The last question we have to ask is, “how much of that $64M spend occurred between our last poll and the election?” We didn’t work for either campaign, (we only polled this race to test and market our accuracy, despite the rumors) so we don’t have the week-over-week competitive ad spend to put a bow on this argument. But we do know campaigns spend most of their budget as close to the election as possible, so it’s more likely than not that there was at least a $15M+ spending advantage between our last survey and election day. The table below shows the expected change in margin by spending level:
The point of all this is not to say “Well Actually, we were right” but all the hand-wringing about a polling miss fails to account for the reality that there was a historic spending disparity. Political ads do, in fact, work. If they didn’t, candidates wouldn’t spend precious resources on them. And candidates wouldn’t talk about how they were getting outspent all the time.
The right way to capture public opinion when something out of the ordinary happens (like a scandal or historic spending) is to do another survey. Contrary to popular belief, polling isn’t that expensive if you automate the labor-intensive drudgery, which is how we can afford to poll all these races without a sponsor. If you’re in a situation like this, you can hire us to help with affordable and accurate opinion research.
And if you’re trying to win an election rather than just know the future, you should hire us to do your ads. Our ads consistently beat the best ad makers on both raw persuasion and holding voters' attention long enough to persuade them
