Part 1 of our Delphi series
Generative AI has changed the economics of making content. This is true in politics just as it is in commercial marketing.
Until recently, making an ad was expensive. A campaign might develop a handful of concepts, produce a few ads, and rely on strategists to decide which were good enough to put in front of voters. That constraint is disappearing.
At Tavern, we’ve built a machine that can generate a huge volume of persuasive content. But that creates a new problem: when you can make almost anything, how do you decide what is actually worth making?
You need another machine that can rapidly say: “This one is good. This one is bad. here’s what needs to change.” That is Delphi.
Delphi evaluates content and identifies which is the most persuasive. It scores what we make and tells our generation systems how to make it better.
Delphi only really makes sense in the context of generation at scale. If you’re making five ads, you can focus group or test them all with RCTs and just see what works. When you can make hundreds (or thousands) of ads, evaluation becomes the bottleneck. Delphi narrows the field before the experts decide what to develop, test, or place.
The important thing, then, isn’t either system by itself. It’s the negotiation between them: rapid generation paired with rapid evaluation, over and over again. We generate a piece of content. Delphi evaluates it. Delphi identifies what’s working and what isn’t. We generate again. That loop lets us explore vastly more creative possibilities while concentrating our time, research, and ultimately media dollars on the ideas most likely to work.
Delphi is built on experiments, not vibes
There is an obvious way to build an AI ad evaluator: ask a large language model whether it thinks an ad is persuasive. That isn’t what we wanted because it won’t actually work. Language models know what persuasive language looks like, but looking persuasive and actually persuading people are different things.
We knew Delphi had to be grounded in what happened when we actually put messages in front of people and measured their responses. Tavern has run tens of thousands of randomized trials that create a proprietary record of what people actually find persuasive, and that record is the foundation for Delphi.
But accumulating all that data created a surprisingly difficult problem. Historically, many of our message experiments used MaxDiff: respondents repeatedly chose which of two messages they found more persuasive. These experiments are very good at answering questions like “which of these messages is most effective at changing a specific opinion?” But each experiment lived on its own scale. A message that scored highly in one experiment couldn’t simply be compared with a message tested six months earlier in another.
Tavern had accumulated years of evidence without a consistent way to compare all of it.
Creating one measure of persuasion
Tavern’s historical record covers over 20,000 messages from 336 persuasion experiments. Some of those experiments shared messages and were already connected to one another. After accounting for those overlaps, the data formed 249 separate pools.
To connect those pools, Tavern selected three messages from each one: a low scorer, a middle scorer, and a high scorer. Those 747 anchor messages were placed into a new survey with more than 10,000 participants. The survey produced close to 60,000 usable head-to-head comparisons.
Each anchor now had two measurements: its original score and its score in the shared survey. That gave Tavern a way to estimate how each historical pool mapped onto the same scale.
We tested the bridge by hiding one anchor at a time and predicting its position from the other two anchors in its pool. On the model’s statistical scale, the typical prediction error was 0.49 logits. The survey’s own measurement error was 0.45.
We also checked whether different kinds of messages belong on the same scale at all — an issue message, a candidate message, and a Trump-focused message. Measuring within a category and measuring across categories produced nearly identical results, correlating about as closely as measurement noise allows. On random pairwise comparisons, the two approaches picked the same winner 77–79% of the time.
In plain English, the bridge missed by about as much as the underlying survey measurement did.
The result is something we didn’t have before: 20,429 experimentally tested political messages measured on a common scale of persuasiveness. That is part of the measurement foundation underneath Delphi.
The intelligence compounds: Every test makes the next test smarter
Political research is often designed to answer one question for one campaign at one moment. Once the decision is made, much of what the experiment found never reaches the next campaign.
Tavern’s experimental record is designed to accumulate. Each new test adds another observation about what people find persuasive and under what conditions. The shared scale allows those observations to inform one another instead of remaining isolated inside separate studies.
A campaign starting with Tavern today doesn’t start from zero. It inherits what we’ve learned from tens of thousands of previous trials, and tomorrow’s campaign inherits what we learn today. Its own experiments add current, campaign-specific evidence to that foundation.
Past results are a starting point rather than a guarantee. Politics changes, audiences differ, and every campaign still needs evidence from its own context. The advantage comes from beginning with an informed estimate and updating it as new results arrive.
How Tavern uses Delphi
A team can run a batch of draft messages through Delphi before committing research, production, or media dollars. Each message can be compared with the full historical corpus, a specific survey, or the other messages in the submitted batch.
Delphi identifies clear leaders and laggards while leaving close calls unresolved. Strategists can then focus their attention on the strongest options and use field experiments to make the final decision.
This becomes especially useful when generative systems produce hundreds of possible scripts or variations. The team does not need to spend equal time on every option. Delphi helps determine which ideas deserve a closer look and which can be set aside.
The world moves
The current version of Delphi has clear boundaries. Its message corpus is concentrated in recent Democratic political content, and its scores reflect an average respondent rather than a specific audience.
Time creates another problem: issue salience changes, voters learn new information, candidates become better known, an argument that works in the spring may land differently by the fall. The shared scale is anchored to measurements taken at particular moments. We don’t yet have evidence that a message measured at one point means precisely the same thing a year later.
Delphi does not yet model those changes directly. Tavern’s next work includes regular bridge surveys, audience-specific estimates, and time calibration.
The aim is to understand how persuasive performance changes across audiences and moments, rather than treating every message as if it existed in the same political environment.
Generative AI lowers the cost of producing another draft; research, production, and airtime remain expensive.
Delphi turns Tavern’s experimental record into a repeatable process for deciding where to spend those resources. Tavern can create more options, compare them with what we have learned, test the strongest candidates, and carry the result into the next round.
Delphi gives Tavern a faster, evidence-based way to choose which ideas deserve the next dollar. A strong message can still make a weak ad. Performance, editing, and emotional tone can reinforce an argument or pull it apart.
In Part 2, we’ll show how Delphi evaluates message, delivery, and tone to identify where an ad gains or loses strength before it goes to air.
Stick with us, because no one is doing it like we are.
Want to put Delphi’s intelligence to work for your campaign? Email us about finding the messages worth testing before you commit production and media dollars.
