AI models tested
5
Research · Synthetic Polling Methodology
Methodology research · July 2026
Every AI has a house lean, just like every pollster. So we stopped asking one. The panel average beat every single model.
Ask one AI model the same poll question a thousand times and you average away random noise — but never its built‑in worldview, the same way asking one pollster twice doesn’t fix that pollster’s lean. So we ran an experiment: five different AI models, from five different labs, each simulating the identical panel of 6,400 real registered voters, “voting” in eight past elections whose results are officially certified. Then we graded everyone against reality. The average of the panel beat every individual model — and when the models disagreed, that disagreement pointed straight at the one race every single one of them misread.
Simulated voters
6,400
Certified races graded
8
Panel average typical miss
±1 pt
The result
The average of the models out‑polled every model
Finding
On ordinary races, averaging the models roughly halved the error of any single model — a typical miss of about 1 point against certified results, versus 2.3 for the best individual model and 4.5 for the weakest. The same reason poll averages beat individual pollsters: each model’s lean points a different way, and the leans cancel.
Each model simulated the same 6,400 voters — real registrations drawn from the Pennsylvania, North Carolina, Florida, and Kansas voter files — answering the 2024 presidential race in all four states, the 2024 Senate races in Pennsylvania and Florida, the 2024 North Carolina governor’s race, and the 2020 Kansas Senate race. We graded every reading against the officially certified result, scored so that no model is ever graded on a race it was corrected against.
| Engine | Typical miss vs. certified results |
|---|---|
| The panel averageMean of the models, one reading per race | ±1 pt |
| GPT‑5 miniOpenAI | ±2.3 |
| Claude Sonnet 5Anthropic — our previous single engine | ±2.6 |
| Gemini 3.5 FlashGoogle | ±3.2 |
| GLM‑5.2Z.AI | ±3.5 |
| DeepSeek v4 FlashDeepSeek | ±4.5 |
Typical miss is measured the honest way: each engine’s reading of a race is corrected only using the other races, then compared to the certified result it never saw — and the table shows the typical size of those misses across the seven ordinary races. Differences under about a point are a tie. The eighth race is below, and it deserves its own section.
The stress test
One race broke every model — and the panel caught it
The 2024 North Carolina governor’s race was the hardest test we could find: Mark Robinson’s campaign collapsed under saturation‑level national scandal coverage, and Josh Stein won by 14.8 points. Every single model misread it — and each one misread it differently.
| Engine | Its reading of NC governor 2024 |
|---|---|
| Claude Sonnet 5 | Stein +68 |
| Gemini 3.5 Flash | Stein +45 |
| GLM‑5.2 | Stein +30 |
| DeepSeek v4 Flash | Stein +1 |
| GPT‑5 mini | Robinson +3 |
| Certified result | Stein +14.8 |
Why this matters
On the seven ordinary races, the five models landed within 4 to 11 points of each other. On the scandal race they spanned 71 points. A single AI can’t tell you when it’s wrong — it just hands you a confident number. A panel can: when the models scatter, the race is volatile, and no point estimate should be trusted — from us or anyone. That disagreement signal is now built into every Civly candidate poll as an automatic volatility flag.
Why you can trust it
Graded against elections that already happened — with the caveats stated
Certified answers only. All eight races have official, certified results — Pennsylvania’s 2024 Senate race, decided by 0.2 points, is in the set precisely because it’s the hardest kind of race to read. Every engine saw the identical voters, questions, and scoring, so the comparison is apples to apples.
No self‑grading. Any correction applied to an engine’s readings was fitted only on the other races, never the one being graded — the polling equivalent of not letting a student grade their own exam.
We checked for memorization. These are past races, and AI models have read the internet. So we asked each model, point blank, to recite each certified result. Four of the five could. That means these scores are a best case — and it makes two details more interesting, not less: the one model that could recite nothing (GPT‑5 mini) was the best single performer anyway, and recitation didn’t save anyone from the scandal race. The truly uncheatable exam is next: we’re registering predictions for real 2026 races, in writing, before election day, and we’ll publish the scorecard either way.
We tried to make it better with news, and it didn’t work — so it’s out. We also tested feeding the simulated voters real, date‑verified pre‑election headlines. It didn’t fix the scandal race, and the headlines’ own slant leaked into the results, so it’s not part of the method. We publish the experiments that fail, too — that’s what makes the ones that pass mean something.
What this changes
Every Civly candidate poll now runs on a panel of AIs
Starting this week, Civly synthetic candidate polls run each simulated voter through multiple AI models from different labs, report the panel average as the topline, and publish a volatility flag on any race where the models meaningfully disagree. You get a more accurate number on normal races, and an honest warning instead of a false number on the races that break polling.
Run it on your race
Validated before you ever see a number
The same method called 9 of 10 winners in the 2026 New York and Maryland primaries before a vote was cast — and now it runs on a panel of models instead of one. Any state, any matchup, any message.
Run dates July 16–17, 2026 · identical simulated panels of 1,600 registered voters per state, drawn from the Pennsylvania, North Carolina, Florida, and Kansas voter files · eight officially certified races graded leave‑one‑out with per‑state corrections · no names, addresses, or phone numbers are ever shown to any model · prepared by Civly.