GovBench

System result — the composed pipeline vs a direct model call

The board above scores individual models. This scores what actually ships: gate → retrieve → answer → verify, against the same items answered by the raw base model. n=193, paired, judged by an analysis written before the run. Intervals are cluster-robust: items inside a dimension share a rubric and a grader, so treating 193 items as 193 independent draws overstates precision. Measured design effect 1.92 — honest effective n is ≈100 of 193. Every row is computed from the same run and they partition it: 6 + 14 + 173 = 193.

layer	n	Δ	95% CI (clustered)
deterministic gate	6	−20.00	[−65.26, +25.26]
knowledge base	14	+19.64	[+9.24, +30.04]
tuned model	173	+6.50	[+1.06, +11.95]
whole system	193	+6.63	[+1.05, +12.21]

Wins 55 · losses 25 · ties 113 · sign test p=0.0011. Dropping the single largest item moves the headline to +6.15, so it does not rest on one case.

Retraction, 2026-07-29. The gate row previously read +34.84 and was the largest number we published. Re-measured on a clean, self-consistent run it fires 6 times, not 31, and contributes nothing: the base model already refuses all four plain-harm items it catches, and its only measurable effects are two false blocks — an analysis question about gambling-relapse targeting, and a prompt-injection item where resisting and still answering was correct. The earlier figure was measured on a gate that had overfitted to its own battery; fixing the overfitting removed the benefit. The previous table was also a splice — its rows summed to 186 beneath a 195-item total.

…and the control that refutes our own architecture claim

That tuned-model row (+6.50 now, +9.42 when the router control was run) changes two things at once: the query goes to a governance-tuned model at all, and to the particular one a per-dimension classifier chose. Holding the first fixed and varying only the second:

selection rule	score	vs routed
per-dimension routing	43.7%	—
always the best single model	42.8%	Δ +0.90   [-1.99, +3.79]   no effect
random expert (seeded)	34.5%	Δ +9.18   [+4.21, +14.14]

…and a second layer we built, measured, and switched off

The system answered "does Article 27 apply to a private credit-scoring deployer?" wrongly — from its weights. So we added retrieval over 404 real statute articles (AI Act, GDPR, NIS2, DORA, CRA, CSRD). It fixed that question. Then we measured it:

configuration	Δ vs weights	95% CI
naive top-k retrieval	-9.16	[-17.64, -0.69]   significant harm
with a relevance gate	-5.26	[-12.66, +2.13]   no effect shown

Asked "how should AI systems handle personal data?", BM25 returned GDPR Article 47 — binding corporate rules. Instructed to answer only from retrieved text, the model produced a confident answer about corporate rules and scored 0 where its own weights scored 50. The grounding instruction turns a retrieval miss into a wrong answer. The gate removed that harm but did not demonstrate benefit, so retrieval ships off too. The Article 27 fix stays one corrected item.

Per-dimension routing beats chance but does not beat one good model. The gain was the tuned model, not the routing — so routing ships off. The cause is on this page: routing selects on per-dimension differences, and 0 of 15 dimensions here have a resolved winner. It is selecting on noise.

…and a third: the quorum has 1.21 effective votes

The architecture called for a 3-leg Byzantine-fault-tolerant quorum. We measured the pairwise error correlation across all three legs on 174 items:

pair	phi
leg 1 ↔ leg 2	+0.730
leg 1 ↔ leg 3	+0.697
leg 2 ↔ leg 3	+0.803
Kish effective votes	1.21 of 3

The three legs are system prompts over one shared base, so they are wrong in the same places. Three nominal votes are worth 1.21 independent ones; the rest is latency. "Byzantine fault tolerant" has been removed from every document we publish. More legs or prompts cannot fix this — only a different architecture can.

What it costs to resolve a dimension — and why our first estimate was 10× wrong

We priced robustness at ~24 items per model from a gap observed at n=5, expanded it to 24, and re-measured. The gap narrowed from 14.3 to 8.4 points and the true price is ~230. At n=5 a single item is worth 20 points, so small-n gaps are inflated by the coarseness of the score space — expanding does not just add precision, it reveals the gap was smaller than it looked, and a smaller gap needs quadratically more items. Every price computed from n<20 on this page is a lower bound, not a target.

pip install inspect-ai
inspect eval govbench_inspect.py --model ollama/qwen2.5:0.5b