Skip to content

Measurements

Every figure here comes from real runs of examples/support-qa on one machine. The run files are committed in examples/support-qa/runs/, and uv run python examples/support-qa/measure.py recomputes the calibration, comparison and latency tables from them without a model. Sections 1–9 use small local models; section 10 repeats part of the study on three hosted frontier models, with its own run files and script in examples/frontier-study.

Setup

Machine Apple M5, 24 GB RAM, macOS 27.0.1 (arm64)
Ollama 0.23.2
System under test qwen3:1.7b, digest 8f68893c685c — temperature 0, seed 0, think: false, max 200 tokens
Judge qwen3:8b, digest 500a1f067a9f — temperature 0, seed 0, think: false, max 256 tokens
Reasoning judge (section 8) the same model with think: true and max 4,096 tokens, in suite.reasoning-judge.yaml
evalproof / Python 0.1.0 / 3.13.15
Golden set 83 questions about a product; 9 are not answerable from the facts. Every run in runs/ used cases.v1.jsonl, dataset hash c6f08ac201023363; cases.jsonl is the revised set from section 8 (55ba005be14c0f20), whose runs are in runs/v2/
"Gold" a per-question regex answer key, written before any model output was seen and not changed after. It stands in for human review; it is not human labelling.
Reviewed labels (section 9) pass/fail for the 30 baseline cases of the seed-0 random sample, read against the rubric and the product facts, in human-review/
Change under test grounded system prompt → "friendly, always give a complete answer" prompt; same model

1. The two runs

Baseline (grounded prompt) Candidate (friendly prompt)
Answer-key pass rate 81.9% [72.3, 88.7] 75.9% [65.7, 83.8]
Judge pass rate 77.1% [67.0, 84.8] 72.3% [61.8, 80.8]
System latency p50 / p95 624 / 1,469 ms 2,243 / 4,422 ms
Mean output length 19 tokens 77 tokens
Run wall time (system + judge, sequential) 363 s 588 s
Re-run with a warm cache 0.05 s, results identical to the cold run —

Intervals are 95% Wilson. Wall times are for 83 cases with one system call and one judge call each, no concurrency.

2. How good is the judge? Measured against the full answer key

Baseline Candidate
Judge pass · key pass (TP) 63 58
Judge pass · key fail (FP) 1 2
Judge fail · key pass (FN) 5 5
Judge fail · key fail (TN) 14 18
TPR / FPR 92.6% / 6.7% 92.1% / 10.0%
Cohen's κ 0.779 0.781

3. Correcting the judge's rate from a 30-case gold sample

A team will not hand-label all 83 cases every release. The suite calibrates against a random 30. Over 200 different random samples of 30 (seeds 0–199), against the answer key on all 83 as the truth:

Baseline (truth 81.9%) Mean point Mean |error| 95% CI covers truth Mean CI width
Raw judge rate (all 83) 77.1% 4.8 pts 100% 17.8 pts
Gold sample alone (30) 82.8% 4.1 pts 100% 26.2 pts
ACC (judge on 83, error rates from 30) 81.5% 3.8 pts 100% 34.1 pts
PPI (judge on 53, residuals from 30) 81.2% 4.7 pts 98.5% 28.6 pts
Candidate (truth 75.9%) Mean point Mean |error| 95% CI covers truth Mean CI width
Raw judge rate 72.3% 3.6 pts 100% 18.9 pts
Gold sample alone 76.1% 4.3 pts 99.0% 29.1 pts
ACC 75.3% 4.0 pts 100% 41.0 pts
PPI 75.4% 4.7 pts 98.5% 31.6 pts

What this shows. The raw judge rate sits 3.6–4.8 points below the key, and both corrections remove that offset on average. At this size it buys little: the offset is smaller than the raw rate's own interval, which covered the truth every time, while the corrected intervals are about twice as wide. Calibration earns its keep when the suite is large enough that the raw interval is narrow and the bias dominates — the regime the prevalence-shift simulation shows, where the raw interval covered the truth 0% of the time at 5,000 reports per period — or when the judge is worse than this one.

4. The gold was wrong more often than the judge

The judge and the answer key disagreed on 13 cases across the two runs. Each was read by hand:

Case Run Key Judge Who was right Why
q76 lifetime licence both pass fail judge the answer invents "no lifetime licence"; the key's abstention regex matched "not … provide"
q78 HIPAA both pass fail judge the answer asserts "not HIPAA compliant"; the key matched "does not mention"
q77 uptime SLA baseline pass fail judge the answer invents 99.9% and 99.99% SLAs; the key matched "no SLA … specified"
q28 which plans have trials candidate pass fail judge the answer adds a trial to the Free plan; the key only checked that Plus and Team appear
q31 cancelling monthly candidate pass fail judge the answer says service stops immediately; the key matched "until the end" in a later bullet
q56 two-factor methods baseline pass fail key correct answer plus a true extra fact (no SMS); the judge penalised the extra fact
q65 integrations baseline pass fail key correct answer plus a true extra fact; same failure
q32 payment methods candidate pass fail key correct answer plus a true extra fact; same failure
q05 Enterprise pricing candidate fail pass key the answer quotes Team's prices for Enterprise; the judge saw "custom pricing" in a bullet and passed it
q66 Microsoft Teams both fail pass ambiguous "Yes, Quillfox offers an integration…" then "Microsoft Teams is not supported"

The key was wrong 7 times, the judge 4 times, 2 are ambiguous. The key's errors are structural: a regex can tell that the right fact is present but not that a wrong one is absent, and it cannot tell an abstention from an invention phrased as one. Correcting the key on these cases puts the baseline at 78.3% and the candidate at 71.1% — within about a point of the raw judge. Most of the "bias" that section 3 corrected was the gold's, not the judge's. Calibration is only as good as its labels, and the labels need the same scrutiny as the judge. The judge's own failure is consistent too: three of its four errors penalise correct extra detail the rubric explicitly allows.

5. Did the prompt change regress?

evalproof compare … --gate on the 83 paired cases:

Metric Baseline Candidate Δ (95% CI) Test p
answer_key 81.9% 75.9% −6.0 pts [−14.5, +1.2] exact McNemar 0.227
judge (gated) 77.1% 72.3% −4.8 pts [−13.3, +3.6] exact McNemar 0.388

The judge saw 8 cases get worse and 4 get better. Gate: pass with a warning (exit 0) — the drop is beyond the 3-point tolerance but p = 0.39 at n = 83, so the data cannot tell it from noise. A delta-only gate would have failed the change on the same numbers.

The regression that is measurable is latency: p95 went from 1,469 ms to 4,422 ms. With --max-p95-ms 2500 the same command fails with exit 3 (budget breach) and still prints the quality warning.

6. Pairwise judging: position and length bias

evalproof pairwise asked the judge to pick the better answer for each of the 83 pairs, in both orders (166 calls, 776 s):

  • 35 of 83 verdicts (42.2%) flipped when the order was swapped. In 32 of those the answer shown first won both times. evalproof reports them as ties.
  • Of the 48 consistent verdicts, the judge preferred the candidate 42 times and the baseline 6 — although the pointwise judge and the answer key both rate the candidate lower. In 31 of the 42, both answers were correct by the key; on those cases the candidate's answers averaged 383 characters against the baseline's 78. In 9, the candidate's answer was wrong by the key.

An 8B pairwise judge without the order swap would have reported a decisive win for the worse prompt. Position bias is cancelled by the swap; length bias is not, and is why evalproof gates on pointwise verdicts and reports pairwise preference as a diagnostic.

7. Determinism and the cache

  • 20 cases re-run with the cache disabled: 20/20 identical system outputs, 20/20 identical verdicts, 20/20 identical judge reasons (temperature 0, seed 0, Ollama 0.23.2).
  • Re-running the full baseline with a warm cache took 0.05 s instead of 363 s and produced results identical to the cold run, apart from the per-case cached flag.
  • Three days later, the full baseline judged again with the judge's cache entries removed (system answers still cached) gave 83/83 identical verdicts in 201 s (baseline.think-off-rerun.json).

8. A reasoning judge

The same qwen3:8b judge with thinking switched on (issue #3), on the same cached answers, so the only change is the judge. measure.py prints every table below from the committed runs.

Thinking off Thinking on
Judge time, 83 baseline verdicts (answers cached) 201 s (2.4 s each) 1,338 s (16.1 s each)
Judge time, 83 candidate verdicts — 1,188 s
Judge tokens per verdict, baseline / candidate 306 / 369 590 / 607
Undecided verdicts 0 0
Verdicts that changed, baseline / candidate — 7 / 6

Against the answer key, thinking looks worse: κ falls from 0.779 to 0.676 on the baseline and from 0.781 to 0.727 on the candidate. That comparison is unfair, because section 4 showed the key is often the one that is wrong. So every case where the key and the two judges did not all agree (22, the 13 from section 4 and 9 that only the thinking judge disputes) was read case by case against the rubric and the product facts. The 9 new ones:

Case Run Key Off On Right Why
q28 which plans have trials baseline pass pass fail on the answer gives the Free plan a trial, the same invention section 4 found in the candidate
q07 annual saving on Plus candidate pass pass fail on the answer states "$60 per year" before working its way to $12
q42 Linux desktop app baseline fail fail pass on "Yes, there is no Linux desktop app; the web app works on Linux": the facts are right; the key rejects any answer that starts with "Yes"
q42 Linux desktop app candidate fail fail pass on the same facts; the off judge's reason contradicts itself
q83 Plus in euros baseline pass pass fail off the answer gives the dollar price and says the euro price is not covered; correct extra detail, not an invented answer
q21 SSO, q22 admin console both pass pass fail off the answer names Team. The reference adds Enterprise, which the product facts do not say; the thinking judge followed the reference

Against those audited labels (the key with every reading applied, q66 left out as ambiguous):

Thinking off Thinking on
Baseline TPR / FPR 95.4% / 5.9% 95.4% / 0.0%
Baseline Cohen's κ 0.858 0.895
Candidate TPR / FPR 96.6% / 8.7% 96.6% / 0.0%
Candidate Cohen's κ 0.879 0.941
Right on the 20 decided contested cases 12 15 (the answer key: 9)

What this shows. Thinking made the judge better where it matters most for a gate: it passed no wrong answer in either run, against three for the judge without it, and it fixed all three cases where that judge penalised correct extra detail. It costs 6.7 times the judge time and about twice the tokens. It also reads the reference more literally, which cuts both ways: it caught an answer that states a wrong number before the right one, and it failed two correct answers because the golden set's reference claims more than the product facts do. That is a defect in two reference answers, now visible because a stricter judge tripped on it.

The gate's decision did not change. With thinking on, the judge saw 6 cases get worse and 1 get better (−6.0 points, 95% CI [−12.0, +0.0], exact McNemar p = 0.125) instead of 8 and 4 (p = 0.388): cleaner evidence of the same small regression, still not distinguishable from noise at n = 83, so the gate still passes with a warning.

After correcting the two references (issue #7)

cases.jsonl now says Team only for q21 and q22, as the product facts do; cases.v1.jsonl keeps the original so the runs above stay reproducible. All four runs were judged again on the revised set (runs/v2/): every answer came from the cache, and the only verdicts that moved were the thinking judge's on q21 and q22, now passes.

Against the audited labels Thinking off Thinking on
Baseline TPR / FPR, κ 95.4% / 5.9%, 0.858 98.5% / 0.0%, 0.964
Candidate TPR / FPR, κ 96.6% / 8.7%, 0.879 100.0% / 0.0%, 1.000
Right on the 16 decided contested cases 8 15 (the answer key: 5)

With the references fixed, the thinking judge's one remaining miss is q83, where it read a correct dollar price as an invented answer to a question about euros. The gate's decision is the same as before: 6 worse, 1 better, p = 0.125, pass with a warning.

The readings in sections 4 and 8 are careful case-by-case reads of the contested cases only. Section 9 reads a random sample instead.

9. Calibrating against reviewed labels (issue #4)

Sections 3 and 4 calibrated against the answer key and then found the key wrong. Here the key is replaced by labels that were read. The 30 baseline cases are the seed-0 random sample that calibrate --gold-sample 30 would draw, so PPI's assumption of a random gold sample holds. Each answer was marked pass or fail against rubric.md and the product facts, in review-sheet.csv (the verdicts are also in labels.jsonl for calibrate --labels). The verdicts were drafted with a model (Claude) and checked and accepted by the maintainer: one reviewer's check, not an independent panel. q22 was read against its corrected reference.

The labels give 22 passes and 8 failures. Against them:

  • the answer key is wrong on one case of the 30: q28, where the answer gives the Free plan a trial and the key only checks that Plus and Team appear (the failure section 4 found in the candidate);
  • the audited labels of sections 4 and 8 agree on all 29 cases they decide. The 30th, q66, which they left ambiguous, is a fail here: asked whether there is a Microsoft Teams integration, the answer opens "Yes" and then says there is none, and the rubric fails an answer that contradicts itself. Those audits were also read with Claude, so this agreement shows a consistent reading, not an independent confirmation.
Judge · gold (30 cases) TPR / FPR κ Gold only ACC PPI
thinking off · answer key 95.7% / 14.3% 0.814 76.7% [59.1, 88.2] 77.2% [55.8, 92.7] 77.4% [62.6, 92.1]
thinking off · reviewed 95.5% / 25.0% 0.734 73.3% [55.6, 85.8] 74.0% [42.2, 92.4] 74.0% [57.9, 90.2]
thinking on · answer key 95.7% / 14.3% 0.814 76.7% [59.1, 88.2] 78.7% [57.6, 94.5] 79.2% [64.8, 93.7]
thinking on · reviewed 100.0% / 12.5% 0.911 73.3% [55.6, 85.8] 75.2% [57.8, 85.5] 75.9% [63.1, 88.7]

The raw judge rate on all 83 is 77.1% with thinking off and 78.3% with it on (the runs/v2/ verdicts). The best truth available for all 83, the audited labels with q66 failed, is 78.3%.

What this shows.

  • One relabelled case moved the corrected rate about 3 points down for the published judge (ACC 77.2% → 74.0%, PPI 77.4% → 74.0%). With 8 failures in the sample, one more wrong answer the judge passed is 11 points of FPR, 14% → 25%. Error rates estimated on 30 labels are that fragile.
  • The move went away from the best truth, not towards it: 74.0% against 78.3%, where the raw judge rate was 1.2 points off. The sample holds 8 failures in 30 (26.7%) against 18 in 83 (21.7%) for the full set, and the correction inherits that luck. Every interval still covers 78.3%; the ACC interval is 50 points wide.
  • Better labels separate the two judges; the key could not. Against the key on these 30 cases the judges tie at κ 0.814. Against the reviewed labels the thinking judge reaches 0.911 and the judge without thinking falls to 0.734: the thinking judge failed q28, which the key passed, and its one miss is q66.
  • On a random sample the key was wrong once in 30 (Wilson 95% CI 0.6%–16.7%), not the 7 of 13 section 4 found. Section 4 read only the cases where the key and the judge disagreed, and the key's errors cluster in the questions the facts do not answer (q76–q78), none of which fell in this sample. "7 of 13" describes the disagreements, not the key's overall error rate.

Section 3's conclusion stands and gets sharper: at this size a 30-case gold sample decides little about the pass rate, whoever labels it. What it does show is the confusion matrix, which is where the reviewed labels earned their keep: they found the judge passing an answer that contradicts itself (q66) and an invented trial (q28), and that the reasoning judge catches the second.

10. Frontier models on a sample

Sections 1–9 measure small local models. This section asks whether the same checks hold on hosted frontier models, at a size meant to show the effect rather than to rank models: every second case of the revised support set (42 of 83) and of the ticket-router set (30 of 60), 558 calls in all, no judge. uv run python examples/frontier-study/measure.py recomputes every figure below from the committed run files and re-scores every stored answer against the committed answer keys; the study's README lists the files and every change made to the key.

Models Anthropic claude-sonnet-5-5; OpenAI gpt-6-astra, reasoning effort medium; Google gemini-4-argon-medium
Date 2026-11-15, evalproof 0.8.0; run files rebuilt from the cache with evalproof 1.0.0, no model called
Sampling the provider default; temperature and seed cannot be set
Support Q&A the grounded prompt twice (84 answers per model), the friendly prompt once (42)
Ticket router one-word queue label with prompt.router.md, twice (60 answers per model)
Scoring the regex answer key as written (answer_key), the same key after review (answer_key_reviewed, the gate metric), and exact label match for the router
Model Grounded, key as written Grounded, reviewed key Friendly, key as written Friendly, reviewed key Router Latency p50 (support / router)
claude-sonnet-5-5 83/84 84/84 39/42 42/42 58/60 2.0 / 1.8 s
gpt-6-astra 82/84 84/84 39/42 42/42 58/60 6.2 / 6.0 s
gemini-4-argon-medium 84/84 84/84 39/42 42/42 58/60 12.4 / 12.0 s

What this shows.

  • Every failure the answer key reported was the key's error. It failed 12 of the 378 support answers, and a reading of each against the product facts found all 12 correct: five refusals written with a typographic apostrophe ("I don’t have information…") that the key's ' did not match, and seven answers that added a true fact about another plan, which a reject pattern took for the wrong answer (q05, q37, q63). The reviewed key accepts ’ and narrows those three patterns to the wrong claim itself; wrong answers written for the purpose ("Enterprise pricing is custom. It typically works out to about $25 per user per month"; "200 MB on Plus") still fail it. It was written after reading the outputs, so it is not blind, and the 100% it gives below is a pass rate on a key fitted to these answers, not evidence that the models make no errors. Both keys are in every run file. A key written against a small model's terse answers failed these models' fuller ones.
  • The key alone showed a regression of the same size on all three models. With the key as written, the friendly prompt drops every model by 7.1 points (3 of 42 cases, none the other way, McNemar p = 0.25), more than the gate's 3-point tolerance. The failing cases differ: q05, q37 and q63 for claude-sonnet-5-5 and gemini-4-argon-medium, the three apostrophe refusals for gpt-6-astra. The gate warned rather than failed because 3 cases are not significant; on 42 cases the exact test needs at least 6 cases worse and none better, a 14.3-point drop, before it can call a drop real. With the reviewed key the change is 0 on every model and the gate passes. A paired test cannot tell a scorer's error from a regression; only reading the failing answers can, which is what compare lists them for.
  • No regression from the friendly prompt was detected here, on 42 cases. On qwen3:1.7b it cost 6.0 points on the key (section 5, p = 0.23, not significant). Here the reviewed key passes every answer, but 42 cases cannot rule out a small drop (the 95% interval for 42 of 42 reaches down to 91.6%), and the five questions the facts do not answer have no reject patterns, so a refusal followed by a guess passes: on q83 gpt-6-astra and gemini-4-argon-medium declined to give a euro price and then said it would depend on the payment provider's exchange rate, which the facts do not say. The prompt did change the style: median answer length went from 19.5 to 60 words for claude-sonnet-5-5, 13 to 17.5 for gpt-6-astra and 20 to 75.5 for gemini-4-argon-medium.
  • The text varies between runs; the router's verdicts did not. Run twice with the same prompt, 17 to 29 of the 42 support answers came back worded differently on each model. The router, scored by labels written before the runs, gave the same label on both runs in all 90 case pairs (95% upper bound on the change rate 3.3% overall, 9.5% per model). The support verdicts are not evidence either way: the reviewed key was fitted to both runs' answers, so agreement on it holds by construction. The key as written changed its verdict between runs three times (q37 on claude-sonnet-5-5, q79 and q81 on gpt-6-astra), each time on wording it misread, which is how text-level variation turns into a flaky test.
  • The router agrees with itself and with the other models. All three models gave identical labels on both runs. The one miss, t53 "Do you have a student discount?", is labelled other; all three said billing, and the router prompt puts prices under billing, so the label is the likelier error.

Latency differs by a factor of six, but it includes each provider's serving path, so it describes the models as reached here, not their APIs.

Limits

  • One machine, one pair of small local models, one product domain, 83 cases. The numbers describe this setup; the method is what transfers.
  • The answer key is a stand-in for human review, and section 4 shows it is wrong in predictable ways. A real deployment should calibrate against human labels. Section 9's reviewed labels cover 30 baseline cases, read by one reviewer with a model's draft; the candidate run has no reviewed sample.
  • Ollama's determinism with a fixed seed was checked on 20 cases on this backend only.
  • Section 8 measures a reasoning judge on one model and one domain. Its literal reading helped here and hurt where the reference itself was wrong; that balance will differ elsewhere.
  • The runs in runs/ keep the original references, including two (q21, q22) that name a plan the product facts do not; section 8 reports the corrected set separately.
  • Section 10 is a sample: 42 support and 30 router cases, two runs, one day. For the Anthropic and OpenAI models the suite's prompt replaced the system prompt. For the Google model it could not be set as a system prompt; it was sent at the start of the message, after the serving client's own instructions, which included two unrelated rules about git commits and pull requests. Its reviewed key was written after the outputs were read, by one reviewer with a model's help, and is not a blind measurement.