Measurements¶
Every figure here comes from real runs of examples/support-qa on
one machine. The run files are committed in examples/support-qa/runs/, and
uv run python examples/support-qa/measure.py recomputes the calibration, comparison and
latency tables from them without a model. Sections 1–9 use small local models; section 10
repeats part of the study on three hosted frontier models, with its own run files and script in
examples/frontier-study.
Setup¶
| Machine | Apple M5, 24 GB RAM, macOS 27.0.1 (arm64) |
| Ollama | 0.23.2 |
| System under test | qwen3:1.7b, digest 8f68893c685c — temperature 0, seed 0, think: false, max 200 tokens |
| Judge | qwen3:8b, digest 500a1f067a9f — temperature 0, seed 0, think: false, max 256 tokens |
| Reasoning judge (section 8) | the same model with think: true and max 4,096 tokens, in suite.reasoning-judge.yaml |
| evalproof / Python | 0.1.0 / 3.13.15 |
| Golden set | 83 questions about a product; 9 are not answerable from the facts. Every run in runs/ used cases.v1.jsonl, dataset hash c6f08ac201023363; cases.jsonl is the revised set from section 8 (55ba005be14c0f20), whose runs are in runs/v2/ |
| "Gold" | a per-question regex answer key, written before any model output was seen and not changed after. It stands in for human review; it is not human labelling. |
| Reviewed labels (section 9) | pass/fail for the 30 baseline cases of the seed-0 random sample, read against the rubric and the product facts, in human-review/ |
| Change under test | grounded system prompt → "friendly, always give a complete answer" prompt; same model |
1. The two runs¶
| Baseline (grounded prompt) | Candidate (friendly prompt) | |
|---|---|---|
| Answer-key pass rate | 81.9% [72.3, 88.7] | 75.9% [65.7, 83.8] |
| Judge pass rate | 77.1% [67.0, 84.8] | 72.3% [61.8, 80.8] |
| System latency p50 / p95 | 624 / 1,469 ms | 2,243 / 4,422 ms |
| Mean output length | 19 tokens | 77 tokens |
| Run wall time (system + judge, sequential) | 363 s | 588 s |
| Re-run with a warm cache | 0.05 s, results identical to the cold run | — |
Intervals are 95% Wilson. Wall times are for 83 cases with one system call and one judge call each, no concurrency.
2. How good is the judge? Measured against the full answer key¶
| Baseline | Candidate | |
|---|---|---|
| Judge pass · key pass (TP) | 63 | 58 |
| Judge pass · key fail (FP) | 1 | 2 |
| Judge fail · key pass (FN) | 5 | 5 |
| Judge fail · key fail (TN) | 14 | 18 |
| TPR / FPR | 92.6% / 6.7% | 92.1% / 10.0% |
| Cohen's κ | 0.779 | 0.781 |
3. Correcting the judge's rate from a 30-case gold sample¶
A team will not hand-label all 83 cases every release. The suite calibrates against a random 30. Over 200 different random samples of 30 (seeds 0–199), against the answer key on all 83 as the truth:
| Baseline (truth 81.9%) | Mean point | Mean |error| | 95% CI covers truth | Mean CI width |
|---|---|---|---|---|
| Raw judge rate (all 83) | 77.1% | 4.8 pts | 100% | 17.8 pts |
| Gold sample alone (30) | 82.8% | 4.1 pts | 100% | 26.2 pts |
| ACC (judge on 83, error rates from 30) | 81.5% | 3.8 pts | 100% | 34.1 pts |
| PPI (judge on 53, residuals from 30) | 81.2% | 4.7 pts | 98.5% | 28.6 pts |
| Candidate (truth 75.9%) | Mean point | Mean |error| | 95% CI covers truth | Mean CI width |
|---|---|---|---|---|
| Raw judge rate | 72.3% | 3.6 pts | 100% | 18.9 pts |
| Gold sample alone | 76.1% | 4.3 pts | 99.0% | 29.1 pts |
| ACC | 75.3% | 4.0 pts | 100% | 41.0 pts |
| PPI | 75.4% | 4.7 pts | 98.5% | 31.6 pts |
What this shows. The raw judge rate sits 3.6–4.8 points below the key, and both corrections remove that offset on average. At this size it buys little: the offset is smaller than the raw rate's own interval, which covered the truth every time, while the corrected intervals are about twice as wide. Calibration earns its keep when the suite is large enough that the raw interval is narrow and the bias dominates — the regime the prevalence-shift simulation shows, where the raw interval covered the truth 0% of the time at 5,000 reports per period — or when the judge is worse than this one.
4. The gold was wrong more often than the judge¶
The judge and the answer key disagreed on 13 cases across the two runs. Each was read by hand:
| Case | Run | Key | Judge | Who was right | Why |
|---|---|---|---|---|---|
| q76 lifetime licence | both | pass | fail | judge | the answer invents "no lifetime licence"; the key's abstention regex matched "not … provide" |
| q78 HIPAA | both | pass | fail | judge | the answer asserts "not HIPAA compliant"; the key matched "does not mention" |
| q77 uptime SLA | baseline | pass | fail | judge | the answer invents 99.9% and 99.99% SLAs; the key matched "no SLA … specified" |
| q28 which plans have trials | candidate | pass | fail | judge | the answer adds a trial to the Free plan; the key only checked that Plus and Team appear |
| q31 cancelling monthly | candidate | pass | fail | judge | the answer says service stops immediately; the key matched "until the end" in a later bullet |
| q56 two-factor methods | baseline | pass | fail | key | correct answer plus a true extra fact (no SMS); the judge penalised the extra fact |
| q65 integrations | baseline | pass | fail | key | correct answer plus a true extra fact; same failure |
| q32 payment methods | candidate | pass | fail | key | correct answer plus a true extra fact; same failure |
| q05 Enterprise pricing | candidate | fail | pass | key | the answer quotes Team's prices for Enterprise; the judge saw "custom pricing" in a bullet and passed it |
| q66 Microsoft Teams | both | fail | pass | ambiguous | "Yes, Quillfox offers an integration…" then "Microsoft Teams is not supported" |
The key was wrong 7 times, the judge 4 times, 2 are ambiguous. The key's errors are structural: a regex can tell that the right fact is present but not that a wrong one is absent, and it cannot tell an abstention from an invention phrased as one. Correcting the key on these cases puts the baseline at 78.3% and the candidate at 71.1% — within about a point of the raw judge. Most of the "bias" that section 3 corrected was the gold's, not the judge's. Calibration is only as good as its labels, and the labels need the same scrutiny as the judge. The judge's own failure is consistent too: three of its four errors penalise correct extra detail the rubric explicitly allows.
5. Did the prompt change regress?¶
evalproof compare … --gate on the 83 paired cases:
| Metric | Baseline | Candidate | Δ (95% CI) | Test | p |
|---|---|---|---|---|---|
| answer_key | 81.9% | 75.9% | −6.0 pts [−14.5, +1.2] | exact McNemar | 0.227 |
| judge (gated) | 77.1% | 72.3% | −4.8 pts [−13.3, +3.6] | exact McNemar | 0.388 |
The judge saw 8 cases get worse and 4 get better. Gate: pass with a warning (exit 0) — the drop is beyond the 3-point tolerance but p = 0.39 at n = 83, so the data cannot tell it from noise. A delta-only gate would have failed the change on the same numbers.
The regression that is measurable is latency: p95 went from 1,469 ms to 4,422 ms. With
--max-p95-ms 2500 the same command fails with exit 3 (budget breach) and still
prints the quality warning.
6. Pairwise judging: position and length bias¶
evalproof pairwise asked the judge to pick the better answer for each of the 83 pairs,
in both orders (166 calls, 776 s):
- 35 of 83 verdicts (42.2%) flipped when the order was swapped. In 32 of those the answer shown first won both times. evalproof reports them as ties.
- Of the 48 consistent verdicts, the judge preferred the candidate 42 times and the baseline 6 — although the pointwise judge and the answer key both rate the candidate lower. In 31 of the 42, both answers were correct by the key; on those cases the candidate's answers averaged 383 characters against the baseline's 78. In 9, the candidate's answer was wrong by the key.
An 8B pairwise judge without the order swap would have reported a decisive win for the worse prompt. Position bias is cancelled by the swap; length bias is not, and is why evalproof gates on pointwise verdicts and reports pairwise preference as a diagnostic.
7. Determinism and the cache¶
- 20 cases re-run with the cache disabled: 20/20 identical system outputs, 20/20 identical verdicts, 20/20 identical judge reasons (temperature 0, seed 0, Ollama 0.23.2).
- Re-running the full baseline with a warm cache took 0.05 s instead of 363 s and produced
results identical to the cold run, apart from the per-case
cachedflag. - Three days later, the full baseline judged again with the judge's cache entries removed
(system answers still cached) gave 83/83 identical verdicts in 201 s
(
baseline.think-off-rerun.json).
8. A reasoning judge¶
The same qwen3:8b judge with thinking switched on (issue #3), on the same cached answers,
so the only change is the judge. measure.py prints every table below from the committed
runs.
| Thinking off | Thinking on | |
|---|---|---|
| Judge time, 83 baseline verdicts (answers cached) | 201 s (2.4 s each) | 1,338 s (16.1 s each) |
| Judge time, 83 candidate verdicts | — | 1,188 s |
| Judge tokens per verdict, baseline / candidate | 306 / 369 | 590 / 607 |
| Undecided verdicts | 0 | 0 |
| Verdicts that changed, baseline / candidate | — | 7 / 6 |
Against the answer key, thinking looks worse: κ falls from 0.779 to 0.676 on the baseline and from 0.781 to 0.727 on the candidate. That comparison is unfair, because section 4 showed the key is often the one that is wrong. So every case where the key and the two judges did not all agree (22, the 13 from section 4 and 9 that only the thinking judge disputes) was read case by case against the rubric and the product facts. The 9 new ones:
| Case | Run | Key | Off | On | Right | Why |
|---|---|---|---|---|---|---|
| q28 which plans have trials | baseline | pass | pass | fail | on | the answer gives the Free plan a trial, the same invention section 4 found in the candidate |
| q07 annual saving on Plus | candidate | pass | pass | fail | on | the answer states "$60 per year" before working its way to $12 |
| q42 Linux desktop app | baseline | fail | fail | pass | on | "Yes, there is no Linux desktop app; the web app works on Linux": the facts are right; the key rejects any answer that starts with "Yes" |
| q42 Linux desktop app | candidate | fail | fail | pass | on | the same facts; the off judge's reason contradicts itself |
| q83 Plus in euros | baseline | pass | pass | fail | off | the answer gives the dollar price and says the euro price is not covered; correct extra detail, not an invented answer |
| q21 SSO, q22 admin console | both | pass | pass | fail | off | the answer names Team. The reference adds Enterprise, which the product facts do not say; the thinking judge followed the reference |
Against those audited labels (the key with every reading applied, q66 left out as ambiguous):
| Thinking off | Thinking on | |
|---|---|---|
| Baseline TPR / FPR | 95.4% / 5.9% | 95.4% / 0.0% |
| Baseline Cohen's κ | 0.858 | 0.895 |
| Candidate TPR / FPR | 96.6% / 8.7% | 96.6% / 0.0% |
| Candidate Cohen's κ | 0.879 | 0.941 |
| Right on the 20 decided contested cases | 12 | 15 (the answer key: 9) |
What this shows. Thinking made the judge better where it matters most for a gate: it passed no wrong answer in either run, against three for the judge without it, and it fixed all three cases where that judge penalised correct extra detail. It costs 6.7 times the judge time and about twice the tokens. It also reads the reference more literally, which cuts both ways: it caught an answer that states a wrong number before the right one, and it failed two correct answers because the golden set's reference claims more than the product facts do. That is a defect in two reference answers, now visible because a stricter judge tripped on it.
The gate's decision did not change. With thinking on, the judge saw 6 cases get worse and 1 get better (−6.0 points, 95% CI [−12.0, +0.0], exact McNemar p = 0.125) instead of 8 and 4 (p = 0.388): cleaner evidence of the same small regression, still not distinguishable from noise at n = 83, so the gate still passes with a warning.
After correcting the two references (issue #7)¶
cases.jsonl now says Team only for q21 and q22, as the product facts do; cases.v1.jsonl
keeps the original so the runs above stay reproducible. All four runs were judged again on
the revised set (runs/v2/): every answer came from the cache,
and the only verdicts that moved were the thinking judge's on q21 and q22, now passes.
| Against the audited labels | Thinking off | Thinking on |
|---|---|---|
| Baseline TPR / FPR, κ | 95.4% / 5.9%, 0.858 | 98.5% / 0.0%, 0.964 |
| Candidate TPR / FPR, κ | 96.6% / 8.7%, 0.879 | 100.0% / 0.0%, 1.000 |
| Right on the 16 decided contested cases | 8 | 15 (the answer key: 5) |
With the references fixed, the thinking judge's one remaining miss is q83, where it read a correct dollar price as an invented answer to a question about euros. The gate's decision is the same as before: 6 worse, 1 better, p = 0.125, pass with a warning.
The readings in sections 4 and 8 are careful case-by-case reads of the contested cases only. Section 9 reads a random sample instead.
9. Calibrating against reviewed labels (issue #4)¶
Sections 3 and 4 calibrated against the answer key and then found the key wrong. Here the key
is replaced by labels that were read. The 30 baseline cases are the seed-0 random sample that
calibrate --gold-sample 30 would draw, so PPI's assumption of a random gold sample holds. Each
answer was marked pass or fail against rubric.md and the
product facts, in review-sheet.csv (the
verdicts are also in labels.jsonl for
calibrate --labels). The verdicts were drafted with a model (Claude) and checked and accepted
by the maintainer: one reviewer's check, not an independent panel. q22 was read against its
corrected reference.
The labels give 22 passes and 8 failures. Against them:
- the answer key is wrong on one case of the 30: q28, where the answer gives the Free plan a trial and the key only checks that Plus and Team appear (the failure section 4 found in the candidate);
- the audited labels of sections 4 and 8 agree on all 29 cases they decide. The 30th, q66, which they left ambiguous, is a fail here: asked whether there is a Microsoft Teams integration, the answer opens "Yes" and then says there is none, and the rubric fails an answer that contradicts itself. Those audits were also read with Claude, so this agreement shows a consistent reading, not an independent confirmation.
| Judge · gold (30 cases) | TPR / FPR | κ | Gold only | ACC | PPI |
|---|---|---|---|---|---|
| thinking off · answer key | 95.7% / 14.3% | 0.814 | 76.7% [59.1, 88.2] | 77.2% [55.8, 92.7] | 77.4% [62.6, 92.1] |
| thinking off · reviewed | 95.5% / 25.0% | 0.734 | 73.3% [55.6, 85.8] | 74.0% [42.2, 92.4] | 74.0% [57.9, 90.2] |
| thinking on · answer key | 95.7% / 14.3% | 0.814 | 76.7% [59.1, 88.2] | 78.7% [57.6, 94.5] | 79.2% [64.8, 93.7] |
| thinking on · reviewed | 100.0% / 12.5% | 0.911 | 73.3% [55.6, 85.8] | 75.2% [57.8, 85.5] | 75.9% [63.1, 88.7] |
The raw judge rate on all 83 is 77.1% with thinking off and 78.3% with it on (the runs/v2/
verdicts). The best truth available for all 83, the audited labels with q66 failed, is 78.3%.
What this shows.
- One relabelled case moved the corrected rate about 3 points down for the published judge (ACC 77.2% → 74.0%, PPI 77.4% → 74.0%). With 8 failures in the sample, one more wrong answer the judge passed is 11 points of FPR, 14% → 25%. Error rates estimated on 30 labels are that fragile.
- The move went away from the best truth, not towards it: 74.0% against 78.3%, where the raw judge rate was 1.2 points off. The sample holds 8 failures in 30 (26.7%) against 18 in 83 (21.7%) for the full set, and the correction inherits that luck. Every interval still covers 78.3%; the ACC interval is 50 points wide.
- Better labels separate the two judges; the key could not. Against the key on these 30 cases the judges tie at κ 0.814. Against the reviewed labels the thinking judge reaches 0.911 and the judge without thinking falls to 0.734: the thinking judge failed q28, which the key passed, and its one miss is q66.
- On a random sample the key was wrong once in 30 (Wilson 95% CI 0.6%–16.7%), not the 7 of 13 section 4 found. Section 4 read only the cases where the key and the judge disagreed, and the key's errors cluster in the questions the facts do not answer (q76–q78), none of which fell in this sample. "7 of 13" describes the disagreements, not the key's overall error rate.
Section 3's conclusion stands and gets sharper: at this size a 30-case gold sample decides little about the pass rate, whoever labels it. What it does show is the confusion matrix, which is where the reviewed labels earned their keep: they found the judge passing an answer that contradicts itself (q66) and an invented trial (q28), and that the reasoning judge catches the second.
10. Frontier models on a sample¶
Sections 1–9 measure small local models. This section asks whether the same checks hold on
hosted frontier models, at a size meant to show the effect rather than to rank models: every
second case of the revised support set (42 of 83) and of the ticket-router
set (30 of 60), 558
calls in all, no judge. uv run python examples/frontier-study/measure.py recomputes every
figure below from the committed run files and re-scores every stored answer against the
committed answer keys; the study's
README
lists the files and every change made to the key.
| Models | Anthropic claude-sonnet-5-5; OpenAI gpt-6-astra, reasoning effort medium; Google gemini-4-argon-medium |
| Date | 2026-11-15, evalproof 0.8.0; run files rebuilt from the cache with evalproof 1.0.0, no model called |
| Sampling | the provider default; temperature and seed cannot be set |
| Support Q&A | the grounded prompt twice (84 answers per model), the friendly prompt once (42) |
| Ticket router | one-word queue label with prompt.router.md, twice (60 answers per model) |
| Scoring | the regex answer key as written (answer_key), the same key after review (answer_key_reviewed, the gate metric), and exact label match for the router |
| Model | Grounded, key as written | Grounded, reviewed key | Friendly, key as written | Friendly, reviewed key | Router | Latency p50 (support / router) |
|---|---|---|---|---|---|---|
claude-sonnet-5-5 |
83/84 | 84/84 | 39/42 | 42/42 | 58/60 | 2.0 / 1.8 s |
gpt-6-astra |
82/84 | 84/84 | 39/42 | 42/42 | 58/60 | 6.2 / 6.0 s |
gemini-4-argon-medium |
84/84 | 84/84 | 39/42 | 42/42 | 58/60 | 12.4 / 12.0 s |
What this shows.
- Every failure the answer key reported was the key's error. It failed 12 of the 378 support
answers, and a reading of each against the product facts found all 12 correct: five refusals
written with a typographic apostrophe ("I don’t have information…") that the key's
'did not match, and seven answers that added a true fact about another plan, which a reject pattern took for the wrong answer (q05, q37, q63). The reviewed key accepts’and narrows those three patterns to the wrong claim itself; wrong answers written for the purpose ("Enterprise pricing is custom. It typically works out to about $25 per user per month"; "200 MB on Plus") still fail it. It was written after reading the outputs, so it is not blind, and the 100% it gives below is a pass rate on a key fitted to these answers, not evidence that the models make no errors. Both keys are in every run file. A key written against a small model's terse answers failed these models' fuller ones. - The key alone showed a regression of the same size on all three models. With the key as
written, the friendly prompt drops every model by 7.1 points (3 of 42 cases, none the other
way, McNemar p = 0.25), more than the gate's 3-point tolerance. The failing cases differ:
q05, q37 and q63 for
claude-sonnet-5-5andgemini-4-argon-medium, the three apostrophe refusals forgpt-6-astra. The gate warned rather than failed because 3 cases are not significant; on 42 cases the exact test needs at least 6 cases worse and none better, a 14.3-point drop, before it can call a drop real. With the reviewed key the change is 0 on every model and the gate passes. A paired test cannot tell a scorer's error from a regression; only reading the failing answers can, which is whatcomparelists them for. - No regression from the friendly prompt was detected here, on 42 cases. On
qwen3:1.7bit cost 6.0 points on the key (section 5, p = 0.23, not significant). Here the reviewed key passes every answer, but 42 cases cannot rule out a small drop (the 95% interval for 42 of 42 reaches down to 91.6%), and the five questions the facts do not answer have no reject patterns, so a refusal followed by a guess passes: on q83gpt-6-astraandgemini-4-argon-mediumdeclined to give a euro price and then said it would depend on the payment provider's exchange rate, which the facts do not say. The prompt did change the style: median answer length went from 19.5 to 60 words forclaude-sonnet-5-5, 13 to 17.5 forgpt-6-astraand 20 to 75.5 forgemini-4-argon-medium. - The text varies between runs; the router's verdicts did not. Run twice with the same
prompt, 17 to 29 of the 42 support answers came back worded differently on each model. The
router, scored by labels written before the runs, gave the same label on both runs in all 90
case pairs (95% upper bound on the change rate 3.3% overall, 9.5% per model). The support
verdicts are not evidence either way: the reviewed key was fitted to both runs' answers, so
agreement on it holds by construction. The key as written changed its verdict between runs
three times (q37 on
claude-sonnet-5-5, q79 and q81 ongpt-6-astra), each time on wording it misread, which is how text-level variation turns into a flaky test. - The router agrees with itself and with the other models. All three models gave identical
labels on both runs. The one miss, t53 "Do you have a student discount?", is labelled
other; all three saidbilling, and the router prompt puts prices under billing, so the label is the likelier error.
Latency differs by a factor of six, but it includes each provider's serving path, so it describes the models as reached here, not their APIs.
Limits¶
- One machine, one pair of small local models, one product domain, 83 cases. The numbers describe this setup; the method is what transfers.
- The answer key is a stand-in for human review, and section 4 shows it is wrong in predictable ways. A real deployment should calibrate against human labels. Section 9's reviewed labels cover 30 baseline cases, read by one reviewer with a model's draft; the candidate run has no reviewed sample.
- Ollama's determinism with a fixed seed was checked on 20 cases on this backend only.
- Section 8 measures a reasoning judge on one model and one domain. Its literal reading helped here and hurt where the reference itself was wrong; that balance will differ elsewhere.
- The runs in
runs/keep the original references, including two (q21, q22) that name a plan the product facts do not; section 8 reports the corrected set separately. - Section 10 is a sample: 42 support and 30 router cases, two runs, one day. For the Anthropic and OpenAI models the suite's prompt replaced the system prompt. For the Google model it could not be set as a system prompt; it was sent at the start of the message, after the serving client's own instructions, which included two unrelated rules about git commits and pull requests. Its reviewed key was written after the outputs were read, by one reviewer with a model's help, and is not a blind measurement.