1. An LLM judge is a classifier, and its pass rate is classify-and-count¶
Status: accepted · 2026-10
Context¶
Most LLM evaluation reports a single number: the fraction of outputs a judge model marked as passing. That number is classify-and-count. If the judge passes bad answers at a false positive rate FPR and passes good answers at a true positive rate TPR, then on a set whose true pass rate is p the judge's rate converges to
q = FPR + (TPR − FPR) · p
which equals p only when FPR = 0 and TPR = 1. Every real judge has error. The bias also moves with p: when a candidate system gets worse, a judge with a non-zero FPR reports a smaller drop than really happened. The same algebra is worked through with a seeded simulation in prevalence-shift.
Decision¶
evalproof treats the judge as an instrument to be calibrated, not an oracle:
- A calibration sample of gold labels — human review, or a trusted deterministic check — measures the judge's confusion matrix, TPR, FPR and Cohen's κ.
- The judge's pass rate is reported next to two corrected estimates:
- ACC (Rogan–Gladen):
p = (q − FPR) / (TPR − FPR), with a bootstrap interval that resamples both the judged set and the gold sample. - PPI (prediction-powered inference): the judge's mean on unlabelled cases plus the mean residual (label − judge) on the gold sample, with a normal interval.
- ACC (Rogan–Gladen):
- Undecided verdicts (an unparseable reply after one retry) are counted and excluded, never silently scored as passes.
Consequences¶
- Someone has to label a sample. The default example uses 30 cases — what a team could review by hand per release — and the measurements show how wide the intervals are at that size.
- ACC can be undefined (a judge no better than chance) and needs both classes in the gold sample; the report says so instead of printing a number.
- PPI needs no assumption about the judge but does need the gold sample to be random. evalproof draws it with a seeded RNG and records the seed.