evalproof¶
Treat LLM and agent evaluation as measurement. Calibrate the judge, test agents through their tool calls, and gate CI on regressions that are real rather than noise.
Try it in the browser: evalproof.streamlit.app. Check whether a drop between two runs is real, or replay the run below and read the twelve answers that changed. It runs this repository's code; after a quiet spell it takes about half a minute to wake.
An LLM judge is a classifier with error rates. If it passes bad answers at rate FPR and
good answers at rate TPR, its pass rate converges to FPR + (TPR − FPR) · p, not to the
true rate p. That is classify-and-count, and it is biased the way any classifier's count
is — the effect is worked through with a seeded simulation in
The classifier was 95% accurate. The trend it counted was a third too flat
(lab).
-
Measure the judge
Confusion matrix, TPR, FPR and Cohen's κ against a small gold-labelled sample, then a pass rate corrected for the judge's errors.
-
Compare runs
Paired cases, exact McNemar for pass/fail and a paired bootstrap for scores, with an interval and a p-value.
-
Gate CI
Fail only when a drop is larger than your tolerance and significant, plus latency and cost budgets, with distinct exit codes.
-
Test agents
Assert on the tools an agent called and what it said, across several turns and several builds of the same system.
-
Repeated trials
Run each case several times and read pass@k, pass^k and the flaky cases, or compare several candidates in one table.
-
Import test cases and results
Turn the test cases you have in promptfoo, DeepEval, Ragas, LangSmith and others into a suite, or gate their results without re-running them.
-
Find what is not tested
List the model calls, agents and tools in a Python or TypeScript repository that no suite reaches, and draft inputs for them.
-
TypeScript and any language
Test a system in any language as a command or HTTP target, with a helper library on npm for TypeScript and JavaScript.
What evalproof does¶
- Measures the judge against a small gold-labelled sample: confusion matrix, TPR, FPR, Cohen's κ.
- Corrects its pass rate with adjusted classify-and-count and prediction-powered inference, each with a confidence interval.
- Compares runs on paired cases: exact McNemar for pass/fail, paired bootstrap for scores, and a paired permutation test when each case runs several times.
- Gates CI only when a drop is both larger than your tolerance and statistically significant, plus latency and cost budgets, with distinct exit codes.
- Checks the run files it gates:
verifyproves, with no model call, that a committed run was made with this suite and covers the dataset as it is now. - Tests agents: asserts on the tools an agent called and what it said, across several turns, several builds of the same system and repeated trials, with adapters for ADK, LangChain/LangGraph, the Claude Agent SDK, OpenAI Agents, LlamaIndex, CrewAI, Pydantic AI, DSPy, Haystack, AutoGen and Semantic Kernel (Testing agents).
- Finds what is not tested:
scanlists the model calls, agents and tools in a Python or TypeScript repository that no suite reaches, anddraftwrites inputs for them, never expected answers (Find what is not tested). - Tests a system in any language: a
command:or HTTP target, and for TypeScript and JavaScript theevalproofnpm package'srunCommandandserve(TypeScript and JavaScript).
It is local-first: Ollama is the default, every major hosted provider is one target: line
away (Providers), a logged-in Claude Code or Codex can be the judge with no API
key, and every model call is cached by content so re-runs are free and reproducible. Results
from promptfoo, Inspect, DeepEval, lm-eval, garak, Google ADK and Claude Code plugin evals can be
imported and gated without re-running
them, and the test cases you already have in promptfoo, DeepEval, Ragas, LangSmith, OpenAI Evals,
Inspect, Braintrust, Langfuse, Phoenix or a Google ADK eval set can be turned into a dataset and a
suite (Import test cases).
Who it is for¶
Teams that ship an LLM feature, a RAG pipeline or an agent and change its prompt, model, retrieval or tools often. Use it when you need to know whether a change made answers worse before it merges, whether a cheaper model is good enough, whether the LLM judge you score with can be trusted, and which model calls and tools in the codebase no test covers. It suits a team that already writes test cases in another tool and wants a CI gate that does not fail on noise. It is not a tracing or monitoring service and does not watch production traffic; it scores runs of a suite, locally or in CI.
Install in one line¶
Then follow the five-minute quickstart.
What it found on a real run¶
A local qwen3:1.7b product support assistant, judged by qwen3:8b, 83
questions, one Apple M5. Full method and every table in Measurements.
| Question | Measured |
|---|---|
| How good is the 8B judge? | TPR 92.6%, FPR 6.7%, Cohen's κ 0.78 against the answer key |
| Did swapping the grounded prompt for a "friendly" one regress quality? | −4.8 pts on the judge, 8 cases worse vs 4 better, McNemar p = 0.39 — gate warns, does not fail; a delta-only gate would have failed it |
| What did regress? | p95 latency, 1,469 → 4,422 ms; mean answer 19 → 77 tokens. A 2,500 ms budget fails the change with exit 3 |
| Was the gold trustworthy? | No: of 13 judge–key disagreements the key was wrong 7 times, the judge 4. Most of the "bias" corrected was the gold's |
| Is pairwise judging safer? | No: 42% of verdicts flipped with the order of the two answers, and the judge preferred the candidate 42 to 6 — mostly where both answers were right and the candidate's was five times longer |
| Is it reproducible? | 20/20 identical outputs and verdicts with the cache off; a warm-cache re-run took 0.05 s instead of 363 s; a re-judge three days later matched 83/83 |
The other findings (correcting the judge's rate, reviewed labels, a reasoning judge) are in Measurements.
On three frontier models¶
A 42-case sample of the same support questions and 30 ticket-router cases on claude-sonnet-5-5,
gpt-6-astra and gemini-4-argon-medium, 558 calls, no judge
(Measurements, section 10).
| Question | Measured |
|---|---|
| Did the answer key hold? | No: it failed 12 answers, all correct on reading, and alone showed a −7.1 pt drop from the "friendly" prompt on all three models. With the key reviewed, no regression was detected (42 cases) |
| Do the answers repeat? | The wording changed between two runs in 17 to 29 of 42 support cases; the router gave the same label in all 90 case pairs |
Documented version: 1.0.x. This site tracks the latest release.