description: A suite file and one or two commands for each common goal: gating, choosing a model, trusting a judge, retrieval, agents.¶
Use cases¶
Each of these is a suite file and one or two commands. The quickstart writes a starting suite. This page is for anyone looking for the recipe that matches a goal.
| Goal | Where to read more |
|---|---|
| Gate a prompt or model change in a pull request | In CI |
| Choose between two models or providers | Providers |
| Find out whether your LLM judge can be trusted | Calibrate |
| Use your Claude Code or Codex subscription as the judge | Providers |
| Gate results you already have from another eval tool | Import results |
| Check a retrieval or RAG change | Scorers |
| Cases that must never fail, and a floor on the first run | Checks without a baseline |
| Hold latency and cost while you swap a model | Compare and gate |
| Test what an agent called and what it said | Testing agents |
| Run each case several times | Repeated trials |
| Start from test cases you already have in another tool | Import test cases |
| Find the model calls, agents and tools no suite tests | Find what is not tested |
| Test a TypeScript or JavaScript system | TypeScript and JavaScript |
| Check the committed run before each push, or run a subset of cases | In CI |
Gate a prompt or model change in a pull request¶
Run the suite on the base branch and on the
change, then gate. The build fails only when the drop is larger than max_drop and
significant, so a two-case wobble on 60 cases does not block a merge:
evalproof run evals/suite.yaml --out base.json # on main
evalproof run evals/suite.yaml --out cand.json # on the branch
evalproof compare base.json cand.json --suite evals/suite.yaml --gate --show 5
Choose between two models or providers¶
Run the same suite with --system and compare the
two runs. The report gives the paired difference with its interval and p-value, the cases that
got worse, p95 latency and cost, so "cheaper and not measurably worse" is something you can show:
evalproof run evals/suite.yaml --system anthropic:claude-sonnet-4-5 --label sonnet --out a.json
evalproof run evals/suite.yaml --system groq:llama-3.3-70b-versatile --label llama --out b.json
evalproof compare a.json b.json --suite evals/suite.yaml
Find out whether your LLM judge can be trusted¶
Label 30 to 50 cases by hand
({"id": ..., "label": true} per line) and calibrate. You get the judge's TPR, FPR and
Cohen's κ, and its pass rate corrected for those errors, each with an interval:
Use your Claude Code or Codex subscription as the judge¶
Set judge.target:
claude-code:haiku (or codex:<model>), run evalproof doctor evals/suite.yaml, and run as
usual. No API key and no per-call bill; the CLI runs isolated, with no tools and none of your
project's instructions (details).
Gate results you already have from another eval tool¶
Import each side, then compare as usual. Nothing is re-run:
evalproof import promptfoo main/output.json --out base.json
evalproof import promptfoo pr/output.json --out cand.json
evalproof compare base.json cand.json --gate --metric pass
Check a retrieval or RAG change¶
Score the ranked ids with recall_at_k, mrr and ndcg
and gate on the one that matters to you. Recall@5 does not move when the right document drops
from rank 1 to rank 5; MRR and nDCG do (examples/retrieval).
Cases that must never fail, and a floor on the first run¶
Tag the critical cases and set
gate.must_pass: [critical] and gate.min_pass_rate. They need no baseline, so they work
from the first run: evalproof run evals/suite.yaml --gate exits 1 when one fails
(Checks without a baseline).
Hold latency and cost while you swap a model¶
gate.max_p95_latency_ms and
gate.max_cost_usd fail a change with exit 3, separate from a quality regression. On the
measured run, a prompt change that was not a significant quality drop still failed on
latency.
Gate in CI with no model calls and no secrets¶
evalproof init --ci keyless writes a
workflow that compares the run file developers commit against the base branch's.
The action runs verify first, so a stale run, a --limit run or a hand-edited
deterministic score fails the job (In CI).
Test a non-LLM system the same way¶
Any Python function (python:module:func), HTTP
endpoint or program in another language (command:node eval/system.js, see
Testing a system in any language) can be the system, scored by deterministic scorers: a classifier, a router, a search
service. The offline example is a keyword router.
Start from test cases you already have in another tool¶
Turn the cases written for promptfoo, DeepEval, Ragas, LangSmith, OpenAI Evals, Inspect,
Braintrust, Langfuse or Phoenix, or a JSONL or CSV file, into a dataset and a suite that gates the
scorer covering the most cases. init --from FILE does the same while setting up a repository
(Import test cases):
evalproof import-dataset auto promptfooconfig.yaml --out evals/cases.jsonl --suite-out evals/suite.yaml
evalproof run evals/suite.yaml --out evals/runs/latest.json
Find the model calls, agents and tools no suite tests¶
scan reads a repository without running it or calling a model, and lists the model calls,
agents and tools in Python and TypeScript that no suite reaches, the tools no check asserts on and
the suites with few cases. draft writes inputs for those gaps, from your existing tests or from a
model, tagged draft. It never writes an expected answer: that comes from your code, your
documentation, your tests or a person (Find what is not tested):
evalproof scan . --out scan.json
evalproof draft --scan scan.json --out evals/drafts.jsonl --from-tests-only
Test a TypeScript or JavaScript system¶
The evalproof npm package is also a small library. runCommand(handler) turns an async
function into a command: target and serve(handler) into an HTTP target. fromVercelAI and
fromLangChain turn an AI SDK result or a LangChain.js agent's messages into a trace that the
checks scorer reads. evalproof init --node writes a starting evals/system.mjs
(TypeScript and JavaScript):
Check the committed run before each push, or run a subset of cases¶
evalproof hook install adds a git pre-push hook that runs verify, so a stale run file is
caught before CI sees it. run --tags and --exclude-tags run part of a suite, and the gate
refuses to compare two runs made with different filters
(In CI).
Next steps¶
- Install and run the first suite: the quickstart.
- Read how the gate decides: Concepts.