Skip to content

description: A suite file and one or two commands for each common goal: gating, choosing a model, trusting a judge, retrieval, agents.

Use cases

Each of these is a suite file and one or two commands. The quickstart writes a starting suite. This page is for anyone looking for the recipe that matches a goal.

Goal Where to read more
Gate a prompt or model change in a pull request In CI
Choose between two models or providers Providers
Find out whether your LLM judge can be trusted Calibrate
Use your Claude Code or Codex subscription as the judge Providers
Gate results you already have from another eval tool Import results
Check a retrieval or RAG change Scorers
Cases that must never fail, and a floor on the first run Checks without a baseline
Hold latency and cost while you swap a model Compare and gate
Test what an agent called and what it said Testing agents
Run each case several times Repeated trials
Start from test cases you already have in another tool Import test cases
Find the model calls, agents and tools no suite tests Find what is not tested
Test a TypeScript or JavaScript system TypeScript and JavaScript
Check the committed run before each push, or run a subset of cases In CI

Gate a prompt or model change in a pull request

Run the suite on the base branch and on the change, then gate. The build fails only when the drop is larger than max_drop and significant, so a two-case wobble on 60 cases does not block a merge:

Terminal
evalproof run evals/suite.yaml --out base.json                     # on main
evalproof run evals/suite.yaml --out cand.json                     # on the branch
evalproof compare base.json cand.json --suite evals/suite.yaml --gate --show 5

Choose between two models or providers

Run the same suite with --system and compare the two runs. The report gives the paired difference with its interval and p-value, the cases that got worse, p95 latency and cost, so "cheaper and not measurably worse" is something you can show:

Terminal
evalproof run evals/suite.yaml --system anthropic:claude-sonnet-4-5 --label sonnet --out a.json
evalproof run evals/suite.yaml --system groq:llama-3.3-70b-versatile --label llama --out b.json
evalproof compare a.json b.json --suite evals/suite.yaml

Find out whether your LLM judge can be trusted

Label 30 to 50 cases by hand ({"id": ..., "label": true} per line) and calibrate. You get the judge's TPR, FPR and Cohen's κ, and its pass rate corrected for those errors, each with an interval:

Terminal
evalproof calibrate runs/latest.json --judge-scorer judge --labels gold.jsonl

Use your Claude Code or Codex subscription as the judge

Set judge.target: claude-code:haiku (or codex:<model>), run evalproof doctor evals/suite.yaml, and run as usual. No API key and no per-call bill; the CLI runs isolated, with no tools and none of your project's instructions (details).

Gate results you already have from another eval tool

Import each side, then compare as usual. Nothing is re-run:

Terminal
evalproof import promptfoo main/output.json --out base.json
evalproof import promptfoo pr/output.json --out cand.json
evalproof compare base.json cand.json --gate --metric pass

Check a retrieval or RAG change

Score the ranked ids with recall_at_k, mrr and ndcg and gate on the one that matters to you. Recall@5 does not move when the right document drops from rank 1 to rank 5; MRR and nDCG do (examples/retrieval).

Cases that must never fail, and a floor on the first run

Tag the critical cases and set gate.must_pass: [critical] and gate.min_pass_rate. They need no baseline, so they work from the first run: evalproof run evals/suite.yaml --gate exits 1 when one fails (Checks without a baseline).

Hold latency and cost while you swap a model

gate.max_p95_latency_ms and gate.max_cost_usd fail a change with exit 3, separate from a quality regression. On the measured run, a prompt change that was not a significant quality drop still failed on latency.

Gate in CI with no model calls and no secrets

evalproof init --ci keyless writes a workflow that compares the run file developers commit against the base branch's. The action runs verify first, so a stale run, a --limit run or a hand-edited deterministic score fails the job (In CI).

Test a non-LLM system the same way

Any Python function (python:module:func), HTTP endpoint or program in another language (command:node eval/system.js, see Testing a system in any language) can be the system, scored by deterministic scorers: a classifier, a router, a search service. The offline example is a keyword router.

Start from test cases you already have in another tool

Turn the cases written for promptfoo, DeepEval, Ragas, LangSmith, OpenAI Evals, Inspect, Braintrust, Langfuse or Phoenix, or a JSONL or CSV file, into a dataset and a suite that gates the scorer covering the most cases. init --from FILE does the same while setting up a repository (Import test cases):

Terminal
evalproof import-dataset auto promptfooconfig.yaml --out evals/cases.jsonl --suite-out evals/suite.yaml
evalproof run evals/suite.yaml --out evals/runs/latest.json

Find the model calls, agents and tools no suite tests

scan reads a repository without running it or calling a model, and lists the model calls, agents and tools in Python and TypeScript that no suite reaches, the tools no check asserts on and the suites with few cases. draft writes inputs for those gaps, from your existing tests or from a model, tagged draft. It never writes an expected answer: that comes from your code, your documentation, your tests or a person (Find what is not tested):

Terminal
evalproof scan . --out scan.json
evalproof draft --scan scan.json --out evals/drafts.jsonl --from-tests-only

Test a TypeScript or JavaScript system

The evalproof npm package is also a small library. runCommand(handler) turns an async function into a command: target and serve(handler) into an HTTP target. fromVercelAI and fromLangChain turn an AI SDK result or a LangChain.js agent's messages into a trace that the checks scorer reads. evalproof init --node writes a starting evals/system.mjs (TypeScript and JavaScript):

Terminal
npm install --save-dev evalproof
evalproof init --node

Check the committed run before each push, or run a subset of cases

evalproof hook install adds a git pre-push hook that runs verify, so a stale run file is caught before CI sees it. run --tags and --exclude-tags run part of a suite, and the gate refuses to compare two runs made with different filters (In CI).

Next steps