Skip to content

Changelog

All notable changes are listed here. The project follows Semantic Versioning.

Unreleased

1.4.0 — 2026-11-15

Added

  • Adapters for LlamaIndex, CrewAI, Pydantic AI, DSPy, Haystack, AutoGen and Semantic Kernel turn a run of each framework into a trace (tool calls, tokens), next to the adapters for LangChain, ADK, the Claude Agent SDK and the OpenAI Agents SDK. They read each framework's objects without importing it. They were written from each SDK's source and tested with fake objects of those shapes, not yet against real runs, and are experimental: they warn when used. See Agent SDK adapters.
  • The evalproof npm package is now a helper library for a JavaScript or TypeScript system as well as the npx launcher. runCommand(handler) runs a command: target: it reads the case from stdin, awaits the handler (which returns a string or {output, retrieved, trace, tokens_in, tokens_out, cost_usd}) and prints the result as JSON, with console.log sent to stderr. serve(handler, {port, host, path}) serves the http:// target protocol with node:http, on 127.0.0.1 by default. fromVercelAI(result) and fromLangChain(messages) build a trace (tool calls, tokens) from an AI SDK generateText result or a LangChain.js agent's messages. It ships TypeScript types, works from ESM and CommonJS, has no dependencies and needs Node 18 or newer.
  • evalproof init --node writes evals/system.mjs, a program that imports runCommand, and a suite whose target is command:node system.mjs. With --ci api-key on GitHub the workflow sets up Node and runs npm ci. It cannot be combined with --agent-files-only, and it is refused with --platform gitlab or bitbucket and --ci api-key, whose job image has no Node.
  • evalproof scan finds Vercel AI SDK tools and agents and LangChain.js tools and agents in TypeScript and JavaScript files, and follows a suite whose target is a node, tsx (or npx tsx), bun or deno command through the relative imports of its script. TypeScript code now takes part in the uncovered_code and tool_unchecked gaps. An import it cannot follow (an unresolved relative path, a path alias, a workspace package, a module loaded by a computed name) or a script it cannot find leaves reachability unknown and uncovered_code not computed. Test files and files that only import an SDK are not reported as untested code.
  • A guide, Testing a TypeScript or JavaScript system.

The helper library, init --node and the scan of TypeScript tools and agents are experimental: they can change in a minor release. The converters print one line to stderr the first time they are called.

Changed

  • scan reads a multi-line import { ... } from "..." and skips comments in TypeScript and JavaScript files, and no longer reports "imports X; no call found" for a LangChain.js file that defines tools. evalproof draft reads only Python files for a function's signature and docstring.

1.3.0 — 2026-11-15

Added

  • evalproof scan [PATH] reads a repository without running it or calling a model (ast for Python, patterns for TypeScript and JavaScript) and reports its model calls, agents, tools, prompts, multi-turn use, test-case files and evalproof suites. It then lists the gaps: no suite, code no suite's target reaches (followed through the repository's own imports from a python: target), tools no check names, no multi-turn case, pytest files that are not evalproof cases yet, and suites with fewer cases than their gate needs. Each gap comes with the command that drafts for it. --json and --out write the report as JSON. It always exits 0: gaps are advice, not a gate.
  • evalproof draft --scan scan.json --out evals/drafts.jsonl turns those gaps into a separate JSONL of draft cases, never a suite's dataset. Cases from pytest files keep the assertion their test already made (==, in, re.search) and call no model. For the other gaps a model writes inputs only, tagged draft, gap:<kind> and from:model, with draft_for and drafted_by: never an expected answer or a check. A person adds those before a draft goes into a dataset.
  • A guide, Find what is not tested.
  • evalproof init --platform bitbucket writes a Bitbucket Pipelines file with a pull-request step that runs and gates (--ci api-key) or verifies and gates the committed run (--ci keyless). An existing bitbucket-pipelines.yml that init did not write is never edited: init writes evals/bitbucket-pipelines.snippet.yml to merge by hand.
  • evalproof hook install writes a git pre-push hook that runs evalproof verify on the committed run file, with no model call; evalproof hook uninstall removes it. It follows core.hooksPath and never replaces a hook it did not write without --force.
  • evalproof run --tags and --exclude-tags (also on pairwise) run only the cases with those tags. The filter is recorded in the run file (tag_filter), and verify checks the run against the cases the same filter selects now.

scan, draft and init --platform bitbucket are experimental: they can change in a minor release.

Changed

  • compare --gate refuses (exit 4) two runs made with different tag filters, because a filter does not show in a pull request's diff and the gate would decide on a subset of the cases. Without --gate, compare reports the shared cases with a warning.

1.2.0 — 2026-11-15

Added

  • evalproof import-dataset reads the test cases you already have in another tool and writes an evalproof dataset: promptfoo (a promptfooconfig.yaml, or its tests as YAML, JSON, JSONL or CSV), DeepEval, Ragas, LangSmith, OpenAI Evals, Inspect, Braintrust, Langfuse, Phoenix, ADK, and any JSONL or CSV table (--id-field, --input-field, --reference-field, --tags-field). auto reads the format off the file and says which it chose; an ambiguous or unknown file exits 2.
  • What the source expected becomes a case field and a scorer. promptfoo's equals, contains/icontains (and their -any/-all forms), starts-with, regex, the not- forms, is-json with a schema, llm-rubric, g-eval, model-graded-closedqa and factuality map to exact_match, contains, regex, checks, json_schema and per-case rubrics. A free-text expected answer becomes the case's reference, graded by llm_judge. An assertion with no counterpart (javascript, python, similar, latency, cost, ...) is never dropped silently: it stays on the case under unmapped, and one line counts them by type.
  • Imported cases keep the source's id, or get <format>- and a hash of the input and the expected answer, so reordering the file never changes an id. They are tagged imported and from:<format>.
  • import-dataset --suite-out FILE also writes a suite with exactly the scorers those cases need, a judge block only when one of them is a judge, and a prompt_template when the input is an object of variables. The gate is set on the scorer that checks the most cases. It refuses to overwrite a file without --force, and writes nothing when no scorer fits.
  • evalproof init --from FILE (--from-format, default auto) starts a repository from those cases instead of the three sample cases.
  • A guide, Import test cases.

The deepeval, ragas, langsmith, openai-evals, inspect, braintrust, langfuse and phoenix dataset formats are experimental: they were written from each tool's documented fields and have not yet been checked against files exported by each tool.

Changed

  • import-dataset adk also accepts --suite-out. The dataset it writes is unchanged.

1.1.0 — 2026-11-15

Added

  • evalproof init adds an evalproof section to the repository's AGENTS.md, so any coding agent that reads that file knows which command to run for a task, how to read the exit codes, what never to weaken and how each kind of provider and target is set up. The section sits between <!-- evalproof:begin --> and <!-- evalproof:end -->. A missing file is created, a file without the markers gets the section appended after one blank line, and a file that has them gets only the text between them replaced; everything else in the file stays byte for byte, line endings included. Damaged markers (one without the other, repeated, or in the wrong order) stop init with exit 2 and nothing is written. --force does not overwrite the file. --no-agents-md skips it.
  • evalproof init --agent-files-only writes only the agent files (the AGENTS.md section and the skill) and no suite or CI workflow. It cannot be combined with --judge, --ci, --platform or --cli-auth. Existing skill copies are kept unless --force is given.
  • A guide, Coding agents.
  • An AGENTS.md at the root of this repository for agents that work on evalproof itself.

The text of the AGENTS.md section, the skill's text and --agent-files-only are experimental and can change in a minor release. The markers and the rule that init never changes anything outside them are stable.

Changed

  • The skill points to the AGENTS.md section for every task other than checking a change.
  • The npm package is published by the release workflow through npm trusted publishing, with a provenance statement, instead of by hand.

1.0.0 — 2026-11-15

The first stable release. The public surface (the names in evalproof.__all__ and evalproof.adapters.__all__, commands, flags, exit codes, --json fields, suite keys, the run file format and dataset fields) now changes incompatibly only in a major release, after at least one minor release of deprecation warnings. Experimental features stay outside that promise. See Versioning and support and Upgrading. Nothing from 0.8 is removed or renamed.

These are experimental, outside that promise until they have been used in more projects: the command: target, --junit, the gate's checks list in --json, init --platform gitlab, the action's pr-comment, the npm launcher and the container image.

Added

  • --cache-only on run and pairwise: every model call, the judge's included, is answered from the response cache, and the first reply that is not cached stops the command with exit 2 and names the case, the trial and the target; no model is called. It cannot be combined with --no-cache, and a python or http system is refused because its code is not cached. A run made this way records cache_only: "true" in its environment and does not query an Ollama server for versions and digests.
  • Run files record format_version (1). A file without it, written before 1.0, reads as 1. From 1.0 on, a file with a newer format than the installed evalproof reads is refused with exit 2 and an error giving the file's format version and the newest one this evalproof reads, and asking for an upgrade, instead of being read with the fields it does not know dropped. Releases before 1.0 do not check it.
  • fold_quotes: true on the regex scorer matches the output's typographic quotes, primes and fullwidth forms as ASCII ' and ", so an answer key written with plain quotes still matches. The patterns are used as written, and a suite that does not set it hashes as before. A value other than true or false stops the suite at load.
  • CacheMiss, ProviderError and RunFileError are exported from evalproof, so code that calls run_suite or load_run can catch them by name.
  • CI runs the test suite on macOS and Windows as well as Linux. On Windows the command line writes UTF-8 to a redirected stdout and stderr, and run files are written with LF line endings on every platform.
  • A versioning and support policy: what the public surface is, how deprecation works, and which versions get fixes.

  • On macOS the cursor: provider now runs cursor-agent inside sandbox-exec, with a profile that denies reading the folders where skills and rules live (~/.claude, ~/.agents, ~/.codex, ~/.gemini, and ~/.cursor/skills, skills-cursor, rules, plugins, agents), so the skills in your home directory no longer reach the prompt. The login is unaffected. EVALPROOF_CURSOR_SANDBOX=off turns it off (on is the default; any other value is an error), and it has no effect outside macOS or without /usr/bin/sandbox-exec. If sandbox-exec fails to start, the error points at that variable. Cursor User Rules come from Cursor's servers and are still sent: clear them in Cursor Settings > Rules.

  • evalproof doctor --cursor-canary [--model MODEL] makes one real cursor-agent call (it uses your Cursor quota; default model composer-2.5-fast) with a canary skill in the call's workspace and asks the model which skills and user-rule titles it can see. It reports whether the canary was seen, any other skill (a leak) and how many user rules were named (a count only: neither titles nor contents are printed). The answer is the model's own account, so it is best effort. Exit 0 when the canary was seen and nothing leaked, 1 otherwise, 4 when the call failed.
  • MEASUREMENTS.md section 10: the support Q&A and ticket-router cases on three hosted frontier models (claude-sonnet-5-5, gpt-6-astra, gemini-4-argon-medium), a 42- and 30-case sample, 558 calls. The regex answer key failed 12 correct answers and alone showed a 7.1-point drop of the same size on every model, from different cases; a reviewed key, written after reading the answers, removes it. Run files, both keys and measure.py, which re-scores every stored answer, are in examples/frontier-study/.
  • A command: system target runs any program once per case, in the suite's directory and without a shell: it reads the same JSON an http system is sent on stdin and prints plain text or a JSON result. A non-zero exit is the case's error with the end of stderr, and a timeout kills the program and everything it started. It is not cached, so --cache-only refuses it. See Command targets.
  • JUnit XML reports: run --junit PATH writes one test case per case (and trial), failed the same way as the report's "Failed cases"; compare --gate --junit PATH writes one test case per gate check, so GitLab, Jenkins and Azure DevOps show a failed gate as failed tests. --junit without --gate on compare is exit 2. See JUnit reports.
  • The gate's --json payload lists every check it evaluated in checks, passed ones included, each with a stable name (max_drop: <metric>, min_pass_rate: <metric>, must_pass, ...), an outcome (pass, fail or error) and a message.
  • A container image, ghcr.io/technobasant/evalproof, for linux/amd64 and linux/arm64, built from the released wheel with a provenance attestation and tagged with the version, the minor version and latest. It runs as a non-root user in /work and has no agent CLIs.
  • python -m evalproof runs the command line.
  • An npm launcher, npx evalproof, runs the PyPI release of the same version through uvx, then pipx, then an evalproof of that exact version on PATH; EVALPROOF_LAUNCHER picks one. It needs Python, and it is published only once npm trusted publishing is set up.
  • The GitHub Action's pr-comment: true posts the report as one pull-request comment and updates it on every push. A comment that cannot be posted is a warning; the gate's exit code is kept.
  • evalproof init --platform gitlab writes .gitlab-ci.evalproof.yml, a merge-request job that gates against the target branch and publishes the JUnit report, for you to include: from .gitlab-ci.yml.
  • scripts/test-in-docker.sh runs the test suite in Linux containers (Python 3.11 to 3.14, Alpine, and linux/amd64 under emulation) for contributors.

Changed

  • The power estimate beside the gate never reports a detectable drop smaller than the exact McNemar test can call significant. For a pass/fail metric compared with that test (one trial per case, or repeated trials combined with all or any) it needs k cases worse and none better (k = 6 at the gate's α of 0.05, whatever the number of gated metrics), so no drop under k / n is significant however the normal approximation reads: on 42 cases the figure is now 14.3 points. The figure and the cases needed are the larger of the two bounds; a uniform baseline reports the floor, and a suite with fewer than k cases is told no drop can be significant. The gate's verdict does not change.
  • When an agent CLI binary is not on PATH while a run is made, the run's environment still records the CLI, its isolation tag and its sampling, with cli_version: unavailable; those keys used to be missing.
  • HTTP clients that evalproof opens (the Ollama, Anthropic and OpenAI-compatible providers, the http system and the Ollama version lookup) are closed when the run or command ends.
  • Development status is Production/Stable.
  • The cursor isolation tag is cursor/3 when the sandbox is on and cursor/2 when it is not, so cached replies and compared runs never mix the two. The doctor warning about the skills list now depends on the sandbox: with it on, it names the blocked folders and the User Rules limit.
  • A rate-limited or failing model call is tried up to six times instead of four, waiting from about 1 second, doubling up to 30, with random jitter so parallel workers do not retry in step. A Retry-After header (seconds or a date) is waited instead, up to 60 seconds.
  • The run file name evalproof picks when there is no --out replaces characters Windows refuses in a file name (<>:"/\|?*) in the suite's name with -.

Fixed

  • Two workers saving the same reply to the response cache at once could make one of them fail its case; each now writes its own temporary file, and a cache entry that cannot be written or read is a miss rather than an error.
  • On Windows:
  • a timed-out agent CLI or command: program is ended with everything it started (taskkill /T), so a child left running no longer holds the output open;
  • the length limits for a prompt passed on the command line or in the environment follow Windows' (32,767 characters, 8,191 through a .cmd or .bat shim), with an error naming the limit;
  • the cursor sandbox profile and the codex instructions path use / separators, and a cursor workspace file named like /etc/x or C:\x is refused;
  • temporary agent workspaces are removed with retries while a just-killed child still holds a file;
  • every file evalproof writes (run files, --json, --junit, reports, datasets) has LF line endings;
  • suite files, datasets, prompt and rubric files and run files saved with a UTF-8 byte-order mark, as Notepad and PowerShell write them, are read.

0.8.0 — 2026-11-15

Importers and grading: bring results from garak, Google ADK, Inspect agent logs, promptfoo red-team runs and claude plugin eval into a run file, turn an ADK eval set into a dataset, and grade with rubric sets and the facts scorer.

Added

  • Rubric sets for the llm_judge scorer. Give it rubrics (strings, or {id, text, weight?, type?} objects) and, with case_rubrics: true, let each case add its own under rubrics. The judge grades every rubric on its own, and the score is the weighted fraction that passed; min_score sets where a case passes. mode: batched (the default) asks about every rubric in one call, per_rubric makes one call per rubric. num_samples asks the judge several times and a rubric passes by majority (a tie fails); each sample uses its own seed offset. rubric_types keeps only the listed types (untyped rubrics always apply). A suite with a single rubric or rubric_file behaves as before.
  • per_rubric_metrics: true on a rubric set also writes one <scorer>/<id> metric per base rubric. The gate, min_pass_rate, calibration and compare accept those names, so a change that breaks one rubric shows up on its own. The extra metrics carry no judge cost.
  • A scorer may define score_all(case, out) to write more than one metric. The runner merges the result; every extra name must be <scorer>/<id>, so it cannot collide with another scorer.
  • The facts scorer: numbers a response must contain (facts, each a number or {value, rel_tol?, abs_tol?, required?, label?}) and phrases it must mention (must_mention). The value is the share found. It is deterministic, so evalproof verify re-scores it, and readiness counts it as correctness.
  • readiness counts case tags stored in a run file (--run) as well as the dataset's, so an imported run with safety or robustness tags is covered without a dataset. The evidence names both sources.
  • evalproof.adapters.otel.from_spans (experimental) builds an agent's trace from OpenTelemetry spans, as OTLP/JSON, the collector's file exporter output, the Python SDK's span.to_json() objects or a list of spans, for the GenAI semantic conventions (also Google ADK and the Vercel AI SDK; ADK's extra execute_tool (merged) span for parallel calls is not a call), OpenInference and the Vercel legacy ai.toolCall spans. When the spans were exported with content capture off it warns, names the setting to turn on, and sets trace.meta["content_captured"] = False. See the OpenTelemetry tab of the adapters guide.
  • evalproof import garak reads the report.jsonl a garak scan writes (0.9 and later). Each attempt is a case tagged safety, garak and probe:<name>, its generations are repeats, and each detector is a metric that passes when the score is below the threshold. A report holds no tool calls, so no trace is imported.
  • evalproof import --threshold N sets the garak detector score at which an output counts as a hit. Without it the scan's run.eval_threshold is used, else 0.5. The run records environment.threshold and environment.threshold_source (cli, eval_threshold or default). Any other format refuses --threshold.
  • evalproof import adk (experimental) reads the *.evalset_result.json that adk eval saves, one case per eval_id with its runs as repeats, one metric per ADK metric. Both the current file and the double-encoded form older ADK versions wrote are read.
  • evalproof import claude-plugin-eval (experimental) reads the aggregate-result.json of claude plugin eval, with the with and without arms as two systems. A file marked partial records environment.partial and environment.partial_reason, and a run that skipped its paid graders records environment.skipped_paid_graders.
  • The adk and claude-plugin-eval importers print evalproof: the <format> importer is experimental once and set environment.import_experimental to true, because they were written from the tool's source or schema and no real result file was available.
  • evalproof import inspect imports the tool trace of each sample from its events: the calls with their arguments, results, errors and working time, the tokens of each model round and the last reply; the model and tool events of scorers (inside a scorers span) are not the agent's and are skipped. tool_called, tool_order and the other agent checks run on an imported log. Strings Inspect stored as attachment:// references are resolved. A condensed log has no events, and one warning says so. A case collapsed from several epochs keeps the trace of its first epoch that did not error, and the run records environment.trace_from as first repeat when that drops another epoch's trace; with --epochs trials every trial keeps its own.
  • evalproof import promptfoo tags red-team tests from their metadata (safety, redteam, plugin:<id>, strategy:<id> and severity:<level>) and reads the repeats of a test run with --repeat as one case, so --epochs trials works on a red-team file.
  • evalproof import-dataset adk FILE --out cases.jsonl turns a Google ADK eval set (*.evalset.json) or the flat legacy *.test.json list (one conversation, so one case) into a dataset that evalproof run reads. The first invocation is the case's input and the later ones are its turns, the last final response is the reference, session_input.state is the state, the tool calls of each invocation become one tool_order check with its turn, and rubrics are kept for the rubric-set judge. User-simulation (conversation_scenario) cases cannot be replayed, so they are skipped under one warning, and a file with no case to import is an error.

Changed

  • The parameters of an llm_judge scorer are checked when the suite loads: an unknown key (with a "did you mean" hint), a wrong type, mode or min_score out of range, rubric-set parameters without rubrics or case_rubrics: true, and rubric or rubric_file combined with a rubric set now fail at load. An unknown key used to fail only when the scorer was built.
  • With num_samples above 1 and a judge temperature of 0, loading the suite warns that the samples will agree.
  • Imported results now store each case's tags on its result, lower-cased and without duplicates (garak and promptfoo red-team tags, and a DeepEval test case's own). Tags label a case and are not part of its hash, so adding one does not unpair two runs.
  • Repeats are ordered by the source's own repeat index (promptfoo repeatIndex, Inspect epoch, ADK repetition_index) instead of the order of the file, so trial n is the n-th run.
  • Collapsing repeats sums the judge cost of every repeat, and a run that was a system error keeps the judge cost the file reported for it.
  • evalproof import adk refused an eval set already; its message now points to evalproof import-dataset adk.

0.7.1 — 2026-11-14

Documentation. No change to behaviour, the run file or the cache.

Documentation

  • The documentation site is rebuilt around four sections: Get started, Guides, Reference and About, each with an overview page, a home page, light and dark themes, "Last updated" dates and an edit link on every page. Fonts and images are served from the site itself, so a page makes no request to a third-party host. Every 0.7.0 URL still resolves.
  • New reference pages: every suite-file key with its type and default (Configuration), the run file format and what older files load as (Run file format), every exit code with its causes (Exit codes), and a glossary. Tests fail when a config field, a command option, a scorer or an exit code is missing from them.
  • New pages: Troubleshooting (symptom, cause and fix for the error messages evalproof prints), Upgrading (what to change from 0.5 onwards), Versioning and support, and Security and data handling (what is sent, what is stored, how credentials and coding-agent logins are handled).
  • The roadmap is current: importers, rubric sets and fact checks for 0.8, a coding-agent add-on for 0.9.
  • SECURITY.md lists 0.7.x as the supported line.

0.7.0 — 2026-11-14

Reliability: run each case several times, and compare and gate on what the trials show. The documentation moves to a site, basantbhattarai.com.np/evalproof.

Added

  • Repeated trials. trials: K in a suite, or evalproof run --trials K (1 to 100), runs each case K times and scores every trial on its own. A python system gets ctx.trial, an http system a trial field in the body (only when K is above 1), and a local model that honours a seed gets seed + trial. The response cache keeps trial 0 under the key it always had and keys later trials apart, and no provider is sent anything new. A run file from before trials loads as one trial per case, and a run with one trial per case is written, read and compared as before.
  • The run report for K > 1 shows the pooled pass rate with a case-resampling interval, pass@K, pass^K, the within-case spread of a continuous score and the flaky cases (a pass and a fail among a case's decided trials), and says when every case gave the same output on every trial. verify checks every trial of every case, rehash covers every trial, and calibration reads trial 0.
  • evalproof compare on runs with K > 1 aggregates each case's trials with --aggregate or gate.aggregate: mean (the pass rate, the default), all or any. mean and continuous scores are tested with a paired sign-flip permutation test, exact by dynamic programming when every case has the same number of decided trials and by enumeration up to 16 differing cases, otherwise 20 000 seeded draws (p = (hits + 1) / (draws + 1)); all and any use exact McNemar. The interval on the change is a paired bootstrap over cases. Runs with a different K cannot be compared (exit 4 under --gate, 2 without), and all or any on a gated continuous metric is refused.
  • The gate over trials: max_drop and alpha apply to the aggregated values; must_pass needs every trial of every tagged case to pass; min_pass_rate is pooled over trials; min_cases counts cases; budgets cover every trial; max_new_failures counts majority flips (a case the baseline passed in more than half its trials and the candidate did not). Flaky cases are listed in the compare report.
  • A power line beside the gate decision, for every run: the smallest drop the suite can detect at 80% power, and the cases needed to detect max_drop, from the spread of the paired differences with alpha divided across candidates and gated metrics. It is information only and never changes the status, a warning or an exit code. GateDecision.power carries it.
  • evalproof compare BASE CAND [CAND ...] compares several candidates against one baseline: one table, each candidate decided on its own, and one Holm correction over every candidate and gated metric. The exit code is the worst of the candidates (2, 4, 1, 3, 0). --json FILE holds {"baseline", "candidates": [{"label", "path", "comparison", "gate"}]}. One candidate is unchanged.
  • evalproof readiness reports reliability from the trials: covered with 3 or more per case, partial with 2, missing with 1; with --run it reads the run's own trials.
  • evalproof import --epochs trials keeps each repeat of a case as one of its trials.
  • examples/agent-tools/suite-flaky.yaml and a flaky_calendar variable on the example agent: a build that forgets the calendar on a fixed 40% of the outing trials (decided by random.Random(f"{case_id}:{trial}"), so the same on every machine). With --trials 5, v2 to flaky fails the gate (behaviour 100.0% to 90.9%, exact p = 0.0078, o01 and o05 regress by majority, all eight outing cases flaky) and v2 to v2 passes; with one trial per case the same build gives McNemar p = 0.25. CI runs both and checks readiness --run on the five trials.

Changed

  • Run.by_id() returns each case's trial 0 (the first, should a file hold two); it used to return the last result with that id. Run.by_case() returns every trial of each case.

Documentation

  • A documentation site, built with MkDocs Material from docs/ and published to GitHub Pages on each release: getting started, guides, a command and configuration reference, an API reference generated from the docstrings, the architecture, the decisions, the measured run and this changelog. The README is now the front page (install, what it needs to run, the quickstart and links into the site); its sections moved to the site unchanged. CI builds the site with --strict on every push.
  • A "Repeated trials" guide (trials, ctx.trial, the aggregates, pass@k and pass^k, the test used, majority flips, must_pass over every trial, power, and the limit that a deterministic system shows no variation), the several-candidate compare and the new gate.aggregate key. The agent example's README walks through the flaky build, and its readiness table no longer says repeated trials are missing from the release.

0.6.0 — 2026-11-14

Testing agents: assert on the tools an agent called and what it said, across several turns and several builds of the same system, and report which kinds of evidence a suite covers.

Breaking

  • The case fields turns, state and expect now have a meaning (the user turns that follow input, the initial session state, and the inline checks of the checks scorer) and are validated when the dataset loads: turns must be a list of strings or objects, state an object, expect a list of valid checks. A dataset that uses one of these names for something else fails to load with a message that says to rename the field.

Added

  • A checks scorer asserts on what an agent did: the tools it called and with which arguments (tool_called, tool_not_called, tool_args_contain, forbidden_arg_values, tool_order, tool_succeeded, tool_call_count, rounds), and what it said (output_contains, output_omits, output_regex, output_cites_tool_result), with any_of, not and python checks to combine them. Checks come from the suite (params.checks) and from each case (expect), are validated when the suite or the dataset loads, and are deterministic, so evalproof verify re-scores them.
  • A python scorer calls a function of your own, fn: module:function, which returns a bool, a number from 0 to 1, a (bool, reason) pair or a Score; an exception or any other return value leaves the case undecided. verify re-scores it only when the scorer sets deterministic: true.
  • evalproof.adapters turns what Google ADK, LangChain and LangGraph, the Claude Agent SDK and the OpenAI Agents SDK return into the trace a python system hands back, and TraceRecorder with mcp_call builds one by hand. No SDK is imported; tokens and cost roll up from the trace.
  • Suite variants run the same cases, scorers and gate against another model, prompt or flag: each has a name, optional vars and an optional system merged over the suite's. evalproof run --variant NAME (repeatable) or --all-variants writes runs/<variant>.json, and the run file records the variant. A python system whose function has a ctx parameter gets a RunContext (case_id, input, turns, state, variant, vars, trial).
  • Multi-turn cases run on every target that can hold a conversation: a python system gets the later turns as ctx.turns, an http system gets turns, state, variant and vars in the POST body, and a direct model target plays the turns in order, each after its previous reply. A coding-agent CLI takes one prompt per call, so a suite with turns cases on one is refused before any case runs.
  • evalproof readiness SUITE [--run FILE] [--json] [--require dim,...] reads a suite and its dataset, and optionally a run file, and reports for each dimension of quality (correctness, behaviour, quality, regression gate, reliability, cost and latency, safety, robustness, multi-turn, load, production monitoring) whether it is covered, partly covered, missing or out of scope, with the evidence. It makes no model call. It exits 1 when a --required dimension is not covered, and --json is for CI.
  • examples/agent-tools, an offline agent example: a plain-Python agent with two fake tools and no model, 35 cases with inline expect checks (three multi-turn, five tagged safety, five robustness), and two variants. v1 plants a bug that skips a tool for one intent, and only the checks scorer sees it, since the reply still reads well. The committed run files in runs/ gate v1 to v2 with exit 0 and v2 to v1 with exit 1, and evalproof readiness reports its gaps. CI runs both directions and the readiness --require on every commit.
  • The README has a "Testing agents" section: the trace contract (what a python system returns, the ToolCall fields and ctx), a table of every check with what it needs and when it passes, multi-turn cases (turns, state), variants (--variant, --all-variants), the example, and what readiness reports. "Agent SDK adapters" is now a subsection of it.
  • Credentials are stated plainly. The README has a "What it needs to run" section, a table of each target, the variable it reads, whether it works locally and in CI, and where it is allowed. evalproof init prints what the chosen judge needs, locally and in CI. evalproof init --ci api-key --cli-auth oauth-token writes a workflow that passes CLAUDE_CODE_OAUTH_TOKEN from a repository secret, for Claude Code only and for your own repository and CI only (a usage error for Codex or any other target). evalproof doctor prints, for each target, the credential it would use and whether that works in CI; it checks variable names only and never reads a value.

0.5.2 — 2026-11-14

No behaviour changes: documentation, and evalproof init now pins 0.5.2 in the workflows it writes.

Documentation

  • MEASUREMENTS §9 calibrates the support-qa judge against reviewed labels for a random 30 baseline cases (examples/support-qa/human-review/), next to the answer key on the same cases; measure.py prints the tables (#4). The README install section and a new use-cases section were rewritten, and PyPI shows the new README from this release.
  • CITATION.cff gives the current version again; it had stayed at 0.3.0.

0.5.1 — 2026-11-13

Upgrading: run files written by 0.5.0 keep working. They have no tag-free hash, so compare pairs them on the per-case hash as before, tags included; a run written by 0.5.1 pairs on the tag-free hash. The first time both the baseline and the candidate come from 0.5.1, adding a tag stops counting as an edit.

Fixed

  • Adding or changing a case's tags no longer counts as an edit in compare. Run files carry a new per-case content_hash, the case hash without its tags, and cases are paired on it when both runs have one (otherwise on case_hash, as in 0.5.0). Before, adding a must_pass tag excluded that case from the comparison until the baseline was refreshed. verify still counts a tag change, because must_pass reads tags from the run file, and its message now says so.

Added

  • evalproof verify scores the stored outputs again with the suite's scorers that need no model (exact_match, contains, regex, json_schema, numeric_tolerance, recall_at_k, mrr, ndcg) and fails, naming the cases, when a stored score differs: a score edited by hand, or made by another scorer version. It says which scorers it re-scored, or that none could be re-scored without a model call. Model-judged scores and the stored outputs themselves still cannot be checked.
  • evalproof rehash RUN SUITE [--out FILE]: adds per-case hashes to a run file written before 0.5, with no model call, so it can be gated against a changed dataset. It works when the cases the run scored are still the first ones in the suite's dataset, unchanged, which the dataset_hash stored in the run file proves: the dataset the run scored, the same with cases appended, or the dataset behind a --limit run. Otherwise it exits 4 and writes nothing. Scores, outputs and the suite hash are untouched, so verify still wants a run made with 0.5 or newer. The gate's exit-4 message for a baseline without hashes names the command.

0.5.0 — 2026-11-13

Breaking, in the order a user is likely to hit them:

  • Unknown keys in a suite are now errors (extra="forbid" on every config model). A typo such as gate.max_drops used to be ignored and now fails, naming the key and the closest valid one.
  • evalproof run exits 2 when every case errored. It still writes the run file and prints the report first. Before, it exited 0.
  • The cache keys of agent-CLI calls (claude-code, codex, opencode, cursor, copilot) change once: the CLI version is no longer part of the key, so a CLI that updates itself keeps its cached replies. Cached replies from 0.4.x are not reused; --no-cache always asks again.
  • evalproof compare --metric NAME exits 2 when neither run has a scorer called NAME. Before, it exited 4.
  • compare no longer exits 4 because the dataset changed, when both runs carry per-case hashes (runs written by 0.5 and later do). Edited cases are excluded and listed instead.
  • The action, given a suite, runs evalproof verify on the candidate first and fails with exit 4 when the committed run does not match the suite and its cases. A run file written by 0.4.x has no suite fingerprint and fails it: re-run the suite with 0.5 and commit the file, or set verify-candidate: "false".
  • A cost budget can trip where it did not: a claude-code system or judge with no pricing now records the CLI's reported total_cost_usd, and the judge's cost counts toward max_cost_usd, so a suite with gate.max_cost_usd can newly exit 3.

Upgrading: run the suite with 0.5 on the base branch and commit evals/runs/latest.json there first. A 0.4.x baseline has no per-case hashes, so a pull request that also changes the dataset cannot be gated (exit 4) until the base branch has a 0.5 run file. Tags are part of a case's hash, so adding a must_pass tag to a case counts as an edit until the baseline is refreshed.

Added

  • Per-case case_hash in run files (runner and importers). compare pairs cases by id and hash: a new case is not compared, an edited case is excluded and listed, so adding or editing cases no longer blocks the gate. A --limit run pairs with a full baseline.
  • evalproof verify RUN SUITE [--markdown FILE]: no model call; checks that a run file was made with the suite's system, judge, scorers and seed (RunInfo.suite_hash) and covers exactly the dataset's cases. Exit 0 on a match, 4 on a mismatch. The action runs it before comparing (input verify-candidate, default "true").
  • Gate checks that need no baseline: gate.min_pass_rate (a number or a dict by scorer), gate.must_pass (case tags) and gate.max_new_failures, with compare --min-pass-rate, --must-pass and --max-new-failures. evalproof run --gate applies the first two to a single run.
  • run and report end with a "Failed cases" list (--show N, default 10). compare lists the regressed ids of every compared metric and, with --show N (default 5), prints both answers and the candidate scorer's reason for that many regressed cases of the gated metrics.
  • ChatResponse.cost_usd: the claude-code provider reports the CLI's total_cost_usd, which the run records as the call's cost when the suite gives no pricing for that target. Under a subscription login it is an API-equivalent estimate, not a bill. The judge's cost is now computed from its pricing and counts toward max_cost_usd.
  • On a continuous gated metric, min_pass_rate cannot gate (exit 4, naming the fix), max_new_failures is skipped with a warning, and must_pass fails only an errored or undecided case, with a warning.
  • import --quiet keeps the report off stdout. compare and run --gate print a warning when a must_pass tag matches no case.
  • docs/specs/0.5.0-dev-workflow.md.

Changed

  • A new judge CLI version alone only warns (judge_cli_version); the gate still refuses when the resolved model, target, CLI or isolation changes.
  • The report says "cannot gate" for exit 4 instead of "fail", and lists up to ten regressed ids per metric (it listed twenty, for gated metrics only).
  • run --system-prompt-file F without --label labels the run <system> + F so two prompt files are told apart.
  • evalproof init --ci api-key writes a workflow with no paths: filter: a first step runs the suite only when WATCH (default evals/) changed, forks are skipped with a notice, and responses are cached in evals/.evalproof/cache with actions/cache. The keyless workflow names what verify checks and what it cannot prove.
  • An unexpected exception prints one line and a hint (EVALPROOF_DEBUG=1 shows the traceback) instead of a traceback. Ctrl-C exits 130. Label file errors in calibrate carry file:line.
  • The agent skill and the evals/README.md that init writes explain adding cases, reading a result, verify, run --gate, must_pass and max_new_failures.

Fixed

  • A --limit run, a stale run file and an edited case no longer pass the keyless gate unnoticed: verify fails the job.
  • A run whose --out could not be written failed after every case had run; the path is now checked before the first case.

0.4.1 — 2026-11-13

Fixed

  • cursor: ask mode is not tool-free: it has a read tool that runs without approval and reads files in the workspace and outside it. The workspace now carries a .cursor/cli.json that allows nothing and denies reads (verified inside and outside the workspace), writes and shell commands (ask mode has no write or shell tool; those entries are a second guard), and any tool_call event in the output fails the call as an isolation break.
  • cursor: wrote every call's chat to ~/.cursor/chats and read the user's ~/.cursor/cli-config.json. CURSOR_CONFIG_DIR and CURSOR_DATA_DIR now point into the call's temp root, which keeps the login (verified) and starts from a config that allows nothing.
  • copilot: kept every call's session in ~/.copilot and read the user's Copilot config. COPILOT_HOME now points into the call's temp root, which keeps the login (verified).
  • copilot: passed --available-tools=, which copilot 1.0.110 ignores, so the model could use its tools (it read a file). It now passes an allow-list with one unknown name, which disables every tool (verified), and any tool event or tool request in the output fails the call.

Changed

  • cursor: uses --output-format stream-json (the json result joined the model's preamble with the answer).
  • copilot: takes the prompt on stdin, prints JSONL, runs with --no-custom-instructions and --disable-builtin-mcps, reports real token counts from --usage-output-file, and records the model that answered each call: with copilot:auto Copilot routes every call, so the run artifact shows the routed model, not auto.
  • Isolation tags are now cursor/2 and copilot/2, so earlier cached replies for these two are not reused.
  • evalproof doctor warns that cursor-agent sends the names and descriptions of the user's skills (~/.claude/skills, ~/.cursor/skills and others) in every prompt, which no option turns off, and that copilot:auto as the judge makes compare and the gate refuse runs as a judge change (Copilot picks the model per call).

Documentation

  • The experimental providers section now says what was verified and what was not: Cursor reads instruction files from the directories above the workspace; Cursor and Copilot keep their own system prompt; Copilot's --no-custom-instructions was verified with canary files; the Copilot tool guard has not fired live; named-model availability and the premium-request rate depend on plan and policy; with tools off Copilot may write a made-up tool call as text.

0.4.0 — 2026-11-13

Added

  • claude-code:<model> and codex:<model> providers: run your own logged-in Claude Code or Codex CLI as a judge or a system under test, with no API key. Each call runs in a fresh empty directory with the agent's tools, settings, MCP servers, skills and memory switched off and no permission-bypass flag; API-key variables are removed from the child's environment unless api_key_env names one or EVALPROOF_CLI_AUTH=api-key keeps them. Neither CLI takes a temperature, seed or token limit, so those params are ignored (Codex maps think to its reasoning effort). Claude Code runs with no tools; Codex still offers a JavaScript isolate, sub-agents and a question to the user, so any item in its output other than a message, reasoning, an error or a plan fails the call as an isolation break, and a codex too old for the flags evalproof sets gets a message to upgrade. ADR 0004 has the reasoning.
  • EVALPROOF_CLI_AUTH (login, the default, or api-key) chooses whether a CLI provider uses the subscription login or its own API key variable; evalproof doctor prints the mode in effect and warns when api-key has no key to use. api_key_env still wins.
  • evalproof doctor [suite.yaml]: reports each CLI's path, version, login mode and the isolation flags it will use, without a model call, and exits 1 when a target of the suite is not usable. Without a suite it surveys the installed CLIs and exits 1 only when none is usable. It prints no account details.
  • Run facts for CLI targets in runner.environment: {judge,system}_cli, {judge,system}_cli_version, {judge,system}_isolation and {judge,system}_model_resolved (the model ids that answered, sorted, cached replies included). The response cache keys on the CLI fingerprint, so a CLI upgrade is a cache miss.
  • compare reports same_judge, judge_changes and system_changes, and the gate refuses to decide (exit 4) when the judge changed between the runs. --allow-judge-change or gate.allow_judge_change: true accepts the change and keeps a warning.
  • Experimental opencode:<provider>/<model>, cursor:<model> and copilot:<model> providers, on the same footing as the CLI providers above (fresh empty directory, the CLI's key variable removed unless api_key_env names one, bounded runtime, CLI fingerprint with the isolation tags opencode/0, cursor/0, copilot/0). Each prints one warning on stderr the first time it is used. Only OpenCode was exercised for real, and only against a local ollama model; Cursor (not logged in) and Copilot (not installed) follow their documentation, and the README's "Experimental providers" says what was and was not verified. evalproof doctor without a suite now also lists whichever of these CLIs are on PATH.
  • evalproof.providers.TransientError, a ProviderError that with_retries retries; the CLI providers raise it for rate limits and server errors.
  • tests/test_live_agent_cli.py: one real call per installed CLI, skipped unless EVALPROOF_LIVE=1.
  • evalproof init [--dir PATH] [--judge TARGET] [--ci keyless|api-key|none] [--no-skill] [--force]: writes evals/suite.yaml, three sample cases in evals/cases.jsonl, an evals/README.md, a GitHub Actions workflow at .github/workflows/evalproof.yml and an agent skill at .agents/skills/evalproof/SKILL.md and .claude/skills/evalproof/SKILL.md. The judge is the first of claude, codex, ollama found on PATH, else a hosted Anthropic model. The keyless workflow makes no model calls and gates the committed evals/runs/latest.json against the base branch; the api-key workflow runs the suite in CI (with EVALPROOF_CLI_AUTH: api-key for a CLI judge, so no second suite file is written) and pins the judge CLI to the version found at init time. Existing files are kept unless --force is passed, nothing is written outside --dir, and no model is called.

Changed

  • A gate that passed in 0.3.0 can now exit 4 when the judge's target, CLI version, isolation or resolved model differs between two runs that both carry these keys. Runs written before this release carry none and compare as before.
  • A changed system target or CLI version between two runs now adds a warning, so the status is warn instead of pass (the exit code stays 0).
  • runner.environment records judge_target and system_target for every model target (not only CLI ones), and the system target it records is the one in effect, with any override applied.
  • The gate's "too few paired cases" reason now says how to fix it (add cases, or lower gate.min_cases).

0.3.0 — 2026-11-12

Added

  • evalproof import and evalproof.import_run: per-case results from promptfoo (JSON and JSONL), Inspect (.json and .eval logs), DeepEval, lm-evaluation-harness (samples_*.jsonl) and generic long or wide jsonl and csv files become a run that compare and the gate read. Case ids are stable across runs, system errors score as failures, repeats collapse by --epochs, and NaN or infinite scores are refused with the case and metric named. Raises evalproof.ImportFormatError (a ValueError; exit 2 on the command line).
  • Gating several metrics: gate.metrics in the suite or --metric repeated on compare, with Holm-adjusted p-values (evalproof.holm). The decision carries adjusted_p, the report gains a Holm-adjusted p column, and a single metric behaves as before. ADR 0003 has the reasoning.
  • The composite action's metric input takes several names, separated by newlines or commas.
  • examples/retrieval: an offline retrieval example for recall@k, MRR and nDCG, with a regression that recall@5 cannot see and MRR and nDCG@5 can. Contributed by @HarshRajSinghania in #5. CI asserts its gate exits 1 on the regression and 0 on no change.
  • MEASUREMENTS §8: the support-qa judge with thinking on (#3), with its runs, suite and analysis in measure.py. Against labels audited case by case it passes no wrong answer and κ rises, at 6.7× the judge time; the gate's decision does not change.
  • A provider profile table (providers/profiles.py) behind one OpenAI-compatible adapter: openai, azure, gemini, mistral, groq, together, fireworks, deepseek, xai, openrouter, cerebras, the local servers vllm, lmstudio and llamacpp, and openai-compatible for any other server (set base_url). Each profile carries its base URL, key variable, auth header, token-limit field and a link to the provider's docs. OpenAIProvider stays importable.
  • TargetConfig.api_key_env names the environment variable that holds the key, and TargetConfig.headers adds request headers to every HTTP provider; suite headers win over a provider's own, and neither is part of the cache key or written to a run. A named key variable that is not set is an error rather than a 401.
  • Anthropic: a refusal stop reason is recorded as a case error naming its category, and so is max_tokens reached before any text, with a hint to raise params.max_tokens. The OpenAI-compatible adapter does the same for finish_reason length with no text, and for content_filter, and reads replies that arrive as a list of typed content parts.
  • A test pins the cache key of an existing Ollama call, so a change to it cannot slip in.

Changed

  • Hosted providers (everything except Ollama and the local servers) send temperature and seed only when the suite sets them. Before, openai sent temperature: 0 and seed: 0 on every call, which reasoning models reject; now an unset value is left to the provider's default. A hosted run that relied on the implicit temperature: 0 should set it in params. Ollama and the local servers still send both, so local runs repeat as before and keep their cache keys; GenerationParams defaults are unchanged. A hosted call that now leaves a value out gets a new cache key, so a reply cached at the old implicit temperature: 0 is not reused for it. To point openai at an older local server through base_url, use openai-compatible or a local profile instead, which send both values.
  • openai sends the token limit as max_completion_tokens, which OpenAI's reasoning models require (they reject max_tokens), and mistral sends its seed as random_seed. The default limit is still 512.
  • The llama.cpp profile defaults to port 9931, the one its server documents now; for an older build on 8080, set base_url.

Fixed

  • Anthropic calls to current Claude models failed with HTTP 400, because temperature was sent on every request. It goes out only when the suite sets it, and no seed is sent, since the Messages API has none.
  • A gate built without validation and given no metric passed by gating nothing; it now cannot gate (exit 4), and compare --metric "" is a usage error (exit 2).
  • Found in review before release, in the new importers: two promptfoo prompts whose first 60 characters matched merged into one system (the label is no longer truncated, and two prompts that share a label are an error), promptfoo output with repeated tests but no repeatIndex now warns that it reads as separate cases, Inspect message ids no longer change the dataset hash, a null promptfoo testIdx no longer crashes, and under --epochs any or mean a case with one good repeat is no longer marked as an error.
  • examples/support-qa/cases.jsonl: the q21 and q22 references named Enterprise, which the product facts do not give SSO or an admin console (#7). The original set is kept as cases.v1.jsonl so the recorded runs stay reproducible; the runs judged on the revised set are in runs/v2/, and MEASUREMENTS §8 reports both.

0.2.1 — 2026-10-16

Fixed

  • The PyPI and Python badges on the 0.2.0 PyPI page read "package or version not found". PyPI's image proxy fetched them about a minute after the first upload, before the badge service could see the package, and keeps that answer for 12 to 24 hours. The PyPI page now leaves those two badges out, since it shows the version and supported Pythons itself, and the README's copies use new URLs so no cache serves the old images. CI fails if a PyPI-data badge reaches the PyPI description again.

0.2.0 — 2026-10-16

The first release on PyPI: pip install evalproof.

Added

  • A supported Python API: from evalproof import compare_runs, decide, calibrate, .... The names in evalproof.__all__ are the public surface; other modules are internal.
  • Python 3.14 support, tested in CI alongside 3.11 to 3.13.
  • Releases publish to PyPI from GitHub Actions with trusted publishing, so no token is stored anywhere. Each file on PyPI carries a PEP 740 attestation, and the GitHub release carries the same files with a build-provenance attestation.
  • SECURITY.md with private vulnerability reporting and the trust boundaries, backed by a test that runs both hosted providers with a placeholder key and finds it in no file written.
  • CODE_OF_CONDUCT.md (Contributor Covenant 2.1), issue and pull-request templates, CITATION.cff, CODEOWNERS, Dependabot for actions and Python dependencies with a three-day cooldown, and a pre-commit configuration.

Changed

  • The version is written once, in src/evalproof/_version.py; the build reads it.
  • The README published to PyPI rewrites relative links to GitHub and replaces the Mermaid diagram with a link, so the PyPI page has no broken links. CI checks this on every commit, along with twine check --strict and a clean install of the built wheel.
  • CI pins every action to a commit SHA, checks out without persisted credentials, sets timeouts and cancels superseded pull-request runs, and audits the workflows with zizmor. The composite action pins setup-uv the same way.
  • Package metadata: documentation, source, changelog and live-demo links; classifiers for typing, Python 3.14 and beta status.

0.1.2 — 2026-10-15

Added

  • app/: an interactive Streamlit app, live at https://evalproof.streamlit.app. Check a drop starts from four scenarios (the recorded run, the same rates on 4× the cases, a small but certain drop, a twenty-case suite) or from sliders, and shows the gate's verdict, every paired case as a grid, the coin-flip distribution behind the exact McNemar p-value, and the suite size at which the same drop would become significant. The recorded run replays the committed examples/support-qa runs: select any of the twelve changed answers to read both versions side by side, add a latency budget, and calibrate the judge on a gold sample of chosen size. It runs the library from src/ through the same compare_runs and decide calls as the CLI; CI installs the app's own requirements and drives every scenario.

Fixed

  • evalproof compare raised OverflowError when a metric had more than about 1,030 discordant pairs: the exact McNemar test summed comb(n, i) * 0.5**n in floating point, and comb(n, i) outgrows the largest float. The binomial tail is now summed as integers and divided once, so it stays exact at any size.

0.1.1 — 2026-10-13

Fixed

  • The Wilson interval now returns exactly 0 at k=0 and exactly 1 at k=n. Computing the boundaries by subtraction left platform-dependent rounding (2.8e-17 on x86_64 Linux, 0.0 on arm64 macOS).

Changed

  • The three-day exclude-newer resolution guard is declared in [tool.uv], so uv.lock means the same thing on every machine and in CI.
  • CI pins uv to the version that wrote uv.lock.

0.1.0 — 2026-10-13

First release.

Added

  • Suites in YAML: a JSONL golden set, a system under test, scorers, an optional judge, calibration settings and a gate.
  • Systems under test: Ollama, Anthropic and OpenAI models with a prompt template; Python functions; HTTP endpoints. System failures are recorded per case, never fatal.
  • Scorers: exact match, contains, regex with accept/reject answer keys, a JSON Schema subset, numeric tolerance, recall@k, MRR, nDCG, and an LLM judge with a strict JSON verdict, one parse retry and an explicit undecided state.
  • Pairwise judging with order swap; position-inconsistent verdicts become ties.
  • Judge calibration: confusion matrix, TPR, FPR, Cohen's κ, and prevalence-corrected pass rates by adjusted classify-and-count (bootstrap interval) and prediction-powered inference (normal interval).
  • Paired comparison of two runs: exact McNemar for pass/fail, paired bootstrap for scores.
  • A CI gate that fails only on a drop that is both beyond max_drop and significant at alpha, plus p95-latency and cost budgets, with distinct exit codes.
  • Content-addressed response cache; runs record the Ollama version and model digests.
  • Markdown reports for GitHub step summaries, JSON reports, and a composite GitHub Action.
  • Two examples: an offline ticket router (CI's gate contract) and a product support assistant judged by a local model, with measurements.