Changelog¶
All notable changes are listed here. The project follows Semantic Versioning.
Unreleased¶
1.4.0 — 2026-11-15¶
Added¶
- Adapters for LlamaIndex, CrewAI, Pydantic AI, DSPy, Haystack, AutoGen and Semantic Kernel turn a run of each framework into a trace (tool calls, tokens), next to the adapters for LangChain, ADK, the Claude Agent SDK and the OpenAI Agents SDK. They read each framework's objects without importing it. They were written from each SDK's source and tested with fake objects of those shapes, not yet against real runs, and are experimental: they warn when used. See Agent SDK adapters.
- The
evalproofnpm package is now a helper library for a JavaScript or TypeScript system as well as thenpxlauncher.runCommand(handler)runs acommand:target: it reads the case from stdin, awaits the handler (which returns a string or{output, retrieved, trace, tokens_in, tokens_out, cost_usd}) and prints the result as JSON, withconsole.logsent to stderr.serve(handler, {port, host, path})serves thehttp://target protocol withnode:http, on127.0.0.1by default.fromVercelAI(result)andfromLangChain(messages)build a trace (tool calls, tokens) from an AI SDKgenerateTextresult or a LangChain.js agent's messages. It ships TypeScript types, works from ESM and CommonJS, has no dependencies and needs Node 18 or newer. evalproof init --nodewritesevals/system.mjs, a program that importsrunCommand, and a suite whose target iscommand:node system.mjs. With--ci api-keyon GitHub the workflow sets up Node and runsnpm ci. It cannot be combined with--agent-files-only, and it is refused with--platform gitlaborbitbucketand--ci api-key, whose job image has no Node.evalproof scanfinds Vercel AI SDK tools and agents and LangChain.js tools and agents in TypeScript and JavaScript files, and follows a suite whose target is anode,tsx(ornpx tsx),bunordenocommand through the relative imports of its script. TypeScript code now takes part in theuncovered_codeandtool_uncheckedgaps. An import it cannot follow (an unresolved relative path, a path alias, a workspace package, a module loaded by a computed name) or a script it cannot find leaves reachability unknown anduncovered_codenot computed. Test files and files that only import an SDK are not reported as untested code.- A guide, Testing a TypeScript or JavaScript system.
The helper library, init --node and the scan of TypeScript tools and agents are experimental:
they can change in a minor release. The converters print one line to stderr the first time they
are called.
Changed¶
scanreads a multi-lineimport { ... } from "..."and skips comments in TypeScript and JavaScript files, and no longer reports "imports X; no call found" for a LangChain.js file that defines tools.evalproof draftreads only Python files for a function's signature and docstring.
1.3.0 — 2026-11-15¶
Added¶
evalproof scan [PATH]reads a repository without running it or calling a model (astfor Python, patterns for TypeScript and JavaScript) and reports its model calls, agents, tools, prompts, multi-turn use, test-case files and evalproof suites. It then lists the gaps: no suite, code no suite's target reaches (followed through the repository's own imports from apython:target), tools no check names, no multi-turn case, pytest files that are not evalproof cases yet, and suites with fewer cases than their gate needs. Each gap comes with the command that drafts for it.--jsonand--outwrite the report as JSON. It always exits 0: gaps are advice, not a gate.evalproof draft --scan scan.json --out evals/drafts.jsonlturns those gaps into a separate JSONL of draft cases, never a suite's dataset. Cases from pytest files keep the assertion their test already made (==,in,re.search) and call no model. For the other gaps a model writes inputs only, taggeddraft,gap:<kind>andfrom:model, withdraft_foranddrafted_by: never an expected answer or a check. A person adds those before a draft goes into a dataset.- A guide, Find what is not tested.
evalproof init --platform bitbucketwrites a Bitbucket Pipelines file with a pull-request step that runs and gates (--ci api-key) or verifies and gates the committed run (--ci keyless). An existingbitbucket-pipelines.ymlthat init did not write is never edited: init writesevals/bitbucket-pipelines.snippet.ymlto merge by hand.evalproof hook installwrites a git pre-push hook that runsevalproof verifyon the committed run file, with no model call;evalproof hook uninstallremoves it. It followscore.hooksPathand never replaces a hook it did not write without--force.evalproof run --tagsand--exclude-tags(also onpairwise) run only the cases with those tags. The filter is recorded in the run file (tag_filter), andverifychecks the run against the cases the same filter selects now.
scan, draft and init --platform bitbucket are experimental: they can change in a minor
release.
Changed¶
compare --gaterefuses (exit 4) two runs made with different tag filters, because a filter does not show in a pull request's diff and the gate would decide on a subset of the cases. Without--gate,comparereports the shared cases with a warning.
1.2.0 — 2026-11-15¶
Added¶
evalproof import-datasetreads the test cases you already have in another tool and writes an evalproof dataset: promptfoo (apromptfooconfig.yaml, or its tests as YAML, JSON, JSONL or CSV), DeepEval, Ragas, LangSmith, OpenAI Evals, Inspect, Braintrust, Langfuse, Phoenix, ADK, and any JSONL or CSV table (--id-field,--input-field,--reference-field,--tags-field).autoreads the format off the file and says which it chose; an ambiguous or unknown file exits 2.- What the source expected becomes a case field and a scorer. promptfoo's
equals,contains/icontains(and their-any/-allforms),starts-with,regex, thenot-forms,is-jsonwith a schema,llm-rubric,g-eval,model-graded-closedqaandfactualitymap toexact_match,contains,regex,checks,json_schemaand per-case rubrics. A free-text expected answer becomes the case'sreference, graded byllm_judge. An assertion with no counterpart (javascript,python,similar,latency,cost, ...) is never dropped silently: it stays on the case underunmapped, and one line counts them by type. - Imported cases keep the source's id, or get
<format>-and a hash of the input and the expected answer, so reordering the file never changes an id. They are taggedimportedandfrom:<format>. import-dataset --suite-out FILEalso writes a suite with exactly the scorers those cases need, ajudgeblock only when one of them is a judge, and aprompt_templatewhen the input is an object of variables. The gate is set on the scorer that checks the most cases. It refuses to overwrite a file without--force, and writes nothing when no scorer fits.evalproof init --from FILE(--from-format, defaultauto) starts a repository from those cases instead of the three sample cases.- A guide, Import test cases.
The deepeval, ragas, langsmith, openai-evals, inspect, braintrust, langfuse and
phoenix dataset formats are experimental: they were written from each tool's documented fields
and have not yet been checked against files exported by each tool.
Changed¶
import-dataset adkalso accepts--suite-out. The dataset it writes is unchanged.
1.1.0 — 2026-11-15¶
Added¶
evalproof initadds an evalproof section to the repository'sAGENTS.md, so any coding agent that reads that file knows which command to run for a task, how to read the exit codes, what never to weaken and how each kind of provider and target is set up. The section sits between<!-- evalproof:begin -->and<!-- evalproof:end -->. A missing file is created, a file without the markers gets the section appended after one blank line, and a file that has them gets only the text between them replaced; everything else in the file stays byte for byte, line endings included. Damaged markers (one without the other, repeated, or in the wrong order) stopinitwith exit 2 and nothing is written.--forcedoes not overwrite the file.--no-agents-mdskips it.evalproof init --agent-files-onlywrites only the agent files (theAGENTS.mdsection and the skill) and no suite or CI workflow. It cannot be combined with--judge,--ci,--platformor--cli-auth. Existing skill copies are kept unless--forceis given.- A guide, Coding agents.
- An
AGENTS.mdat the root of this repository for agents that work on evalproof itself.
The text of the AGENTS.md section, the skill's text and --agent-files-only are experimental
and can change in a minor release. The markers and the rule that init never changes anything
outside them are stable.
Changed¶
- The skill points to the
AGENTS.mdsection for every task other than checking a change. - The npm package is published by the release workflow through npm trusted publishing, with a provenance statement, instead of by hand.
1.0.0 — 2026-11-15¶
The first stable release. The public surface (the names in evalproof.__all__ and
evalproof.adapters.__all__, commands, flags, exit codes, --json fields, suite keys, the run
file format and dataset fields) now changes incompatibly only in a major release, after at least
one minor release of deprecation warnings. Experimental features stay outside that promise. See
Versioning and support and
Upgrading. Nothing from 0.8 is
removed or renamed.
These are experimental, outside that promise until they have been used in more projects: the
command: target, --junit, the gate's checks list in --json, init --platform gitlab, the
action's pr-comment, the npm launcher and the container image.
Added¶
--cache-onlyonrunandpairwise: every model call, the judge's included, is answered from the response cache, and the first reply that is not cached stops the command with exit 2 and names the case, the trial and the target; no model is called. It cannot be combined with--no-cache, and apythonorhttpsystem is refused because its code is not cached. A run made this way recordscache_only: "true"in its environment and does not query an Ollama server for versions and digests.- Run files record
format_version(1). A file without it, written before 1.0, reads as 1. From 1.0 on, a file with a newer format than the installed evalproof reads is refused with exit 2 and an error giving the file's format version and the newest one this evalproof reads, and asking for an upgrade, instead of being read with the fields it does not know dropped. Releases before 1.0 do not check it. fold_quotes: trueon theregexscorer matches the output's typographic quotes, primes and fullwidth forms as ASCII'and", so an answer key written with plain quotes still matches. The patterns are used as written, and a suite that does not set it hashes as before. A value other than true or false stops the suite at load.CacheMiss,ProviderErrorandRunFileErrorare exported fromevalproof, so code that callsrun_suiteorload_runcan catch them by name.- CI runs the test suite on macOS and Windows as well as Linux. On Windows the command line writes UTF-8 to a redirected stdout and stderr, and run files are written with LF line endings on every platform.
-
A versioning and support policy: what the public surface is, how deprecation works, and which versions get fixes.
-
On macOS the
cursor:provider now runscursor-agentinsidesandbox-exec, with a profile that denies reading the folders where skills and rules live (~/.claude,~/.agents,~/.codex,~/.gemini, and~/.cursor/skills,skills-cursor,rules,plugins,agents), so the skills in your home directory no longer reach the prompt. The login is unaffected.EVALPROOF_CURSOR_SANDBOX=offturns it off (onis the default; any other value is an error), and it has no effect outside macOS or without/usr/bin/sandbox-exec. Ifsandbox-execfails to start, the error points at that variable. Cursor User Rules come from Cursor's servers and are still sent: clear them in Cursor Settings > Rules. evalproof doctor --cursor-canary [--model MODEL]makes one realcursor-agentcall (it uses your Cursor quota; default modelcomposer-2.5-fast) with a canary skill in the call's workspace and asks the model which skills and user-rule titles it can see. It reports whether the canary was seen, any other skill (a leak) and how many user rules were named (a count only: neither titles nor contents are printed). The answer is the model's own account, so it is best effort. Exit 0 when the canary was seen and nothing leaked, 1 otherwise, 4 when the call failed.- MEASUREMENTS.md section 10: the support Q&A
and ticket-router cases on three hosted frontier models (
claude-sonnet-5-5,gpt-6-astra,gemini-4-argon-medium), a 42- and 30-case sample, 558 calls. The regex answer key failed 12 correct answers and alone showed a 7.1-point drop of the same size on every model, from different cases; a reviewed key, written after reading the answers, removes it. Run files, both keys andmeasure.py, which re-scores every stored answer, are inexamples/frontier-study/. - A
command:system target runs any program once per case, in the suite's directory and without a shell: it reads the same JSON anhttpsystem is sent on stdin and prints plain text or a JSON result. A non-zero exit is the case's error with the end of stderr, and a timeout kills the program and everything it started. It is not cached, so--cache-onlyrefuses it. See Command targets. - JUnit XML reports:
run --junit PATHwrites one test case per case (and trial), failed the same way as the report's "Failed cases";compare --gate --junit PATHwrites one test case per gate check, so GitLab, Jenkins and Azure DevOps show a failed gate as failed tests.--junitwithout--gateoncompareis exit 2. See JUnit reports. - The gate's
--jsonpayload lists every check it evaluated inchecks, passed ones included, each with a stable name (max_drop: <metric>,min_pass_rate: <metric>,must_pass, ...), an outcome (pass,failorerror) and a message. - A container image,
ghcr.io/technobasant/evalproof, forlinux/amd64andlinux/arm64, built from the released wheel with a provenance attestation and tagged with the version, the minor version andlatest. It runs as a non-root user in/workand has no agent CLIs. python -m evalproofruns the command line.- An npm launcher,
npx evalproof, runs the PyPI release of the same version throughuvx, thenpipx, then anevalproofof that exact version onPATH;EVALPROOF_LAUNCHERpicks one. It needs Python, and it is published only once npm trusted publishing is set up. - The GitHub Action's
pr-comment: trueposts the report as one pull-request comment and updates it on every push. A comment that cannot be posted is a warning; the gate's exit code is kept. evalproof init --platform gitlabwrites.gitlab-ci.evalproof.yml, a merge-request job that gates against the target branch and publishes the JUnit report, for you toinclude:from.gitlab-ci.yml.scripts/test-in-docker.shruns the test suite in Linux containers (Python 3.11 to 3.14, Alpine, andlinux/amd64under emulation) for contributors.
Changed¶
- The power estimate beside the gate never reports a detectable drop smaller than the exact
McNemar test can call significant. For a pass/fail metric compared with that test (one trial
per case, or repeated trials combined with
allorany) it needskcases worse and none better (k= 6 at the gate's α of 0.05, whatever the number of gated metrics), so no drop underk / nis significant however the normal approximation reads: on 42 cases the figure is now 14.3 points. The figure and the cases needed are the larger of the two bounds; a uniform baseline reports the floor, and a suite with fewer thankcases is told no drop can be significant. The gate's verdict does not change. - When an agent CLI binary is not on
PATHwhile a run is made, the run's environment still records the CLI, its isolation tag and its sampling, withcli_version: unavailable; those keys used to be missing. - HTTP clients that evalproof opens (the Ollama, Anthropic and OpenAI-compatible providers, the
httpsystem and the Ollama version lookup) are closed when the run or command ends. - Development status is Production/Stable.
- The cursor isolation tag is
cursor/3when the sandbox is on andcursor/2when it is not, so cached replies and compared runs never mix the two. Thedoctorwarning about the skills list now depends on the sandbox: with it on, it names the blocked folders and the User Rules limit. - A rate-limited or failing model call is tried up to six times instead of four, waiting from
about 1 second, doubling up to 30, with random jitter so parallel workers do not retry in step.
A
Retry-Afterheader (seconds or a date) is waited instead, up to 60 seconds. - The run file name evalproof picks when there is no
--outreplaces characters Windows refuses in a file name (<>:"/\|?*) in the suite's name with-.
Fixed¶
- Two workers saving the same reply to the response cache at once could make one of them fail its case; each now writes its own temporary file, and a cache entry that cannot be written or read is a miss rather than an error.
- On Windows:
- a timed-out agent CLI or
command:program is ended with everything it started (taskkill /T), so a child left running no longer holds the output open; - the length limits for a prompt passed on the command line or in the environment follow
Windows' (32,767 characters, 8,191 through a
.cmdor.batshim), with an error naming the limit; - the cursor sandbox profile and the codex instructions path use
/separators, and a cursor workspace file named like/etc/xorC:\xis refused; - temporary agent workspaces are removed with retries while a just-killed child still holds a file;
- every file evalproof writes (run files,
--json,--junit, reports, datasets) has LF line endings; - suite files, datasets, prompt and rubric files and run files saved with a UTF-8 byte-order mark, as Notepad and PowerShell write them, are read.
0.8.0 — 2026-11-15¶
Importers and grading: bring results from garak, Google ADK, Inspect agent logs, promptfoo
red-team runs and claude plugin eval into a run file, turn an ADK eval set into a dataset, and
grade with rubric sets and the facts scorer.
Added¶
- Rubric sets for the
llm_judgescorer. Give itrubrics(strings, or{id, text, weight?, type?}objects) and, withcase_rubrics: true, let each case add its own underrubrics. The judge grades every rubric on its own, and the score is the weighted fraction that passed;min_scoresets where a case passes.mode: batched(the default) asks about every rubric in one call,per_rubricmakes one call per rubric.num_samplesasks the judge several times and a rubric passes by majority (a tie fails); each sample uses its own seed offset.rubric_typeskeeps only the listed types (untyped rubrics always apply). A suite with a singlerubricorrubric_filebehaves as before. per_rubric_metrics: trueon a rubric set also writes one<scorer>/<id>metric per base rubric. The gate,min_pass_rate, calibration andcompareaccept those names, so a change that breaks one rubric shows up on its own. The extra metrics carry no judge cost.- A scorer may define
score_all(case, out)to write more than one metric. The runner merges the result; every extra name must be<scorer>/<id>, so it cannot collide with another scorer. - The
factsscorer: numbers a response must contain (facts, each a number or{value, rel_tol?, abs_tol?, required?, label?}) and phrases it must mention (must_mention). The value is the share found. It is deterministic, soevalproof verifyre-scores it, andreadinesscounts it as correctness. readinesscounts case tags stored in a run file (--run) as well as the dataset's, so an imported run withsafetyorrobustnesstags is covered without a dataset. The evidence names both sources.evalproof.adapters.otel.from_spans(experimental) builds an agent's trace from OpenTelemetry spans, as OTLP/JSON, the collector's file exporter output, the Python SDK'sspan.to_json()objects or a list of spans, for the GenAI semantic conventions (also Google ADK and the Vercel AI SDK; ADK's extraexecute_tool (merged)span for parallel calls is not a call), OpenInference and the Vercel legacyai.toolCallspans. When the spans were exported with content capture off it warns, names the setting to turn on, and setstrace.meta["content_captured"] = False. See the OpenTelemetry tab of the adapters guide.evalproof import garakreads thereport.jsonla garak scan writes (0.9 and later). Each attempt is a case taggedsafety,garakandprobe:<name>, its generations are repeats, and each detector is a metric that passes when the score is below the threshold. A report holds no tool calls, so no trace is imported.evalproof import --threshold Nsets the garak detector score at which an output counts as a hit. Without it the scan'srun.eval_thresholdis used, else 0.5. The run recordsenvironment.thresholdandenvironment.threshold_source(cli,eval_thresholdordefault). Any other format refuses--threshold.evalproof import adk(experimental) reads the*.evalset_result.jsonthatadk evalsaves, one case pereval_idwith its runs as repeats, one metric per ADK metric. Both the current file and the double-encoded form older ADK versions wrote are read.evalproof import claude-plugin-eval(experimental) reads theaggregate-result.jsonofclaude plugin eval, with the with and without arms as two systems. A file marked partial recordsenvironment.partialandenvironment.partial_reason, and a run that skipped its paid graders recordsenvironment.skipped_paid_graders.- The
adkandclaude-plugin-evalimporters printevalproof: the <format> importer is experimentalonce and setenvironment.import_experimentaltotrue, because they were written from the tool's source or schema and no real result file was available. evalproof import inspectimports the tool trace of each sample from itsevents: the calls with their arguments, results, errors and working time, the tokens of each model round and the last reply; the model and tool events of scorers (inside ascorersspan) are not the agent's and are skipped.tool_called,tool_orderand the other agent checks run on an imported log. Strings Inspect stored asattachment://references are resolved. A condensed log has no events, and one warning says so. A case collapsed from several epochs keeps the trace of its first epoch that did not error, and the run recordsenvironment.trace_fromasfirst repeatwhen that drops another epoch's trace; with--epochs trialsevery trial keeps its own.evalproof import promptfootags red-team tests from their metadata (safety,redteam,plugin:<id>,strategy:<id>andseverity:<level>) and reads the repeats of a test run with--repeatas one case, so--epochs trialsworks on a red-team file.evalproof import-dataset adk FILE --out cases.jsonlturns a Google ADK eval set (*.evalset.json) or the flat legacy*.test.jsonlist (one conversation, so one case) into a dataset thatevalproof runreads. The first invocation is the case'sinputand the later ones are itsturns, the last final response is thereference,session_input.stateis thestate, the tool calls of each invocation become onetool_ordercheck with its turn, andrubricsare kept for the rubric-set judge. User-simulation (conversation_scenario) cases cannot be replayed, so they are skipped under one warning, and a file with no case to import is an error.
Changed¶
- The parameters of an
llm_judgescorer are checked when the suite loads: an unknown key (with a "did you mean" hint), a wrong type,modeormin_scoreout of range, rubric-set parameters withoutrubricsorcase_rubrics: true, andrubricorrubric_filecombined with a rubric set now fail at load. An unknown key used to fail only when the scorer was built. - With
num_samplesabove 1 and a judgetemperatureof 0, loading the suite warns that the samples will agree. - Imported results now store each case's tags on its result, lower-cased and without duplicates (garak and promptfoo red-team tags, and a DeepEval test case's own). Tags label a case and are not part of its hash, so adding one does not unpair two runs.
- Repeats are ordered by the source's own repeat index (promptfoo
repeatIndex, Inspectepoch, ADKrepetition_index) instead of the order of the file, so trialnis then-th run. - Collapsing repeats sums the judge cost of every repeat, and a run that was a system error keeps the judge cost the file reported for it.
evalproof import adkrefused an eval set already; its message now points toevalproof import-dataset adk.
0.7.1 — 2026-11-14¶
Documentation. No change to behaviour, the run file or the cache.
Documentation¶
- The documentation site is rebuilt around four sections: Get started, Guides, Reference and About, each with an overview page, a home page, light and dark themes, "Last updated" dates and an edit link on every page. Fonts and images are served from the site itself, so a page makes no request to a third-party host. Every 0.7.0 URL still resolves.
- New reference pages: every suite-file key with its type and default (Configuration), the run file format and what older files load as (Run file format), every exit code with its causes (Exit codes), and a glossary. Tests fail when a config field, a command option, a scorer or an exit code is missing from them.
- New pages: Troubleshooting (symptom, cause and fix for the error messages evalproof prints), Upgrading (what to change from 0.5 onwards), Versioning and support, and Security and data handling (what is sent, what is stored, how credentials and coding-agent logins are handled).
- The roadmap is current: importers, rubric sets and fact checks for 0.8, a coding-agent add-on for 0.9.
- SECURITY.md lists 0.7.x as the supported line.
0.7.0 — 2026-11-14¶
Reliability: run each case several times, and compare and gate on what the trials show. The documentation moves to a site, basantbhattarai.com.np/evalproof.
Added¶
- Repeated trials.
trials: Kin a suite, orevalproof run --trials K(1 to 100), runs each case K times and scores every trial on its own. A python system getsctx.trial, an http system atrialfield in the body (only when K is above 1), and a local model that honours a seed getsseed + trial. The response cache keeps trial 0 under the key it always had and keys later trials apart, and no provider is sent anything new. A run file from before trials loads as one trial per case, and a run with one trial per case is written, read and compared as before. - The run report for K > 1 shows the pooled pass rate with a case-resampling interval,
pass@K,pass^K, the within-case spread of a continuous score and the flaky cases (a pass and a fail among a case's decided trials), and says when every case gave the same output on every trial.verifychecks every trial of every case,rehashcovers every trial, and calibration reads trial 0. evalproof compareon runs with K > 1 aggregates each case's trials with--aggregateorgate.aggregate:mean(the pass rate, the default),allorany.meanand continuous scores are tested with a paired sign-flip permutation test, exact by dynamic programming when every case has the same number of decided trials and by enumeration up to 16 differing cases, otherwise 20 000 seeded draws (p = (hits + 1) / (draws + 1));allandanyuse exact McNemar. The interval on the change is a paired bootstrap over cases. Runs with a different K cannot be compared (exit 4 under--gate, 2 without), andalloranyon a gated continuous metric is refused.- The gate over trials:
max_dropandalphaapply to the aggregated values;must_passneeds every trial of every tagged case to pass;min_pass_rateis pooled over trials;min_casescounts cases; budgets cover every trial;max_new_failurescounts majority flips (a case the baseline passed in more than half its trials and the candidate did not). Flaky cases are listed in the compare report. - A power line beside the gate decision, for every run: the smallest drop the suite can detect
at 80% power, and the cases needed to detect
max_drop, from the spread of the paired differences withalphadivided across candidates and gated metrics. It is information only and never changes the status, a warning or an exit code.GateDecision.powercarries it. evalproof compare BASE CAND [CAND ...]compares several candidates against one baseline: one table, each candidate decided on its own, and one Holm correction over every candidate and gated metric. The exit code is the worst of the candidates (2, 4, 1, 3, 0).--json FILEholds{"baseline", "candidates": [{"label", "path", "comparison", "gate"}]}. One candidate is unchanged.evalproof readinessreports reliability from the trials: covered with 3 or more per case, partial with 2, missing with 1; with--runit reads the run's own trials.evalproof import --epochs trialskeeps each repeat of a case as one of its trials.examples/agent-tools/suite-flaky.yamland aflaky_calendarvariable on the example agent: a build that forgets the calendar on a fixed 40% of the outing trials (decided byrandom.Random(f"{case_id}:{trial}"), so the same on every machine). With--trials 5,v2toflakyfails the gate (behaviour 100.0% to 90.9%, exact p = 0.0078,o01ando05regress by majority, all eight outing cases flaky) andv2tov2passes; with one trial per case the same build gives McNemar p = 0.25. CI runs both and checksreadiness --runon the five trials.
Changed¶
Run.by_id()returns each case's trial 0 (the first, should a file hold two); it used to return the last result with that id.Run.by_case()returns every trial of each case.
Documentation¶
- A documentation site, built with MkDocs Material from
docs/and published to GitHub Pages on each release: getting started, guides, a command and configuration reference, an API reference generated from the docstrings, the architecture, the decisions, the measured run and this changelog. The README is now the front page (install, what it needs to run, the quickstart and links into the site); its sections moved to the site unchanged. CI builds the site with--stricton every push. - A "Repeated trials" guide (trials,
ctx.trial, the aggregates, pass@k and pass^k, the test used, majority flips,must_passover every trial, power, and the limit that a deterministic system shows no variation), the several-candidatecompareand the newgate.aggregatekey. The agent example's README walks through the flaky build, and its readiness table no longer says repeated trials are missing from the release.
0.6.0 — 2026-11-14¶
Testing agents: assert on the tools an agent called and what it said, across several turns and several builds of the same system, and report which kinds of evidence a suite covers.
Breaking¶
- The case fields
turns,stateandexpectnow have a meaning (the user turns that followinput, the initial session state, and the inline checks of thechecksscorer) and are validated when the dataset loads:turnsmust be a list of strings or objects,statean object,expecta list of valid checks. A dataset that uses one of these names for something else fails to load with a message that says to rename the field.
Added¶
- A
checksscorer asserts on what an agent did: the tools it called and with which arguments (tool_called,tool_not_called,tool_args_contain,forbidden_arg_values,tool_order,tool_succeeded,tool_call_count,rounds), and what it said (output_contains,output_omits,output_regex,output_cites_tool_result), withany_of,notandpythonchecks to combine them. Checks come from the suite (params.checks) and from each case (expect), are validated when the suite or the dataset loads, and are deterministic, soevalproof verifyre-scores them. - A
pythonscorer calls a function of your own,fn: module:function, which returns a bool, a number from 0 to 1, a(bool, reason)pair or aScore; an exception or any other return value leaves the case undecided.verifyre-scores it only when the scorer setsdeterministic: true. evalproof.adaptersturns what Google ADK, LangChain and LangGraph, the Claude Agent SDK and the OpenAI Agents SDK return into the trace a python system hands back, andTraceRecorderwithmcp_callbuilds one by hand. No SDK is imported; tokens and cost roll up from the trace.- Suite
variantsrun the same cases, scorers and gate against another model, prompt or flag: each has aname, optionalvarsand an optionalsystemmerged over the suite's.evalproof run --variant NAME(repeatable) or--all-variantswritesruns/<variant>.json, and the run file records the variant. A python system whose function has actxparameter gets aRunContext(case_id,input,turns,state,variant,vars,trial). - Multi-turn cases run on every target that can hold a conversation: a python system gets the
later turns as
ctx.turns, an http system getsturns,state,variantandvarsin the POST body, and a direct model target plays the turns in order, each after its previous reply. A coding-agent CLI takes one prompt per call, so a suite withturnscases on one is refused before any case runs. evalproof readiness SUITE [--run FILE] [--json] [--require dim,...]reads a suite and its dataset, and optionally a run file, and reports for each dimension of quality (correctness, behaviour, quality, regression gate, reliability, cost and latency, safety, robustness, multi-turn, load, production monitoring) whether it is covered, partly covered, missing or out of scope, with the evidence. It makes no model call. It exits 1 when a--required dimension is not covered, and--jsonis for CI.examples/agent-tools, an offline agent example: a plain-Python agent with two fake tools and no model, 35 cases with inlineexpectchecks (three multi-turn, five taggedsafety, fiverobustness), and two variants.v1plants a bug that skips a tool for one intent, and only thechecksscorer sees it, since the reply still reads well. The committed run files inruns/gatev1tov2with exit 0 andv2tov1with exit 1, andevalproof readinessreports its gaps. CI runs both directions and the readiness--requireon every commit.- The README has a "Testing agents" section: the trace contract (what a python system returns,
the
ToolCallfields andctx), a table of every check with what it needs and when it passes, multi-turn cases (turns,state), variants (--variant,--all-variants), the example, and whatreadinessreports. "Agent SDK adapters" is now a subsection of it. - Credentials are stated plainly. The README has a "What it needs to run" section, a table of
each target, the variable it reads, whether it works locally and in CI, and where it is allowed.
evalproof initprints what the chosen judge needs, locally and in CI.evalproof init --ci api-key --cli-auth oauth-tokenwrites a workflow that passesCLAUDE_CODE_OAUTH_TOKENfrom a repository secret, for Claude Code only and for your own repository and CI only (a usage error for Codex or any other target).evalproof doctorprints, for each target, the credential it would use and whether that works in CI; it checks variable names only and never reads a value.
0.5.2 — 2026-11-14¶
No behaviour changes: documentation, and evalproof init now pins 0.5.2 in the workflows it writes.
Documentation¶
- MEASUREMENTS §9 calibrates the support-qa judge against reviewed labels for a random 30
baseline cases (
examples/support-qa/human-review/), next to the answer key on the same cases;measure.pyprints the tables (#4). The README install section and a new use-cases section were rewritten, and PyPI shows the new README from this release. CITATION.cffgives the current version again; it had stayed at 0.3.0.
0.5.1 — 2026-11-13¶
Upgrading: run files written by 0.5.0 keep working. They have no tag-free hash, so compare
pairs them on the per-case hash as before, tags included; a run written by 0.5.1 pairs on the
tag-free hash. The first time both the baseline and the candidate come from 0.5.1, adding a tag
stops counting as an edit.
Fixed¶
- Adding or changing a case's tags no longer counts as an edit in
compare. Run files carry a new per-casecontent_hash, the case hash without its tags, and cases are paired on it when both runs have one (otherwise oncase_hash, as in 0.5.0). Before, adding amust_passtag excluded that case from the comparison until the baseline was refreshed.verifystill counts a tag change, becausemust_passreads tags from the run file, and its message now says so.
Added¶
evalproof verifyscores the stored outputs again with the suite's scorers that need no model (exact_match,contains,regex,json_schema,numeric_tolerance,recall_at_k,mrr,ndcg) and fails, naming the cases, when a stored score differs: a score edited by hand, or made by another scorer version. It says which scorers it re-scored, or that none could be re-scored without a model call. Model-judged scores and the stored outputs themselves still cannot be checked.evalproof rehash RUN SUITE [--out FILE]: adds per-case hashes to a run file written before 0.5, with no model call, so it can be gated against a changed dataset. It works when the cases the run scored are still the first ones in the suite's dataset, unchanged, which thedataset_hashstored in the run file proves: the dataset the run scored, the same with cases appended, or the dataset behind a--limitrun. Otherwise it exits 4 and writes nothing. Scores, outputs and the suite hash are untouched, soverifystill wants a run made with 0.5 or newer. The gate's exit-4 message for a baseline without hashes names the command.
0.5.0 — 2026-11-13¶
Breaking, in the order a user is likely to hit them:
- Unknown keys in a suite are now errors (
extra="forbid"on every config model). A typo such asgate.max_dropsused to be ignored and now fails, naming the key and the closest valid one. evalproof runexits 2 when every case errored. It still writes the run file and prints the report first. Before, it exited 0.- The cache keys of agent-CLI calls (
claude-code,codex,opencode,cursor,copilot) change once: the CLI version is no longer part of the key, so a CLI that updates itself keeps its cached replies. Cached replies from 0.4.x are not reused;--no-cachealways asks again. evalproof compare --metric NAMEexits 2 when neither run has a scorer calledNAME. Before, it exited 4.compareno longer exits 4 because the dataset changed, when both runs carry per-case hashes (runs written by 0.5 and later do). Edited cases are excluded and listed instead.- The action, given a
suite, runsevalproof verifyon the candidate first and fails with exit 4 when the committed run does not match the suite and its cases. A run file written by 0.4.x has no suite fingerprint and fails it: re-run the suite with 0.5 and commit the file, or setverify-candidate: "false". - A cost budget can trip where it did not: a
claude-codesystem or judge with nopricingnow records the CLI's reportedtotal_cost_usd, and the judge's cost counts towardmax_cost_usd, so a suite withgate.max_cost_usdcan newly exit 3.
Upgrading: run the suite with 0.5 on the base branch and commit evals/runs/latest.json there
first. A 0.4.x baseline has no per-case hashes, so a pull request that also changes the dataset
cannot be gated (exit 4) until the base branch has a 0.5 run file. Tags are part of a case's
hash, so adding a must_pass tag to a case counts as an edit until the baseline is refreshed.
Added¶
- Per-case
case_hashin run files (runner and importers).comparepairs cases by id and hash: a new case is not compared, an edited case is excluded and listed, so adding or editing cases no longer blocks the gate. A--limitrun pairs with a full baseline. evalproof verify RUN SUITE [--markdown FILE]: no model call; checks that a run file was made with the suite's system, judge, scorers and seed (RunInfo.suite_hash) and covers exactly the dataset's cases. Exit 0 on a match, 4 on a mismatch. The action runs it before comparing (inputverify-candidate, default"true").- Gate checks that need no baseline:
gate.min_pass_rate(a number or a dict by scorer),gate.must_pass(case tags) andgate.max_new_failures, withcompare --min-pass-rate,--must-passand--max-new-failures.evalproof run --gateapplies the first two to a single run. runandreportend with a "Failed cases" list (--show N, default 10).comparelists the regressed ids of every compared metric and, with--show N(default 5), prints both answers and the candidate scorer's reason for that many regressed cases of the gated metrics.ChatResponse.cost_usd: theclaude-codeprovider reports the CLI'stotal_cost_usd, which the run records as the call's cost when the suite gives no pricing for that target. Under a subscription login it is an API-equivalent estimate, not a bill. The judge's cost is now computed from itspricingand counts towardmax_cost_usd.- On a continuous gated metric,
min_pass_ratecannot gate (exit 4, naming the fix),max_new_failuresis skipped with a warning, andmust_passfails only an errored or undecided case, with a warning. import --quietkeeps the report off stdout.compareandrun --gateprint a warning when amust_passtag matches no case.docs/specs/0.5.0-dev-workflow.md.
Changed¶
- A new judge CLI version alone only warns (
judge_cli_version); the gate still refuses when the resolved model, target, CLI or isolation changes. - The report says "cannot gate" for exit 4 instead of "fail", and lists up to ten regressed ids per metric (it listed twenty, for gated metrics only).
run --system-prompt-file Fwithout--labellabels the run<system> + Fso two prompt files are told apart.evalproof init --ci api-keywrites a workflow with nopaths:filter: a first step runs the suite only whenWATCH(defaultevals/) changed, forks are skipped with a notice, and responses are cached inevals/.evalproof/cachewithactions/cache. The keyless workflow names whatverifychecks and what it cannot prove.- An unexpected exception prints one line and a hint (
EVALPROOF_DEBUG=1shows the traceback) instead of a traceback. Ctrl-C exits 130. Label file errors incalibratecarryfile:line. - The agent skill and the
evals/README.mdthatinitwrites explain adding cases, reading a result,verify,run --gate,must_passandmax_new_failures.
Fixed¶
- A
--limitrun, a stale run file and an edited case no longer pass the keyless gate unnoticed:verifyfails the job. - A run whose
--outcould not be written failed after every case had run; the path is now checked before the first case.
0.4.1 — 2026-11-13¶
Fixed¶
cursor:ask mode is not tool-free: it has a read tool that runs without approval and reads files in the workspace and outside it. The workspace now carries a.cursor/cli.jsonthat allows nothing and denies reads (verified inside and outside the workspace), writes and shell commands (ask mode has no write or shell tool; those entries are a second guard), and anytool_callevent in the output fails the call as an isolation break.cursor:wrote every call's chat to~/.cursor/chatsand read the user's~/.cursor/cli-config.json.CURSOR_CONFIG_DIRandCURSOR_DATA_DIRnow point into the call's temp root, which keeps the login (verified) and starts from a config that allows nothing.copilot:kept every call's session in~/.copilotand read the user's Copilot config.COPILOT_HOMEnow points into the call's temp root, which keeps the login (verified).copilot:passed--available-tools=, which copilot 1.0.110 ignores, so the model could use its tools (it read a file). It now passes an allow-list with one unknown name, which disables every tool (verified), and any tool event or tool request in the output fails the call.
Changed¶
cursor:uses--output-format stream-json(the json result joined the model's preamble with the answer).copilot:takes the prompt on stdin, prints JSONL, runs with--no-custom-instructionsand--disable-builtin-mcps, reports real token counts from--usage-output-file, and records the model that answered each call: withcopilot:autoCopilot routes every call, so the run artifact shows the routed model, notauto.- Isolation tags are now
cursor/2andcopilot/2, so earlier cached replies for these two are not reused. evalproof doctorwarns thatcursor-agentsends the names and descriptions of the user's skills (~/.claude/skills,~/.cursor/skillsand others) in every prompt, which no option turns off, and thatcopilot:autoas the judge makes compare and the gate refuse runs as a judge change (Copilot picks the model per call).
Documentation¶
- The experimental providers section now says what was verified and what was not: Cursor
reads instruction files from the directories above the workspace; Cursor and Copilot keep
their own system prompt; Copilot's
--no-custom-instructionswas verified with canary files; the Copilot tool guard has not fired live; named-model availability and the premium-request rate depend on plan and policy; with tools off Copilot may write a made-up tool call as text.
0.4.0 — 2026-11-13¶
Added¶
claude-code:<model>andcodex:<model>providers: run your own logged-in Claude Code or Codex CLI as a judge or a system under test, with no API key. Each call runs in a fresh empty directory with the agent's tools, settings, MCP servers, skills and memory switched off and no permission-bypass flag; API-key variables are removed from the child's environment unlessapi_key_envnames one orEVALPROOF_CLI_AUTH=api-keykeeps them. Neither CLI takes a temperature, seed or token limit, so those params are ignored (Codex mapsthinkto its reasoning effort). Claude Code runs with no tools; Codex still offers a JavaScript isolate, sub-agents and a question to the user, so any item in its output other than a message, reasoning, an error or a plan fails the call as an isolation break, and a codex too old for the flags evalproof sets gets a message to upgrade. ADR 0004 has the reasoning.EVALPROOF_CLI_AUTH(login, the default, orapi-key) chooses whether a CLI provider uses the subscription login or its own API key variable;evalproof doctorprints the mode in effect and warns whenapi-keyhas no key to use.api_key_envstill wins.evalproof doctor [suite.yaml]: reports each CLI's path, version, login mode and the isolation flags it will use, without a model call, and exits 1 when a target of the suite is not usable. Without a suite it surveys the installed CLIs and exits 1 only when none is usable. It prints no account details.- Run facts for CLI targets in
runner.environment:{judge,system}_cli,{judge,system}_cli_version,{judge,system}_isolationand{judge,system}_model_resolved(the model ids that answered, sorted, cached replies included). The response cache keys on the CLI fingerprint, so a CLI upgrade is a cache miss. comparereportssame_judge,judge_changesandsystem_changes, and the gate refuses to decide (exit 4) when the judge changed between the runs.--allow-judge-changeorgate.allow_judge_change: trueaccepts the change and keeps a warning.- Experimental
opencode:<provider>/<model>,cursor:<model>andcopilot:<model>providers, on the same footing as the CLI providers above (fresh empty directory, the CLI's key variable removed unlessapi_key_envnames one, bounded runtime, CLI fingerprint with the isolation tagsopencode/0,cursor/0,copilot/0). Each prints one warning on stderr the first time it is used. Only OpenCode was exercised for real, and only against a local ollama model; Cursor (not logged in) and Copilot (not installed) follow their documentation, and the README's "Experimental providers" says what was and was not verified.evalproof doctorwithout a suite now also lists whichever of these CLIs are onPATH. evalproof.providers.TransientError, aProviderErrorthatwith_retriesretries; the CLI providers raise it for rate limits and server errors.tests/test_live_agent_cli.py: one real call per installed CLI, skipped unlessEVALPROOF_LIVE=1.evalproof init [--dir PATH] [--judge TARGET] [--ci keyless|api-key|none] [--no-skill] [--force]: writesevals/suite.yaml, three sample cases inevals/cases.jsonl, anevals/README.md, a GitHub Actions workflow at.github/workflows/evalproof.ymland an agent skill at.agents/skills/evalproof/SKILL.mdand.claude/skills/evalproof/SKILL.md. The judge is the first ofclaude,codex,ollamafound onPATH, else a hosted Anthropic model. The keyless workflow makes no model calls and gates the committedevals/runs/latest.jsonagainst the base branch; the api-key workflow runs the suite in CI (withEVALPROOF_CLI_AUTH: api-keyfor a CLI judge, so no second suite file is written) and pins the judge CLI to the version found at init time. Existing files are kept unless--forceis passed, nothing is written outside--dir, and no model is called.
Changed¶
- A gate that passed in 0.3.0 can now exit 4 when the judge's target, CLI version, isolation or resolved model differs between two runs that both carry these keys. Runs written before this release carry none and compare as before.
- A changed system target or CLI version between two runs now adds a warning, so the status
is
warninstead ofpass(the exit code stays 0). runner.environmentrecordsjudge_targetandsystem_targetfor every model target (not only CLI ones), and the system target it records is the one in effect, with any override applied.- The gate's "too few paired cases" reason now says how to fix it (add cases, or lower
gate.min_cases).
0.3.0 — 2026-11-12¶
Added¶
evalproof importandevalproof.import_run: per-case results from promptfoo (JSON and JSONL), Inspect (.jsonand.evallogs), DeepEval, lm-evaluation-harness (samples_*.jsonl) and generic long or widejsonlandcsvfiles become a run thatcompareand the gate read. Case ids are stable across runs, system errors score as failures, repeats collapse by--epochs, and NaN or infinite scores are refused with the case and metric named. Raisesevalproof.ImportFormatError(aValueError; exit 2 on the command line).- Gating several metrics:
gate.metricsin the suite or--metricrepeated oncompare, with Holm-adjusted p-values (evalproof.holm). The decision carriesadjusted_p, the report gains a Holm-adjusted p column, and a single metric behaves as before. ADR 0003 has the reasoning. - The composite action's
metricinput takes several names, separated by newlines or commas. examples/retrieval: an offline retrieval example for recall@k, MRR and nDCG, with a regression that recall@5 cannot see and MRR and nDCG@5 can. Contributed by @HarshRajSinghania in #5. CI asserts its gate exits 1 on the regression and 0 on no change.- MEASUREMENTS §8: the support-qa judge with thinking on (#3), with its runs, suite and
analysis in
measure.py. Against labels audited case by case it passes no wrong answer and κ rises, at 6.7× the judge time; the gate's decision does not change. - A provider profile table (
providers/profiles.py) behind one OpenAI-compatible adapter:openai,azure,gemini,mistral,groq,together,fireworks,deepseek,xai,openrouter,cerebras, the local serversvllm,lmstudioandllamacpp, andopenai-compatiblefor any other server (setbase_url). Each profile carries its base URL, key variable, auth header, token-limit field and a link to the provider's docs.OpenAIProviderstays importable. TargetConfig.api_key_envnames the environment variable that holds the key, andTargetConfig.headersadds request headers to every HTTP provider; suite headers win over a provider's own, and neither is part of the cache key or written to a run. A named key variable that is not set is an error rather than a 401.- Anthropic: a
refusalstop reason is recorded as a case error naming its category, and so ismax_tokensreached before any text, with a hint to raiseparams.max_tokens. The OpenAI-compatible adapter does the same forfinish_reasonlengthwith no text, and forcontent_filter, and reads replies that arrive as a list of typed content parts. - A test pins the cache key of an existing Ollama call, so a change to it cannot slip in.
Changed¶
- Hosted providers (everything except Ollama and the local servers) send
temperatureandseedonly when the suite sets them. Before,openaisenttemperature: 0andseed: 0on every call, which reasoning models reject; now an unset value is left to the provider's default. A hosted run that relied on the implicittemperature: 0should set it inparams. Ollama and the local servers still send both, so local runs repeat as before and keep their cache keys;GenerationParamsdefaults are unchanged. A hosted call that now leaves a value out gets a new cache key, so a reply cached at the old implicittemperature: 0is not reused for it. To pointopenaiat an older local server throughbase_url, useopenai-compatibleor a local profile instead, which send both values. openaisends the token limit asmax_completion_tokens, which OpenAI's reasoning models require (they rejectmax_tokens), andmistralsends its seed asrandom_seed. The default limit is still 512.- The llama.cpp profile defaults to port 9931, the one its server documents now; for an older
build on 8080, set
base_url.
Fixed¶
- Anthropic calls to current Claude models failed with HTTP 400, because
temperaturewas sent on every request. It goes out only when the suite sets it, and no seed is sent, since the Messages API has none. - A gate built without validation and given no metric passed by gating nothing; it now
cannot gate (exit 4), and
compare --metric ""is a usage error (exit 2). - Found in review before release, in the new importers: two promptfoo prompts whose first 60
characters matched merged into one system (the label is no longer truncated, and two
prompts that share a label are an error), promptfoo output with repeated tests but no
repeatIndexnow warns that it reads as separate cases, Inspect message ids no longer change the dataset hash, anullpromptfootestIdxno longer crashes, and under--epochs anyormeana case with one good repeat is no longer marked as an error. examples/support-qa/cases.jsonl: the q21 and q22 references named Enterprise, which the product facts do not give SSO or an admin console (#7). The original set is kept ascases.v1.jsonlso the recorded runs stay reproducible; the runs judged on the revised set are inruns/v2/, and MEASUREMENTS §8 reports both.
0.2.1 — 2026-10-16¶
Fixed¶
- The PyPI and Python badges on the 0.2.0 PyPI page read "package or version not found". PyPI's image proxy fetched them about a minute after the first upload, before the badge service could see the package, and keeps that answer for 12 to 24 hours. The PyPI page now leaves those two badges out, since it shows the version and supported Pythons itself, and the README's copies use new URLs so no cache serves the old images. CI fails if a PyPI-data badge reaches the PyPI description again.
0.2.0 — 2026-10-16¶
The first release on PyPI: pip install evalproof.
Added¶
- A supported Python API:
from evalproof import compare_runs, decide, calibrate, .... The names inevalproof.__all__are the public surface; other modules are internal. - Python 3.14 support, tested in CI alongside 3.11 to 3.13.
- Releases publish to PyPI from GitHub Actions with trusted publishing, so no token is stored anywhere. Each file on PyPI carries a PEP 740 attestation, and the GitHub release carries the same files with a build-provenance attestation.
SECURITY.mdwith private vulnerability reporting and the trust boundaries, backed by a test that runs both hosted providers with a placeholder key and finds it in no file written.CODE_OF_CONDUCT.md(Contributor Covenant 2.1), issue and pull-request templates,CITATION.cff,CODEOWNERS, Dependabot for actions and Python dependencies with a three-day cooldown, and a pre-commit configuration.
Changed¶
- The version is written once, in
src/evalproof/_version.py; the build reads it. - The README published to PyPI rewrites relative links to GitHub and replaces the Mermaid
diagram with a link, so the PyPI page has no broken links. CI checks this on every
commit, along with
twine check --strictand a clean install of the built wheel. - CI pins every action to a commit SHA, checks out without persisted credentials, sets
timeouts and cancels superseded pull-request runs, and audits the workflows with zizmor.
The composite action pins
setup-uvthe same way. - Package metadata: documentation, source, changelog and live-demo links; classifiers for typing, Python 3.14 and beta status.
0.1.2 — 2026-10-15¶
Added¶
app/: an interactive Streamlit app, live at https://evalproof.streamlit.app. Check a drop starts from four scenarios (the recorded run, the same rates on 4× the cases, a small but certain drop, a twenty-case suite) or from sliders, and shows the gate's verdict, every paired case as a grid, the coin-flip distribution behind the exact McNemar p-value, and the suite size at which the same drop would become significant. The recorded run replays the committedexamples/support-qaruns: select any of the twelve changed answers to read both versions side by side, add a latency budget, and calibrate the judge on a gold sample of chosen size. It runs the library fromsrc/through the samecompare_runsanddecidecalls as the CLI; CI installs the app's own requirements and drives every scenario.
Fixed¶
evalproof compareraised OverflowError when a metric had more than about 1,030 discordant pairs: the exact McNemar test summedcomb(n, i) * 0.5**nin floating point, andcomb(n, i)outgrows the largest float. The binomial tail is now summed as integers and divided once, so it stays exact at any size.
0.1.1 — 2026-10-13¶
Fixed¶
- The Wilson interval now returns exactly 0 at k=0 and exactly 1 at k=n. Computing the boundaries by subtraction left platform-dependent rounding (2.8e-17 on x86_64 Linux, 0.0 on arm64 macOS).
Changed¶
- The three-day
exclude-newerresolution guard is declared in[tool.uv], souv.lockmeans the same thing on every machine and in CI. - CI pins uv to the version that wrote
uv.lock.
0.1.0 — 2026-10-13¶
First release.
Added¶
- Suites in YAML: a JSONL golden set, a system under test, scorers, an optional judge, calibration settings and a gate.
- Systems under test: Ollama, Anthropic and OpenAI models with a prompt template; Python functions; HTTP endpoints. System failures are recorded per case, never fatal.
- Scorers: exact match, contains, regex with accept/reject answer keys, a JSON Schema subset, numeric tolerance, recall@k, MRR, nDCG, and an LLM judge with a strict JSON verdict, one parse retry and an explicit undecided state.
- Pairwise judging with order swap; position-inconsistent verdicts become ties.
- Judge calibration: confusion matrix, TPR, FPR, Cohen's κ, and prevalence-corrected pass rates by adjusted classify-and-count (bootstrap interval) and prediction-powered inference (normal interval).
- Paired comparison of two runs: exact McNemar for pass/fail, paired bootstrap for scores.
- A CI gate that fails only on a drop that is both beyond
max_dropand significant atalpha, plus p95-latency and cost budgets, with distinct exit codes. - Content-addressed response cache; runs record the Ollama version and model digests.
- Markdown reports for GitHub step summaries, JSON reports, and a composite GitHub Action.
- Two examples: an offline ticket router (CI's gate contract) and a product support assistant judged by a local model, with measurements.