Skip to content

In CI

The composite action runs compare --gate, appends the report to the job summary, and fails the job on exit codes 1, 3 or 4. This page is for anyone wiring the gate into GitHub Actions, GitLab CI or Bitbucket Pipelines, or checking the committed run file before every push.

The action

.github/workflows/evalproof.yml
- uses: technobasant/evalproof@v1.4.0
  with:
    baseline: eval/baseline.json     # produced on main, e.g. as a build artifact
    candidate: eval/candidate.json   # produced on this branch
    suite: eval/suite.yaml
    # metric: "judge, answer_key"    # optional: gate several metrics, newline- or comma-separated

What the action checks first

With a suite, the action first runs evalproof verify on the candidate (set verify-candidate: "false" to skip it): the run file must have been made with that suite's system, judge, scorers and seed and cover exactly the dataset's cases as they are now (a tag change counts, because must_pass reads tags from the run file). A stale file, a --limit run and an edited case fail the job with exit 4 before any comparison. It also scores the stored outputs again with the suite's scorers that need no model (exact_match, contains, regex, json_schema, numeric_tolerance and the retrieval scorers) and fails when a stored score differs, so a hand-edited deterministic score is caught. It cannot check model-judged scores or the stored outputs themselves; for pull requests from people you do not trust, review the run file or use the api-key mode.

Action inputs

Input Required Default What it does
baseline yes none path to the baseline run file, for example the one produced on the base branch
candidate yes none path to the candidate run file, produced on this branch
suite no none the suite file; its gate block sets the metric and thresholds
verify-candidate no "true" with a suite, run evalproof verify on the candidate first; "false" skips it
metric no the suite's scorer to gate on; several, separated by newlines or commas, are gated with a Holm correction
max-drop no the suite's largest tolerated drop, as a fraction (0.02 is two points)
alpha no the suite's significance level
pr-comment no "false" "true" posts the report as one comment on the pull request and updates it on later runs
github-token no ${{ github.token }} the token that posts the comment when pr-comment is "true"

The action has one output, exit-code: 0 pass, 1 regression, 3 budget breach, 4 cannot gate, 2 usage error.

Comment on the pull request

Set pr-comment: "true" and the action posts the report as a comment on the pull request. It posts one comment, marked with <!-- evalproof-report -->, and updates that comment on later runs instead of adding a new one. The job summary still gets the whole report.

.github/workflows/evalproof.yml
permissions:
  contents: read
  pull-requests: write

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: technobasant/evalproof@v1.4.0
        with:
          baseline: eval/baseline.json
          candidate: eval/candidate.json
          suite: eval/suite.yaml
          pr-comment: "true"

The comment needs pull-requests: write on the token. A comment is cut at 65,000 bytes, with a note that says where to find the rest. The comment step runs after the gate, so a failed gate is still posted.

Fork pull requests

A pull request from a fork gets a read-only token, so the comment cannot be posted. The action prints a ::warning:: and keeps the gate's result. The comment never changes whether the job passes.

Choose a workflow

evalproof init writes one of two workflows:

--ci keyless makes no model call in CI: developers run the suite on their own login and commit evals/runs/latest.json; the job takes the baseline from the base branch, verifies the committed run, and gates. It also warns when files you watch (WATCH, default evals/) changed but the run file did not.

--ci api-key makes the run in CI with a repository secret. It starts on every pull request (a path filter would leave a required check pending), and its first step compares against the base branch: when nothing in WATCH changed it says so and runs nothing. Pull requests from forks get no secrets, so for them it runs nothing and says so. Responses are cached in evals/.evalproof/cache between runs, keyed on the contents of evals/, so a push that changes no prompt or case is answered from the cache. The cache is saved right after the run, before the gate, so a pull request that fails the gate keeps it too.

If the judge is a Claude Code or Codex CLI, evalproof init --ci api-key writes a workflow that installs the CLI at the version found at init time and runs evals/suite.yaml as it is. The run step sets EVALPROOF_CLI_AUTH: api-key and the repository secret (ANTHROPIC_API_KEY for Claude Code, OPENAI_API_KEY for Codex), so the CLI uses that key instead of a login. The suite names no key and there is no second suite file; a suite with an api_key_env still uses that variable. The pinned CLI version is advisory: a different version than the baseline's only warns, and the gate still refuses when the resolved model changes.

GitLab CI

evalproof init --platform gitlab writes .gitlab-ci.evalproof.yml. It never edits your .gitlab-ci.yml. It prints the two lines to add there:

.gitlab-ci.yml
include:
  - local: .gitlab-ci.evalproof.yml

If you have no .gitlab-ci.yml, create one with those two lines. --ci keyless (the default) and --ci api-key work as on GitHub:

Terminal
evalproof init --platform gitlab                    # keyless
evalproof init --platform gitlab --ci api-key --judge anthropic:claude-haiku-5-5

The file defines one job, evalproof, in the test stage. Change the stage if your pipeline has no test stage. The job runs in merge request pipelines only, because the baseline is the file evals/runs/latest.json on the target branch. It runs in python:3.13-slim and installs git when the image has none.

If your other jobs also run on every branch push, add a workflow: rules: block so a push to a branch with an open merge request does not start two pipelines.

  • Keyless. The job needs no variable and makes no model call. It verifies the committed run file, then gates it against the target branch, and warns when files in EVALPROOF_WATCH changed but the run file did not.
  • API key. The job runs the suite with a key kept in a CI/CD variable. Add a variable named after the judge's key (ANTHROPIC_API_KEY for anthropic:) under Settings, CI/CD, Variables. Mark it Masked. Do not mark it Protected: GitLab passes a protected variable only to pipelines on protected branches, and a merge request from a feature branch is not one. Responses are cached in evals/.evalproof/cache, and the cache is kept when the gate fails.
  • Fork merge requests. A merge request from a fork gets no CI/CD variables. In api-key mode the job runs nothing and says so.
  • Coding-agent CLI judges. --platform gitlab --ci api-key refuses a judge that is a Claude Code or Codex CLI, because the image has no Node to install the CLI with. Use a hosted API target, or --ci keyless.

The job writes a JUnit report, evalproof-junit.xml, and GitLab shows it in the merge request's test summary. The artifact is kept for a week, and it is saved when the gate fails, which is the report you want to read. What the report holds, and how to use it in Jenkins or Azure DevOps: JUnit reports.

Bitbucket Pipelines

Experimental

init --platform bitbucket is experimental, like --platform gitlab: the file it writes can change in a minor release (Versioning).

evalproof init --platform bitbucket writes bitbucket-pipelines.yml. Bitbucket has no way to include a file from the repository, so the pipeline is the file itself. init never edits a bitbucket-pipelines.yml that someone else wrote, and --force does not change that: it writes evals/bitbucket-pipelines.snippet.yml instead and prints a line telling you to merge it. The comment at the top of the snippet says where each part goes. A bitbucket-pipelines.yml that init wrote itself follows the usual rule: kept, or replaced with --force.

Terminal
evalproof init --platform bitbucket                    # keyless
evalproof init --platform bitbucket --ci api-key --judge anthropic:claude-haiku-5-5

Turn Pipelines on for the repository (Repository settings, Pipelines) if you have not.

The pull-request pipeline is under pipelines: pull-requests: '**':, so it starts for every pull request, and its single step compares against the file evals/runs/latest.json on the destination branch. The step runs in python:3.13-slim, installs git when the image has none, and installs the evalproof release that wrote the file. The file sets clone: depth: full because the destination branch has to be in the clone, and Bitbucket clones the last 50 commits by default.

  • Keyless. The step needs no variable and makes no model call. It verifies the committed run file, then gates it against the destination branch, and warns when files in EVALPROOF_WATCH changed but the run file did not. The file has no branches section: with no pull request there is no destination branch to compare with.
  • API key. The step runs the suite with a key kept in a repository variable. Add a variable named after the judge's key (ANTHROPIC_API_KEY for anthropic:) under Repository settings, Pipelines, Repository variables, and mark it Secured. Responses are cached in evals/.evalproof/cache through a cache defined under definitions: caches:. A second step, under branches: main:, runs the suite on every push to main and keeps the run as the artifact evalproof-run.json; download it and commit it as evals/runs/latest.json to move the baseline. Change main to your default branch, or delete the step to run nothing there. Bitbucket expires artifacts after 14 days.
  • Fork pull requests. Bitbucket starts no pipeline for a pull request from a fork, so the key never reaches one and nothing runs for it.
  • Coding-agent CLI judges. --platform bitbucket --ci api-key refuses a judge that is a Claude Code or Codex CLI, because the image has no Node to install the CLI with, and so refuses --cli-auth oauth-token as well. Use a hosted API target, or --ci keyless.

The file relies on variables Bitbucket sets for every step: BITBUCKET_PR_DESTINATION_BRANCH (the branch the pull request targets, set in pull-request pipelines only) and, in the comments, BITBUCKET_PR_ID, BITBUCKET_BRANCH and BITBUCKET_CLONE_DIR. Their definitions are in Atlassian's variables and secrets page, and the rule that pull-request pipelines are not started for forks is in pipeline start conditions. The pipeline writes no JUnit report; add --junit FILE to the compare line if a tool of yours reads one (JUnit reports).

Check before every push

evalproof hook install writes a git pre-push hook that runs evalproof verify on the committed run file. It makes no model call. A run file that no longer matches the suite or its dataset (an edited case, a changed prompt, a run made with --limit) stops the push before CI would have, and the message says how to refresh it.

Terminal
evalproof hook install                      # evals/suite.yaml and evals/runs/latest.json
evalproof hook install --suite evals/support/suite.yaml --run evals/support/runs/latest.json
evalproof hook uninstall
  • Skipping it. git push --no-verify skips the hook once. The hook skips itself, with a message and without stopping the push, when the suite or the run file does not exist, or when neither evalproof nor uvx is available.
  • Your own hook. install replaces only a hook that starts with its marker line. A pre-push hook you wrote is refused unless you pass --force; to keep it, add evalproof verify evals/runs/latest.json evals/suite.yaml to it yourself. uninstall removes only a hook that has the marker.
  • Where it goes. .git/hooks, or the folder core.hooksPath names; linked worktrees share it. A hook lives in the clone and is not committed, so each clone installs its own.
  • Runs made with a tag filter. A run made with run --tags verifies against the cases that filter selects. Do not commit one as latest.json: the gate refuses (exit 4) to compare runs made with different tag filters, because it would decide on a subset of the cases.

The hook is the same check CI's keyless mode starts with, run earlier. Command reference: evalproof hook.

This repository's own CI

This repository's own CI runs the offline example on every commit and asserts that the gate returns 1 for the real regression and 0 for the noisy and refactored candidates.

Next steps