JUnit test reports¶
--junit PATH writes the result as JUnit XML, so GitLab, Jenkins, Azure DevOps and other CI systems show cases and gate checks as tests, with their own failure lists and history. This page is for anyone wiring evalproof into one of them. The report is an extra output: it changes no exit code (Exit codes).
Experimental
The JUnit report, including its test case names, is new in 1.0.0 and experimental: it can change in a minor release until it has been used in more projects. See Versioning.
Two commands write it.
| Command | The report holds |
|---|---|
evalproof run --junit PATH |
one testsuite per run written, one testcase per case result |
evalproof compare --gate --junit PATH |
one testsuite, evalproof gate, with one testcase per gate check |
The file is UTF-8 with an XML declaration and LF line endings. Characters that XML 1.0 cannot hold, such as control characters in a model's output, are removed from every value, so a parser never rejects the file. A file already at the path is replaced, and the folders above it are created.
A run¶
evalproof run --junit junit.xml writes the report after each run. The root is <testsuites> with the totals of its suites.
- A
<testsuite>is one run. Itsnameis<suite>.<variant>, or<suite>.<system>for a run without a variant. With--all-variantsor several--variantflags the one file holds a testsuite per variant. - A
<testcase>is one case result. Itsclassnameis the testsuite's name, and itstimeis the system's latency in seconds. - With trials above 1 there is a testcase per trial, named
<case id> [trial N], with N counted from 0. With one trial the name is the case id. - The case's output goes in
<system-out>, cut at 2000 characters. - The testsuite's
<properties>hold the evalproof version, the dataset and suite hashes, the system, the variant, the trials, the seed, the start and finish times, the wall time, the cost and the run's environment facts.
A testcase is marked as the run report's "Failed cases" list marks it:
| Case | Testcase holds | Message |
|---|---|---|
| every scorer decided and passed | nothing: it passed | none |
| a scorer failed | <failure type="scorer failed"> |
failed <scorer> (<score>) for each failing scorer |
| the system raised an error | <error type="system error"> |
the error text |
| a scorer could not decide, and none failed | <skipped> |
undecided on <scorer>: no scorer reached a verdict |
The body of a failure holds each failing scorer and its reason. A case with a failing scorer and an undecided one is a failure. An errored case has no <system-out>.
The report is written when every case errored too, so the errors show in CI. The command still exits 2 then, as it does without --junit.
A gate¶
evalproof compare baseline.json candidate.json --gate --junit gate.xml writes one <testsuite name="evalproof gate">. It holds a testcase per check the gate evaluated, with classname evalproof gate and a time of 0. The testcases come from the checks of the gate's decision, so a check the gate adds later appears here with no change to the report.
| Testcase name | The check |
|---|---|
max_drop: <metric> |
the significance test on the metric's drop, one per gated metric |
max_new_failures: <metric> |
cases that went from pass to fail, when max_new_failures is set |
min_pass_rate: <metric> |
the candidate's pass rate, when min_pass_rate is set |
must_pass |
the cases tagged in must_pass, when it is set |
max_p95_latency_ms |
the candidate's p95 latency, when max_p95_latency_ms is set |
max_cost_usd |
the candidate's total cost, when max_cost_usd is set |
A check that does not apply is left out: a metric with only continuous scores has no max_new_failures testcase.
- A passing check is a testcase with its measurement in
<system-out>. - A failed check is a
<failure type="gate check failed">whosemessageand body are the reason the terminal report prints. - A check the gate could not decide is an
<error type="cannot gate">. These are the causes of exit 4: too few paired cases (min_cases: <metric>), a judge that changed (same_judge), runs that cannot be compared (comparable), and the others Exit codes lists. When the runs cannot be compared at all the file holds one error testcase,comparison. - The gate's warnings are lines in the testsuite's
<system-out>. - The testsuite's
<properties>hold the exit code, the baseline and candidate, and the gate settings.
With several candidates each testcase name starts with the candidate's label, as in steady / max_drop: m, and the properties are candidate.1, candidate.2 and so on. The file is one testsuite whatever the number of candidates.
When the file is written¶
The file is written for every verdict, 0, 1, 3 and 4, so a failed gate always has its report. A usage error (exit 2) writes none. Two checks come before any work:
--junitwithout--gateexits 2 with--junit needs --gate. A comparison with no gate has no checks to report.- A path that cannot be written, such as a folder, exits 2 before the command runs.
Example¶
A candidate that dropped on m and is under the pass rate, as the XML shows it, with the properties trimmed:
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="evalproof" tests="2" failures="2" errors="0" skipped="0" time="0.000">
<testsuite name="evalproof gate" tests="2" failures="2" errors="0" skipped="0" time="0.000">
<properties>
<property name="exit_code" value="1" />
<property name="baseline" value="main" />
<property name="candidate" value="pr" />
</properties>
<testcase classname="evalproof gate" name="max_drop: m" time="0.000">
<failure message="m: 1.000 → 0.667 (Δ -0.333, 95% CI [-0.500, -0.167], exact McNemar p=0.0020) — a drop beyond 0.020 at α=0.05" type="gate check failed">m: 1.000 → 0.667 (Δ -0.333, 95% CI [-0.500, -0.167], exact McNemar p=0.0020) — a drop beyond 0.020 at α=0.05</failure>
</testcase>
<testcase classname="evalproof gate" name="min_pass_rate: m" time="0.000">
<failure message="m: pass rate 0.667 below the minimum 0.900 (20/30 cases)" type="gate check failed">m: pass rate 0.667 below the minimum 0.900 (20/30 cases)</failure>
</testcase>
</testsuite>
</testsuites>
Show the report in CI¶
Write the file in the same job that gates, and upload it even when the gate fails. The gate's exit code still fails the job.
GitLab¶
evals:
script:
- evalproof compare runs/baseline.json runs/candidate.json --suite evals/suite.yaml --gate --junit gate.xml
artifacts:
when: always
reports:
junit: gate.xml
GitLab lists each failed check under the pipeline's Tests tab. when: always keeps the report from a job that exited non-zero.
Jenkins¶
stage('Evals') {
steps {
sh 'evalproof compare runs/baseline.json runs/candidate.json --suite evals/suite.yaml --gate --junit gate.xml'
}
post {
always {
junit 'gate.xml'
}
}
}
The junit step sits in post { always }, so it reads the file after a failed gate. The sh step's non-zero exit code still fails the build.
Azure DevOps¶
Use the PublishTestResults@2 task with testResultsFormat: JUnit, testResultsFiles: gate.xml and condition: always().
Next steps¶
- Every flag of the two commands: Commands.
- What each exit code means: Exit codes.
- A GitHub Actions workflow for the gate: Continuous integration.
- The gate's settings: Configuration.