What a suite covers: evalproof readiness¶
evalproof readiness reports which parts of quality a suite covers and which it leaves untested.
It reads the suite and the dataset (and a run file, if you pass one) and reports, with no model
call, which of eleven dimensions the suite covers: correctness, behaviour (tool use), quality,
regression gate, reliability, cost and latency, safety, robustness, multi-turn, load and
production monitoring. Each is covered, partial, missing or out of scope, with the evidence.
- correctness is covered by a deterministic scorer that grades answers against a key
(
contains,regex,facts, and so on); behaviour by achecksscorer, when at least half the cases have a check on the tool calls or rounds; quality by anllm_judgescorer, calibrated. - regression gate needs a
gateblock, andmin_casesno larger than the dataset; cost and latency needmax_cost_usdormax_p95_latency_msin it. - safety and robustness are covered when a case is tagged
safetyorrobustness. With--runthe tags in the run file count too, so a run imported from a red-team tool covers safety without a dataset; the evidence names both ("12 of 40 cases taggedsafety; run: 85 imported cases taggedsafety"). multi-turn is at most partial, since evalproof cannot say how many multi-turn cases are enough. - reliability is covered with 3 or more
trialsper case, partial with 2 (two trials show disagreement but not a rate) and missing with 1; with--runit reads the run's owntrials. load and production monitoring are out of scope: use a load-testing tool and an observability platform.
The report says what the suite measures, not that the measurement is good enough. --require
turns it into a gate: the exit code is 1 unless every named dimension is covered, and 2 for a
usage error. The names are correctness, behaviour, quality, regression-gate,
reliability, cost-latency, safety, robustness, multi-turn, load and
production-monitoring.
Next steps¶
- Cover reliability with repeated trials and behaviour with testing agents.