Skip to content

description: The JSON run file evalproof writes: every field of the run, case result, output, score and trace, with the version each appeared in.

Run file format

A run file is the JSON record of one run of a suite: what the system answered for every case and how every scorer scored it. This page is for anyone reading run files from a script, importing results from another tool, or committing runs for CI, and a test checks it against the models that write them.

evalproof run and evalproof import write run files; compare, verify, rehash, report, calibrate and pairwise read them. They are pretty-printed JSON with two-space indent, written by Run.model_dump_json and read back by load_run, so a file you write by hand must pass the same validation.

An example

One case of a run of the ticket-router example, as evalproof 1.0.0 writes it. A real file has one entry in results per case and trial.

runs/baseline.json
{
  "format_version": 1,
  "info": {
    "suite": "ticket-router",
    "system": "python:router:baseline_route",
    "dataset_path": "cases.jsonl",
    "dataset_hash": "bc97ea936a7d2a2a",
    "started_at": "2026-11-14T14:34:46+00:00",
    "finished_at": "2026-11-14T14:34:46+00:00",
    "wall_time_s": 0.001,
    "evalproof_version": "1.0.0",
    "seed": 0,
    "environment": {
      "python": "3.13.15",
      "platform": "macOS-27.0.1-arm64-arm-64bit-Mach-O",
      "machine": "arm64",
      "processor": "arm"
    },
    "suite_hash": "2600c532d6653efe",
    "variant": null,
    "trials": 1,
    "tag_filter": null
  },
  "results": [
    {
      "case_id": "t01",
      "tags": [
        "billing"
      ],
      "output": {
        "output": "billing",
        "retrieved": [],
        "latency_ms": 0.0037080608308315277,
        "tokens_in": 0,
        "tokens_out": 0,
        "cost_usd": 0.0,
        "cached": false,
        "error": null,
        "trace": null
      },
      "scores": {
        "queue": {
          "value": 1.0,
          "passed": true,
          "reason": null,
          "judge_tokens": 0,
          "judge_cost_usd": 0.0
        }
      },
      "case_hash": "58122e3549d8b9e0",
      "content_hash": "413a3f1b96fc5896",
      "trial": 0
    }
  ]
}

Run

Field Type What it holds
format_version integer the layout of the file, 1 in evalproof 1.0; a file without it was written before 1.0 and is read as 1; see File format version
info object facts about the run; see Run info
results list one case result per case and trial

File format version

format_version says which layout a run file has. It rises only when a change would make an older evalproof misread a file; adding an optional field does not raise it, because an older evalproof ignores a field it does not know. A run file written before 1.0 has no format_version, and evalproof reads it as format 1.

A file whose format_version is higher than the running evalproof knows is refused, never half-read. Every command that reads a run file prints evalproof: error: <file>: this run file is format version N, written by a newer evalproof than <version> (which reads up to format version M): upgrade evalproof to read it and exits 2. A format_version that is not a whole number from 1 is refused the same way. evalproof rehash writes the current format version into the file it rewrites.

Run info

Field Type What it holds Since
suite string the suite's name 0.1
system string the system's label, or its target when it has none 0.1
dataset_path string the dataset path as written in the suite 0.1
dataset_hash string a short hash of the dataset's cases 0.1
started_at string start time, ISO 8601 in UTC 0.1
finished_at string end time, ISO 8601 in UTC 0.1
wall_time_s number seconds from start to end 0.1
evalproof_version string the version that wrote the file 0.1
seed integer the suite's seed 0.1
environment object python, platform, machine and processor, plus facts about the model targets, such as judge_model_resolved, the model id the provider reported; cache_only is "true" when the run was made with --cache-only, so its replies came from the response cache and no model ran 0.1
suite_hash string or null the suite fingerprint; null before 0.5 and in imported runs 0.5
variant string or null the variant the run was made with; null for the suite as written and before 0.6 0.6
trials integer repeats per case; 1 before 0.7 and in imported runs that kept no repeats 0.7
tag_filter object or null the tag filter the run was made with (--tags, --exclude-tags); null when every case of the dataset ran, and before 1.3 1.3

Tag filter

Present in info.tag_filter when the run was made with --tags or --exclude-tags. verify applies the same filter to the dataset to know which cases the run should hold, and compare warns when its two runs used different filters. Both lists are sorted and hold no duplicates, so the same filter always writes the same text. Added in 1.3; it is an optional field, so format_version stays 1.

Field Type What it holds
include list of strings the tags from --tags: a case is selected when it carries at least one; empty when --tags was not given
exclude list of strings the tags from --exclude-tags: a case that carries any of them is left out, even one include selects

Case result

Field Type What it holds Since
case_id string the case's id 0.1
tags list of strings the case's tags 0.1
output object what the system returned; see System output 0.1
scores object one score per scorer, by scorer name 0.1
case_hash string or null a short hash of the case as scored, so compare can tell an edited case from an unchanged one; null before 0.5 0.5
content_hash string or null the same without the tags, so a tag change is not an edit; null in files written by 0.5.0 0.5.1
trial integer which repeat of the case this is, counting from 0; 0 before 0.7 0.7

System output

Field Type What it holds
output string what the system answered; for a multi-turn case, the last reply
retrieved list of strings the ranked ids a retrieval system returned
latency_ms number milliseconds taken
tokens_in integer input tokens
tokens_out integer output tokens
cost_usd number cost in US dollars
cached boolean true when the answer came from the response cache
error string or null the error when the system failed; null otherwise
trace object or null what the agent did; see Trace; null for a system that reports none and in files written before the trace existed

Score

Field Type What it holds
value number or null the continuous score, used by bootstrap comparisons; null when the scorer could not decide
passed boolean or null the pass/fail outcome, used by pass rates and McNemar; null when undecided or when the scorer has no threshold
reason string or null why, when the scorer says
judge_tokens integer tokens the judge used for this case
judge_cost_usd number what the judge cost for this case

An undecided case, with a null value, is counted and reported. It is never scored as a pass.

Trace

Field Type What it holds
responses list of strings the reply per user turn; the last is the output
tool_calls list every tool call the agent made
rounds integer or null model calls or generation rounds, when known
usage list model usage by role
meta object anything else, such as the agent's name

Tool call

Field Type What it holds
name string the tool's name
args object the arguments
result any what the tool returned
ok boolean false when the call failed
error string or null the error text
id string or null the call id the SDK gave it
turn integer the user turn the call belongs to, counting from 0
latency_ms number or null milliseconds the call took

Model usage

Field Type What it holds
role string free text: agent, judge, worker and so on
model string or null the model id
tokens_in integer input tokens
tokens_out integer output tokens
cached_tokens integer tokens served from a provider cache
cost_usd number or null cost in US dollars

Repeated trials

With trials above 1, results holds one entry per case and trial, each with its own trial number. Run.by_id() returns the first trial of each case, and Run.by_case() returns every trial of each, in trial order. Run files written before 0.7 hold one result per case, all with trial 0.

Reading older files

Every field added after 0.1 has a default, so any older file still loads, format_version included: a file without it is format 1. compare and the gate say what an older file lacks: without case_hash they cannot tell an edited case from an unchanged one, and verify needs a run made with evalproof 0.5 or newer. evalproof rehash adds the case hashes to a file from before 0.5. Upgrading lists what each version lacks.

Next steps