Skip to content

description: The Python objects evalproof exports, generated from the docstrings: load, compare, gate, calibrate, traces and adapters.

API reference

The objects the From Python and Agent SDK adapters guides use, generated from the docstrings in the package. Import them from evalproof and evalproof.adapters; the names in evalproof.__all__ are the supported API.

Load, compare and gate

Read run files, compare two of them and turn the comparison into a gate decision.

load_run

load_run(path: str | Path) -> Run

Read a run file. A file written by a newer evalproof (a higher format_version) raises RunFileError rather than loading as a run it may misread.

compare_runs

compare_runs(
    baseline: Run,
    candidate: Run,
    *,
    alpha: float = 0.05,
    n_boot: int = 5000,
    seed: int = 0,
    metrics: list[str] | None = None,
    aggregate: str = "mean",
) -> Comparison

Compare two runs on the cases they share. aggregate (mean, all or any) says how a case's trials become one value when the runs repeat each case; it is ignored at one trial. Raises ComparisonRefused when the runs have a different number of trials.

Comparison dataclass

Comparison(
    paired_ids: list[str],
    only_baseline: list[str],
    only_candidate: list[str],
    same_dataset: bool,
    metrics: dict[str, MetricComparison],
    baseline: RunStats,
    candidate: RunStats,
    baseline_label: str = "baseline",
    candidate_label: str = "candidate",
    same_judge: bool = True,
    judge_changes: list[str] = list(),
    system_changes: list[str] = list(),
    comparable: bool = True,
    edited: list[str] = list(),
    judge_warnings: list[str] = list(),
    trials: int = 1,
    aggregate: str | None = None,
    aggregate_ignored: list[str] = list(),
    tag_filter_change: str | None = None,
)

paired_ids instance-attribute

paired_ids: list[str]

only_baseline instance-attribute

only_baseline: list[str]

only_candidate instance-attribute

only_candidate: list[str]

same_dataset instance-attribute

same_dataset: bool

metrics instance-attribute

metrics: dict[str, MetricComparison]

baseline instance-attribute

baseline: RunStats

candidate instance-attribute

candidate: RunStats

baseline_label class-attribute instance-attribute

baseline_label: str = 'baseline'

candidate_label class-attribute instance-attribute

candidate_label: str = 'candidate'

same_judge class-attribute instance-attribute

same_judge: bool = True

judge_changes class-attribute instance-attribute

judge_changes: list[str] = field(default_factory=list)

system_changes class-attribute instance-attribute

system_changes: list[str] = field(default_factory=list)

comparable class-attribute instance-attribute

comparable: bool = True

edited class-attribute instance-attribute

edited: list[str] = field(default_factory=list)

judge_warnings class-attribute instance-attribute

judge_warnings: list[str] = field(default_factory=list)

trials class-attribute instance-attribute

trials: int = 1

aggregate class-attribute instance-attribute

aggregate: str | None = None

aggregate_ignored class-attribute instance-attribute

aggregate_ignored: list[str] = field(default_factory=list)

tag_filter_change class-attribute instance-attribute

tag_filter_change: str | None = None

GateConfig

Bases: BaseModel

Fail only when a drop is both larger than max_drop and statistically significant.

metric gates one scorer. Add metrics to gate several; their p-values are then Holm-corrected so the chance of a false failure across the set stays at alpha. Either may be omitted as long as one scorer is named; gated is the full list.

model_config class-attribute instance-attribute

model_config = ConfigDict(extra='forbid')

metric class-attribute instance-attribute

metric: str | None = None

metrics class-attribute instance-attribute

metrics: list[str] = Field(default_factory=list)

max_drop class-attribute instance-attribute

max_drop: float = 0.02

alpha class-attribute instance-attribute

alpha: float = 0.05

min_cases class-attribute instance-attribute

min_cases: int = 20

max_p95_latency_ms class-attribute instance-attribute

max_p95_latency_ms: float | None = None

max_cost_usd class-attribute instance-attribute

max_cost_usd: float | None = None

n_boot class-attribute instance-attribute

n_boot: int = 5000

allow_judge_change class-attribute instance-attribute

allow_judge_change: bool = False

min_pass_rate class-attribute instance-attribute

min_pass_rate: float | dict[str, float] | None = None

must_pass class-attribute instance-attribute

must_pass: list[str] = Field(default_factory=list)

max_new_failures class-attribute instance-attribute

max_new_failures: int | None = None

aggregate class-attribute instance-attribute

aggregate: Literal['mean', 'all', 'any'] = 'mean'

gated property

gated: list[str]

The scorers to gate on, metric first, without repeats.

pass_rate_floors property

pass_rate_floors: dict[str, float]

The minimum pass rate per scorer: a float applies to every gated metric.

has_absolute_checks property

has_absolute_checks: bool

True when the gate sets a check that needs no baseline.

decide

decide(
    cmp: Comparison,
    gate: GateConfig,
    candidate: Run | None = None,
    *,
    adjusted: dict[str, float] | None = None,
    candidates: int = 1,
) -> GateDecision

Decide the gate. candidate is the candidate run; the absolute checks (min_pass_rate, must_pass) need its scores, so a gate that sets one cannot be decided without it.

A matrix comparison decides each candidate on its own but corrects the p-values for the whole set at once: it passes the Holm-adjusted p per gated metric as adjusted and the number of candidates, which the power estimate divides alpha by as well.

GateDecision dataclass

GateDecision(
    status: str,
    exit_code: int,
    reasons: list[str] = list(),
    warnings: list[str] = list(),
    adjusted_p: dict[str, float] = dict(),
    power: dict[str, Power] = dict(),
    checks: list[GateCheck] = list(),
)

status instance-attribute

status: str

exit_code instance-attribute

exit_code: int

reasons class-attribute instance-attribute

reasons: list[str] = field(default_factory=list)

warnings class-attribute instance-attribute

warnings: list[str] = field(default_factory=list)

adjusted_p class-attribute instance-attribute

adjusted_p: dict[str, float] = field(default_factory=dict)

power class-attribute instance-attribute

power: dict[str, Power] = field(default_factory=dict)

checks class-attribute instance-attribute

checks: list[GateCheck] = field(default_factory=list)

RunFileError

Bases: ValueError

A run file this evalproof cannot read: its format version is unknown to it.

CacheMiss

Bases: ProviderError

--cache-only found no cached reply for a call. It is final (never retried) and it stops the run instead of being recorded on the case: a run that was to make no model call must not come back half-answered.

ProviderError

Bases: RuntimeError

A model call failed. Inside a run it is recorded on the case as an error and the run goes on; CacheMiss is the exception that stops the run instead.

Calibrate

Measure a judge against gold labels and correct its pass rate.

calibrate

calibrate(
    run: Run,
    judge_scorer: str,
    *,
    labels: dict[str, bool] | None = None,
    gold_scorer: str | None = None,
    gold_sample: int | None = None,
    seed: int = 0,
    alpha: float = 0.05,
    n_boot: int = 5000,
) -> CalibrationReport

Calibrate judge_scorer against human labels or a trusted gold_scorer.

With gold_scorer and gold_sample, a random sample of that size stands in for a human-reviewed set; the remaining gold labels are used only to report the truth the estimates are trying to recover.

A run with several trials per case is calibrated on trial 0 of each case: a verdict per case, as the labels are, and not the same case counted k times.

CalibrationReport dataclass

CalibrationReport(
    judge_scorer: str,
    gold_source: str,
    n_judged: int,
    n_gold: int,
    undecided_judge: int,
    confusion: Confusion,
    estimates: list[Estimate],
    truth: Estimate | None = None,
    disagreements: list[str] = list(),
    trials: int = 1,
)

judge_scorer instance-attribute

judge_scorer: str

gold_source instance-attribute

gold_source: str

n_judged instance-attribute

n_judged: int

n_gold instance-attribute

n_gold: int

undecided_judge instance-attribute

undecided_judge: int

confusion instance-attribute

confusion: Confusion

estimates instance-attribute

estimates: list[Estimate]

truth class-attribute instance-attribute

truth: Estimate | None = None

disagreements class-attribute instance-attribute

disagreements: list[str] = field(default_factory=list)

trials class-attribute instance-attribute

trials: int = 1

Estimate dataclass

Estimate(
    method: str,
    point: float | None,
    low: float | None,
    high: float | None,
    n: int,
    note: str = "",
)

method instance-attribute

method: str

point instance-attribute

point: float | None

low instance-attribute

low: float | None

high instance-attribute

high: float | None

n instance-attribute

n: int

note class-attribute instance-attribute

note: str = ''

Systems and traces

The objects a python system receives and returns.

RunContext dataclass

RunContext(
    case_id: str,
    input: str | dict[str, Any],
    turns: list[str | dict[str, Any]] = list(),
    state: dict[str, Any] = dict(),
    variant: str | None = None,
    vars: dict[str, Any] = dict(),
    trial: int = 0,
)

What a python system may ask for beside the case input: declare a parameter named ctx and the system is called as func(input, ctx=ctx).

turns are the user turns after input (turn 0); state is the case's initial session state; variant and vars are the suite variant being run and its variables; trial is the repeat index, 0 to trials - 1 (always 0 when the suite runs each case once). Each call gets its own copies of the mutable parts.

case_id instance-attribute

case_id: str

input instance-attribute

input: str | dict[str, Any]

turns class-attribute instance-attribute

turns: list[str | dict[str, Any]] = field(
    default_factory=list
)

state class-attribute instance-attribute

state: dict[str, Any] = field(default_factory=dict)

variant class-attribute instance-attribute

variant: str | None = None

vars class-attribute instance-attribute

vars: dict[str, Any] = field(default_factory=dict)

trial class-attribute instance-attribute

trial: int = 0

Trace

Bases: BaseModel

What an agent did, not only what it said: its replies, tool calls and model usage.

responses class-attribute instance-attribute

responses: list[str] = Field(default_factory=list)

tool_calls class-attribute instance-attribute

tool_calls: list[ToolCall] = Field(default_factory=list)

rounds class-attribute instance-attribute

rounds: int | None = None

usage class-attribute instance-attribute

usage: list[ModelUsage] = Field(default_factory=list)

meta class-attribute instance-attribute

meta: dict[str, Any] = Field(default_factory=dict)

ToolCall

Bases: BaseModel

One tool call an agent made while answering.

name instance-attribute

name: str

args class-attribute instance-attribute

args: dict[str, Any] = Field(default_factory=dict)

result class-attribute instance-attribute

result: Any = None

ok class-attribute instance-attribute

ok: bool = True

error class-attribute instance-attribute

error: str | None = None

id class-attribute instance-attribute

id: str | None = None

turn class-attribute instance-attribute

turn: int = 0

latency_ms class-attribute instance-attribute

latency_ms: float | None = None

Adapters

Build a trace from what an agent SDK returns.

TraceRecorder

TraceRecorder()

Record what a plain-Python agent did, then result() a dict a python system returns.

rec = TraceRecorder()
rec.round()
rec.tool("get_weather", {"city": "Paris"}, result={"temp_c": 20})
rec.usage(model="m", tokens_in=120, tokens_out=30)
rec.reply("It is 20 degrees in Paris.")
return rec.result()

A tool call belongs to the user turn that is under way: the number of replies recorded so far, unless turn says otherwise. Pass ok=False with an error for a call that failed.

tool

tool(
    name: str,
    args: dict[str, Any] | None = None,
    result: Any = None,
    ok: bool = True,
    error: str | None = None,
    turn: int | None = None,
) -> ToolCall

usage

usage(
    model: str | None = None,
    tokens_in: int = 0,
    tokens_out: int = 0,
    cost_usd: float | None = None,
    role: str = "agent",
) -> ModelUsage

round

round() -> None

Count one model call (a generation round).

reply

reply(text: str) -> None

Record the assistant's reply to the current user turn.

result

result(output: str | None = None) -> AgentResult

The dict a python system returns. output defaults to the last reply.

mcp_call

mcp_call(
    rec: TraceRecorder,
    name: str,
    args: dict[str, Any] | None,
    call_tool_result: Any,
) -> ToolCall

Record an MCP tool call on rec from the CallToolResult the server returned.

The result is the structured content when the server sent any, else the text of the content blocks (the raw blocks when none is text). isError makes the call fail, with the text as the error. Works on the SDK object or the same thing as a dict.

from_events

from_events(events: Iterable[Any]) -> AgentResult

Build {"output", "trace"} from ADK events (objects or dicts), in order.

events is the events of one run, or a list of such lists, one per user turn. Calls and responses pair by id, or by order and name when an id is missing. The reply of a turn is the text of its last final-response event. Usage is summed per model over the complete (not partial) events that report it; rounds counts the model responses.

from_messages

from_messages(
    messages: Iterable[Any] | Mapping[str, Any],
) -> AgentResult

Build {"output", "trace"} from LangChain messages (objects or dicts).

Accepts the list, or the state dict a create_agent run returns (its messages). Each AIMessage.tool_calls entry is a call; a ToolMessage answers the call with its tool_call_id, and status == "error" makes it fail. The reply of a turn is the text of its last AI message that calls no tool; rounds counts the AI messages.

from_messages

from_messages(messages: Iterable[Any]) -> AgentResult

Build {"output", "trace", "cost_usd"?} from Claude Agent SDK messages (objects or dicts).

messages is what one session yielded, or a list of such lists, one per query() call (one per user turn). Within a session the totals are running totals, so each run's last ones count; across runs they are added up.

ToolUseBlock entries are calls and ToolResultBlock entries (in user messages) answer them by id, is_error making the call fail. The reply of a turn is ResultMessage.result, else the text of its last assistant message. cost_usd is total_cost_usd, the only cost the SDK reports for the run; usage comes from model_usage. rounds counts the assistant responses (messages with the same message_id are one response).

from_result

from_result(
    result: Any,
    *,
    error_if: Callable[[Any], bool] | None = None,
) -> AgentResult

Build {"output", "trace"} from the result of Runner.run / Runner.run_sync.

Tool calls come from the tool_call_item entries of new_items and are paired with their tool_call_output_item by call_id, else by order. output is final_output (JSON when it is not a string), else the text of the last message. Tokens come from context_wrapper.usage and rounds is the number of raw_responses.

DEFAULT_TOOL_ERROR module-attribute

DEFAULT_TOOL_ERROR = "An error occurred while running the tool. Please try again."

What the SDK's default_tool_error_function returns for a tool that raised (tool.py:2002).

from_spans

from_spans(
    source: str
    | PathLike[str]
    | Mapping[str, Any]
    | list[Any],
    *,
    trace_id: str | None = None,
) -> AgentResult

Build {"output", "trace"} from the spans of one trace (experimental).

source is a path to a .json or .jsonl file, an OTLP/JSON object (resourceSpans), one span or a list of spans, as OTLP spans or as the Python SDK's span.to_json() objects. A file may hold one OTLP object, one per line (the collector's file exporter), or several objects in a row (ConsoleSpanExporter). Spans are ordered by start time. When the spans belong to more than one trace, trace_id picks one (hex, with or without 0x) and ValueError is raised without it.

Tool calls come from tool spans of GenAI semantic conventions (also Google ADK schema 2 and the Vercel OpenTelemetry integration), Google ADK schema 1, OpenInference and the Vercel legacy ai.toolCall spans; an error type or an ERROR status makes a call fail. rounds counts inference spans, usage is summed per model, and the reply is the text of the last inference span. A tool span exported with content capture off has no arguments or result: the call is kept, a UserWarning names the setting to turn on, and trace.meta ["content_captured"] is False. Every call is turn 0.

from_events

from_events(events: Any) -> AgentResult

Build {"output", "trace"} from LlamaIndex agent events or an AgentChatResponse.

events is the list of events a workflow agent emitted (ToolCall, ToolCallResult, AgentOutput; others are ignored), a list of such lists (one per turn), an AgentChatResponse, or a list of them (one per turn).

from_output

from_output(output: Any) -> AgentResult

Build {"output", "trace"} from a CrewAI run (objects or dicts).

output is the CrewOutput of crew.kickoff(...), a TaskOutput, or a list of them (one per turn). Each entry of an assistant message's tool_calls is a call; a message with role "tool" answers the call with its tool_call_id.

from_messages

from_messages(source: Any) -> AgentResult

Build {"output", "trace"} from a Pydantic AI run (objects or dicts).

source is the run result (agent.run_sync(...)), the list of messages (result.all_messages()), or a dict with messages (and optionally output). Each ToolCallPart is a call; a ToolReturnPart answers the call with its tool_call_id. rounds counts the model responses.

from_prediction

from_prediction(
    prediction: Any, *, output_field: str | None = None
) -> AgentResult

Build {"output", "trace"} from a DSPy prediction (object or dict), or a list of them.

Tool calls come from history (DSPy 3.5) or trajectory (DSPy 3.4 and earlier). A prediction with neither is a reply with no tool calls.

from_messages

from_messages(
    messages: Iterable[Any] | Mapping[str, Any] | None,
) -> AgentResult

Build {"output", "trace"} from Haystack chat messages (objects or dicts).

Accepts the list, or the dict Agent.run returns (its messages). Each ToolCall of an assistant message is a call; a ToolCallResult answers the call it names as its origin (by id, else by name and order). rounds counts the assistant messages.

from_messages

from_messages(source: Any) -> AgentResult

Build {"output", "trace"} from an AutoGen AgentChat run (objects or dicts).

source is the TaskResult of agent.run(...) or team.run(...), its list of messages, or a dict with messages. Each FunctionCall of a ToolCallRequestEvent is a call; a FunctionExecutionResult answers the call with its call_id.

from_history

from_history(history: Any) -> AgentResult

Build {"output", "trace"} from a Semantic Kernel chat history (objects or dicts).

history is a ChatHistory, the list of its messages, or a dict with messages. Function calls are read from FunctionCallContent items, function results from FunctionResultContent items.

AgentResult

Bases: TypedDict

What a python system returns for an agent: output is required in practice, the rest is optional. Tokens and cost default to the sums over trace.usage.

output instance-attribute

output: str

trace instance-attribute

trace: Trace | dict[str, Any]

tokens_in instance-attribute

tokens_in: int

tokens_out instance-attribute

tokens_out: int

cost_usd instance-attribute

cost_usd: float