description: The Python objects evalproof exports, generated from the docstrings: load, compare, gate, calibrate, traces and adapters.¶
API reference¶
The objects the From Python and Agent SDK adapters
guides use, generated from the docstrings in the package. Import them from evalproof and
evalproof.adapters; the names in evalproof.__all__ are the supported API.
Load, compare and gate¶
Read run files, compare two of them and turn the comparison into a gate decision.
load_run ¶
Read a run file. A file written by a newer evalproof (a higher format_version) raises
RunFileError rather than loading as a run it may misread.
compare_runs ¶
compare_runs(
baseline: Run,
candidate: Run,
*,
alpha: float = 0.05,
n_boot: int = 5000,
seed: int = 0,
metrics: list[str] | None = None,
aggregate: str = "mean",
) -> Comparison
Compare two runs on the cases they share. aggregate (mean, all or any) says
how a case's trials become one value when the runs repeat each case; it is ignored at one
trial. Raises ComparisonRefused when the runs have a different number of trials.
Comparison
dataclass
¶
Comparison(
paired_ids: list[str],
only_baseline: list[str],
only_candidate: list[str],
same_dataset: bool,
metrics: dict[str, MetricComparison],
baseline: RunStats,
candidate: RunStats,
baseline_label: str = "baseline",
candidate_label: str = "candidate",
same_judge: bool = True,
judge_changes: list[str] = list(),
system_changes: list[str] = list(),
comparable: bool = True,
edited: list[str] = list(),
judge_warnings: list[str] = list(),
trials: int = 1,
aggregate: str | None = None,
aggregate_ignored: list[str] = list(),
tag_filter_change: str | None = None,
)
GateConfig ¶
Bases: BaseModel
Fail only when a drop is both larger than max_drop and statistically significant.
metric gates one scorer. Add metrics to gate several; their p-values are then
Holm-corrected so the chance of a false failure across the set stays at alpha.
Either may be omitted as long as one scorer is named; gated is the full list.
decide ¶
decide(
cmp: Comparison,
gate: GateConfig,
candidate: Run | None = None,
*,
adjusted: dict[str, float] | None = None,
candidates: int = 1,
) -> GateDecision
Decide the gate. candidate is the candidate run; the absolute checks
(min_pass_rate, must_pass) need its scores, so a gate that sets one cannot be
decided without it.
A matrix comparison decides each candidate on its own but corrects the p-values for the whole
set at once: it passes the Holm-adjusted p per gated metric as adjusted and the number of
candidates, which the power estimate divides alpha by as well.
GateDecision
dataclass
¶
GateDecision(
status: str,
exit_code: int,
reasons: list[str] = list(),
warnings: list[str] = list(),
adjusted_p: dict[str, float] = dict(),
power: dict[str, Power] = dict(),
checks: list[GateCheck] = list(),
)
adjusted_p
class-attribute
instance-attribute
¶
RunFileError ¶
Bases: ValueError
A run file this evalproof cannot read: its format version is unknown to it.
CacheMiss ¶
Bases: ProviderError
--cache-only found no cached reply for a call. It is final (never retried) and it
stops the run instead of being recorded on the case: a run that was to make no model call
must not come back half-answered.
ProviderError ¶
Bases: RuntimeError
A model call failed. Inside a run it is recorded on the case as an error and the run
goes on; CacheMiss is the exception that stops the run instead.
Calibrate¶
Measure a judge against gold labels and correct its pass rate.
calibrate ¶
calibrate(
run: Run,
judge_scorer: str,
*,
labels: dict[str, bool] | None = None,
gold_scorer: str | None = None,
gold_sample: int | None = None,
seed: int = 0,
alpha: float = 0.05,
n_boot: int = 5000,
) -> CalibrationReport
Calibrate judge_scorer against human labels or a trusted gold_scorer.
With gold_scorer and gold_sample, a random sample of that size stands in for a
human-reviewed set; the remaining gold labels are used only to report the truth the
estimates are trying to recover.
A run with several trials per case is calibrated on trial 0 of each case: a verdict per case,
as the labels are, and not the same case counted k times.
CalibrationReport
dataclass
¶
CalibrationReport(
judge_scorer: str,
gold_source: str,
n_judged: int,
n_gold: int,
undecided_judge: int,
confusion: Confusion,
estimates: list[Estimate],
truth: Estimate | None = None,
disagreements: list[str] = list(),
trials: int = 1,
)
disagreements
class-attribute
instance-attribute
¶
Estimate
dataclass
¶
Systems and traces¶
The objects a python system receives and returns.
RunContext
dataclass
¶
RunContext(
case_id: str,
input: str | dict[str, Any],
turns: list[str | dict[str, Any]] = list(),
state: dict[str, Any] = dict(),
variant: str | None = None,
vars: dict[str, Any] = dict(),
trial: int = 0,
)
What a python system may ask for beside the case input: declare a parameter named
ctx and the system is called as func(input, ctx=ctx).
turns are the user turns after input (turn 0); state is the case's initial session
state; variant and vars are the suite variant being run and its variables; trial is
the repeat index, 0 to trials - 1 (always 0 when the suite runs each case once). Each
call gets its own copies of the mutable parts.
turns
class-attribute
instance-attribute
¶
Trace ¶
Bases: BaseModel
What an agent did, not only what it said: its replies, tool calls and model usage.
tool_calls
class-attribute
instance-attribute
¶
ToolCall ¶
Bases: BaseModel
One tool call an agent made while answering.
Adapters¶
Build a trace from what an agent SDK returns.
TraceRecorder ¶
Record what a plain-Python agent did, then result() a dict a python system returns.
rec = TraceRecorder()
rec.round()
rec.tool("get_weather", {"city": "Paris"}, result={"temp_c": 20})
rec.usage(model="m", tokens_in=120, tokens_out=30)
rec.reply("It is 20 degrees in Paris.")
return rec.result()
A tool call belongs to the user turn that is under way: the number of replies recorded so far,
unless turn says otherwise. Pass ok=False with an error for a call that failed.
tool ¶
tool(
name: str,
args: dict[str, Any] | None = None,
result: Any = None,
ok: bool = True,
error: str | None = None,
turn: int | None = None,
) -> ToolCall
usage ¶
usage(
model: str | None = None,
tokens_in: int = 0,
tokens_out: int = 0,
cost_usd: float | None = None,
role: str = "agent",
) -> ModelUsage
result ¶
The dict a python system returns. output defaults to the last reply.
mcp_call ¶
mcp_call(
rec: TraceRecorder,
name: str,
args: dict[str, Any] | None,
call_tool_result: Any,
) -> ToolCall
Record an MCP tool call on rec from the CallToolResult the server returned.
The result is the structured content when the server sent any, else the text of the content
blocks (the raw blocks when none is text). isError makes the call fail, with the text
as the error. Works on the SDK object or the same thing as a dict.
from_events ¶
Build {"output", "trace"} from ADK events (objects or dicts), in order.
events is the events of one run, or a list of such lists, one per user turn. Calls and
responses pair by id, or by order and name when an id is missing. The reply of a turn is the
text of its last final-response event. Usage is summed per model over the complete (not
partial) events that report it; rounds counts the model responses.
from_messages ¶
Build {"output", "trace"} from LangChain messages (objects or dicts).
Accepts the list, or the state dict a create_agent run returns (its messages). Each
AIMessage.tool_calls entry is a call; a ToolMessage answers the call with its
tool_call_id, and status == "error" makes it fail. The reply of a turn is the text of
its last AI message that calls no tool; rounds counts the AI messages.
from_messages ¶
Build {"output", "trace", "cost_usd"?} from Claude Agent SDK messages (objects or dicts).
messages is what one session yielded, or a list of such lists, one per query() call
(one per user turn). Within a session the totals are running totals, so each run's last ones
count; across runs they are added up.
ToolUseBlock entries are calls and ToolResultBlock entries (in user messages) answer
them by id, is_error making the call fail. The reply of a turn is ResultMessage.result,
else the text of its last assistant message. cost_usd is total_cost_usd, the only
cost the SDK reports for the run; usage comes from model_usage. rounds counts the
assistant responses (messages with the same message_id are one response).
from_result ¶
Build {"output", "trace"} from the result of Runner.run / Runner.run_sync.
Tool calls come from the tool_call_item entries of new_items and are paired with their
tool_call_output_item by call_id, else by order. output is final_output (JSON
when it is not a string), else the text of the last message. Tokens come from
context_wrapper.usage and rounds is the number of raw_responses.
DEFAULT_TOOL_ERROR
module-attribute
¶
What the SDK's default_tool_error_function returns for a tool that raised (tool.py:2002).
from_spans ¶
from_spans(
source: str
| PathLike[str]
| Mapping[str, Any]
| list[Any],
*,
trace_id: str | None = None,
) -> AgentResult
Build {"output", "trace"} from the spans of one trace (experimental).
source is a path to a .json or .jsonl file, an OTLP/JSON object
(resourceSpans), one span or a list of spans, as OTLP spans or as the Python SDK's
span.to_json() objects. A file may hold one OTLP object, one per line (the collector's
file exporter), or several objects in a row (ConsoleSpanExporter). Spans are ordered by
start time. When the spans belong to more than one trace, trace_id picks one (hex, with
or without 0x) and ValueError is raised without it.
Tool calls come from tool spans of GenAI semantic conventions (also Google ADK schema 2 and
the Vercel OpenTelemetry integration), Google ADK schema 1, OpenInference and the Vercel
legacy ai.toolCall spans; an error type or an ERROR status makes a call fail. rounds
counts inference spans, usage is summed per model, and the reply is the text of the last
inference span. A tool span exported with content capture off has no arguments or result: the
call is kept, a UserWarning names the setting to turn on, and trace.meta
["content_captured"] is False. Every call is turn 0.
from_events ¶
Build {"output", "trace"} from LlamaIndex agent events or an AgentChatResponse.
events is the list of events a workflow agent emitted (ToolCall, ToolCallResult,
AgentOutput; others are ignored), a list of such lists (one per turn), an
AgentChatResponse, or a list of them (one per turn).
from_output ¶
Build {"output", "trace"} from a CrewAI run (objects or dicts).
output is the CrewOutput of crew.kickoff(...), a TaskOutput, or a list of
them (one per turn). Each entry of an assistant message's tool_calls is a call; a message
with role "tool" answers the call with its tool_call_id.
from_messages ¶
Build {"output", "trace"} from a Pydantic AI run (objects or dicts).
source is the run result (agent.run_sync(...)), the list of messages
(result.all_messages()), or a dict with messages (and optionally output). Each
ToolCallPart is a call; a ToolReturnPart answers the call with its tool_call_id.
rounds counts the model responses.
from_prediction ¶
Build {"output", "trace"} from a DSPy prediction (object or dict), or a list of them.
Tool calls come from history (DSPy 3.5) or trajectory (DSPy 3.4 and earlier). A
prediction with neither is a reply with no tool calls.
from_messages ¶
Build {"output", "trace"} from Haystack chat messages (objects or dicts).
Accepts the list, or the dict Agent.run returns (its messages). Each ToolCall of
an assistant message is a call; a ToolCallResult answers the call it names as its
origin (by id, else by name and order). rounds counts the assistant messages.
from_messages ¶
Build {"output", "trace"} from an AutoGen AgentChat run (objects or dicts).
source is the TaskResult of agent.run(...) or team.run(...), its list of
messages, or a dict with messages. Each FunctionCall of a
ToolCallRequestEvent is a call; a FunctionExecutionResult answers the call with its
call_id.
from_history ¶
Build {"output", "trace"} from a Semantic Kernel chat history (objects or dicts).
history is a ChatHistory, the list of its messages, or a dict with messages.
Function calls are read from FunctionCallContent items, function results from
FunctionResultContent items.
AgentResult ¶
Bases: TypedDict
What a python system returns for an agent: output is required in practice, the rest
is optional. Tokens and cost default to the sums over trace.usage.