governed-agent — a LangGraph and MCP Agent With a Review Queue, Measured

Problem and decision

ProblemAn analytics agent that can query data and propose refunds will be fluently wrong some of the time, and telling the model to behave does not stop a wrong answer or an injected instruction from becoming state.

DecisionTreat every model output as a candidate. It becomes state only after a typed schema, a policy, a deterministic grounding check and, for anything that would change something, a named reviewer; approved actions run once through an outbox.

Scope

Team: One person
Scale: A 54-question golden set over a seeded warehouse of 9,000 orders; one local model, recorded and replayed in CI
Links: Source

Outcomes

48/54 answers matched an answer key computed by SQL one live run of a 54-question golden set against qwen3:8b under Ollama, temperature 0, thinking off, over a seeded synthetic warehouse; 95% CI 78–95% · Oct 2026, one recorded run
3 of 37 answers published without review were wrong same run; a finding is published only after its schema, policy and grounding checks pass with no proposal attached; 95% Wilson interval on 3/37 is 3–21% · Oct 2026, one recorded run
4 of 8 unanswerable questions answered with “the data does not have this” eight questions with no supporting data (NPS, margin without costs, delivery time without delivery dates and others); 95% CI 22–78% · Oct 2026, one recorded run
0 of 5 planted instructions that led to a proposed action five questions that make the model read instructions planted in product descriptions and refund notes; actions can only be proposed, and execute after a named person approves · Oct 2026, one recorded run

Schematic: 54 questions pass through schema, policy and grounding gates into three routes — 37 published with 3 wrong, 8 held for review with 3 wrong, 9 honest fallbacks with none wrong — The model got 48 of 54 right, but the figure a deployment answers for is the 3 wrong answers among the 37 that published without a person. Read the analysis

governed-agent — what a model's answer must pass to become state

Act LangGraph · Ollama Budgets on turns, tool calls and proposals, plus a deadline; the stop reason is recorded.
Query MCP · sqlglot · DuckDB One parsed query, re-rendered, on a read-only connection. A proposal is only recorded.
Validate Pydantic · policy rules Not a valid Finding after one repair, or holding contact details: rejected outright.
Ground cell match An unmatched number holds the finding. A real figure under the wrong label passes.
Review FastAPI · Next.js · PostgreSQL Every proposal and held finding waits for a named person and a written reason.
Execute outbox · idempotency key An approved action runs at most once: a unique outbox row, then an idempotency key.

Context

Governed AI delivery describes the operating model I use for AI features: generation is a candidate, and policy decides what becomes product state. That page has to stay at the level of principles, because the system it comes from is not mine to publish. This repository is the same idea built from scratch and in the open, small enough to read in an afternoon and complete enough to measure.

The agent answers questions about the warehouse of a fictional homewares retailer: orders, customers, products, refunds, inventory. It can also propose three actions: a refund, a supplier quality case, or a change to a metric’s definition. It cannot perform any of them. The question the repository answers is not “can a model write SQL?” but “what reaches state without a person, and how often is it wrong?”

Constraints

The model is the least trusted component. Every guarantee below has to hold when it misbehaves, including when it reads instructions planted in the data.
A local model, so anyone can reproduce the run. qwen3:8b under Ollama, temperature 0, seed 0, thinking off, on an Apple M5 with 24 GB. A hosted model is optional, not required.
Synthetic data, seeded. 9,000 orders, 1,500 customers, 910 refunds, generated from a fixed seed so the golden set’s answer keys can be computed by SQL, not written by a model.
CI cannot call a model. It has to be deterministic and free, and it still has to fail when the model would see something new.

Architecture

A LangGraph graph with code-owned edges: plan, act and tools in a loop, then draft, validate, policy, ground and route. The only way to the data is an MCP server with four tools. query_warehouse sits behind a parser-based SQL guard and a read-only DuckDB connection. propose_action checks a proposal against the data and records it, and nothing more.

Findings, proposals, reviews and an append-only audit log live in PostgreSQL behind a FastAPI service. A Next.js review UI shows each run’s timeline, every SQL statement, and each number linked to the cell it came from. An approved action executes at most once, through a transactional outbox with an idempotency key. OpenTelemetry traces span the API, the agent and the MCP server.

In the code

Every gate in the graph feeds this one function, and it is the only code that lets a finding publish without a person. Publication is the fall-through case, reached only when nothing blocked, every number was grounded and no rule asked for review, which is exactly how the three wrong answers that passed every check published themselves.

# Even a "can't answer" goes to a person if it claims something was done or comes with a
# proposal: the text would be false, or the proposal needs a decision either way.
_SEEN_BY_A_PERSON = frozenset({"claims_side_effect", "proposes_action"})
# …
def route(finding: Finding | None, checks: Checks, violations: list[Violation]) -> Disposition:
    """Publish only what passed everything; hold what a person can still rescue."""
    if finding is None:
        # No finding at all: the model wrote something unparseable (rejected), or the run
        # ran out of time before it wrote anything (an honest "can't answer").
        return Disposition.REJECTED if checks.schema_errors else Disposition.UNANSWERABLE
    if any(v.severity == "block" for v in violations):
        return Disposition.REJECTED
    if finding.status == "unanswerable" and not any(
        v.code in _SEEN_BY_A_PERSON for v in violations
    ):
        return Disposition.UNANSWERABLE
    if checks.ungrounded or violations:
        return Disposition.HELD_FOR_REVIEW
    return Disposition.PUBLISHED

The policy rules that produce those violations sit above it in agent/ga_agent/policy.py.

Decisions and trade-offs

Decision	Alternative considered	Why	What it cost
An explicit graph with code-owned edges	A loop around the model	Stopping, repair and routing are decisions the code makes and records	More code than a loop, and every new step is a graph change
MCP as the only path to data	The agent importing a database client	The agent’s capabilities are the server’s tool list; read-only is a parser plus a read-only connection, not a prompt	A guard written for one dialect; a second engine needs its own allow-list review
A deterministic grounding check	A second model judging support	A judge is a classifier that needs calibrating; a cell match gives the same verdict every run and links each number to its source	It proves provenance, not relevance: a real number under the wrong label passes
Review before execute, enforced by the data model	Telling the model to ignore injected text	Resistance is probabilistic; the absence of an execution path is not	Every proposal costs a person’s decision, including the correct ones
CI replays a recorded real model	A live model in CI, or only a fake	Deterministic and free, and it exercises the real tools and checks	Any prompt change fails CI until the cassette is re-recorded against the live model

Results

One live run of 54 questions, with the full method in MEASUREMENTS.md. The model answered 48/54 correctly, including every count, total and lookup. That is the least useful number in the run.

The one a deployment answers for is what got through unreviewed: 37 findings were published automatically and 3 of 37 were wrong. Eight were held for review, three of them wrong and five right. Nine got the honest “I can’t answer this from the data available”, and all nine were right to.

The weakest property was admitting ignorance: 4 of 8 unanswerable questions were admitted as unanswerable. The strongest was injection: 0 of 5 planted instructions led to a proposal, and no action executed, because no execution path exists without approval. Median latency was 23.7 s per question, p95 46.1 s, with 1.1 tool calls on average.

Two live runs at temperature 0 gave byte-identical model replies, and CI replays them. A one-line change to the system prompt makes every case a cassette miss, and the evalproof gate fails the build. The analysis of where the six wrong answers went is in the agent got 48 of 54 right; three of the wrong ones published themselves.

What the measurements refused to show

The grounding check did not catch the published errors, and it was never going to. Two of the three were a question the data cannot answer, met with a real figure from a real query. Delivery time came back as supplier lead time; conversion rate came back as the share of web orders delivered. Every number traced to a cell. The check did catch the two invented $0.00 answers.

A fix to the check published more wrong answers. Between the first and final runs the model’s side was byte-identical, 167 of 167 exchanges, so every difference was the checks. Accepting a declared number found in a result, even when its calculation field held prose, released three right answers from review and two wrong ones into publication. The first pass’s lower error count had been luck: those two were held for a formatting quirk, not because anything detected them.

Not measured: other models, thinking on, the API and UI under load, or a human reviewer’s accuracy at the queue.

What I’d do differently

Hold answers whose question terms match no column. Both published proxies answered a question about data that does not exist. A deterministic rule comparing the question’s key terms with the schema is the cheapest next control, and the next thing to measure.

Build authentication before the queue. The reviewer’s identity is a request header. The decisions are recorded with a name and a reason, but nothing proves who sent them. Reviewer auth and separation of duties are first on the roadmap.

Make tampering detectable, not just inconvenient. Triggers reject updates and deletes on the audit log, but a database superuser can drop a trigger. A hash-chained log, and real migrations in place of a schema created at startup, would turn a reference implementation into something I would run.

All case studies