dataset-registry: data contracts that wait for the consumers they break

Context

A familiar kind of data incident starts like this. Someone changes a table or a topic, the change looks harmless, and a team two hops downstream finds out from a broken dashboard. Data contracts are meant to stop that, and in practice they tend to fail in one of two ways. A checker that rejects every breaking change teaches teams to route around it: a new topic, a _v2 table, a column nobody removes. Or there is no checker, and the contract describes what someone intended last quarter.

Both failures have the same root. Nothing in the system knows who depends on what, so there is nobody to ask when a change is genuinely breaking and still the right change.

dataset-registry is the control plane I built to test a different arrangement. Every dataset has an owning team and a named person, a freshness SLO, a versioned JSON Schema contract, lineage from OpenLineage, and a list of the teams that consume it. A breaking change is neither blocked nor shipped quietly. It waits until every one of those teams has acknowledged it. The demo data belongs to a fictional retailer, the same one used in governed-agent, streaming-platform and realtime-ml.

Constraints

This is a from-scratch reference system, not a production deployment, and it has no real users. That shaped what I could claim. I could measure the checker against labelled changes, the API under load, and the delay before a freshness breach is noticed. I could not measure how real teams behave when they are asked to acknowledge a change, so the post about the checker does not pretend to.

Everything runs as containers on one host: an Apple M5 with 24 GB, Docker’s Linux VM given 8 CPUs, PostgreSQL 18 with its stock configuration, and other projects’ containers idle alongside. There is no network latency in any figure. Absolute numbers are this host’s, and the point is the shape: what grows with what.

Architecture

One Go binary holds five parts, all writing to PostgreSQL 18:

• Lineage ingest. Accepts OpenLineage RunEvents exactly as the OpenLineage HTTP transport sends them. Each event is validated against the 2-0-2 schema and stored in one transaction.
• Registry rules. Ownership, consumers and contracts. A dataset first seen in lineage is recorded as unowned, and nobody can register as its consumer until a team claims it.
• Compatibility checker. Compares two contract versions and reports, change by change, whether each one breaks BACKWARD, FORWARD or both, with a reason.
• Change-request state machine. open → approved → published, or rejected or withdrawn. There is one open request per dataset, enforced by a partial unique index.
• Freshness evaluator. Runs on an interval, records every status transition, and exports staleness to Prometheus, which owns the alerting.

Every transition goes into an append-only audit log whose triggers refuse UPDATE, DELETE and TRUNCATE. The Next.js UI is server-rendered and calls the API with bearer tokens. It shows datasets by SLO status, contract diffs with the checker’s report, a React Flow lineage graph and the change-request review.

In the code

The zero in the results comes from two choices in the checker. Each change records which directions it breaks before verdict applies the dataset’s mode, and any keyword the checker cannot reason about counts as breaking both ways, which is where the anyOf and multipleOf false alarms come from.

func verdict(mode Mode, bb, bf bool) (bool, string) {
	switch {
	case mode.checksForward() && bf:
		return true, "breaking under " + string(mode) + ": consumers validating against the old version can receive documents that fail it"
	case mode.checksBackward() && bb:
		return true, "breaking under " + string(mode) + ": documents written under the old version (history, replays) can fail the new one"
	case mode == None:
		return false, "not checked: the dataset's compatibility mode is NONE"
	case bb || bf:
		return false, "safe under " + string(mode) + ": it only breaks the direction this mode does not require"
	default:
		return false, "safe in both directions"
	}
}
// …
	for _, k := range sortedKeys(keys) {
		if supported[k] || annotations[k] {
			continue
		}
		if reflect.DeepEqual(o[k], n[k]) {
			continue
		}
		c.add(path, "unsupported_keyword",
			fmt.Sprintf("`%s` differs; the checker does not reason about it, so it is treated as breaking both ways", k), true, true)
	}

The rest of the checker, including the rule that a field missing from properties takes the schema of additionalProperties, is in api/internal/compat/compat.go.

Decisions and trade-offs

Results

The checker. On 51 hand-labelled schema changes, the checker passed 0 breaking changes as safe, in any of the three modes. A unit test enforces that on every case, so a false negative fails CI. In each direction it flagged 2 of 51 changes as breaking that are in fact safe, all from keywords it deliberately does not reason about. The labels are not just my opinion. An oracle generated 8,092 probe documents from both versions of every schema, ran them through a real JSON Schema validator, and agreed with all 51. The post on the checker covers which “safe” changes break which direction, and why the checker leans the way it does.

Ingest. OpenLineage ingest stored 8,782/s events from 32 concurrent clients, with no errors. Throughput grew about 7× from one client to 32 while p50 latency grew about 4×, which puts the limit in the database rather than the handler.

Lineage. A five-hop downstream query in a 10,000-dataset graph took 2.34 ms at p50. One hop cost the same at 100 datasets as at 10,000. Latency followed the size of the neighbourhood, not the size of the graph.

Freshness. At the default 15 s evaluator interval, a breach was noticed 7.5 s after its deadline at the median. That is half an interval, exactly what an interval scheduler predicts, so I measured the lag rather than assuming it.

What the measurements refused to show

Whether the checker is accurate on real contracts. I wrote the labelled set alongside the checker. It tests the rules the checker claims to implement and is not a sample of real-world schema changes. A real corpus would have far more $ref and oneOf, which this checker always calls breaking.

Whether people answer. The acknowledgement workflow is tested end to end by Playwright, from register to consume to breaking change to acknowledge to publish. Nothing measures how long a real consumer team takes to respond, or whether teams start rubber-stamping reviews once they arrive often enough.

Whether data arrived. Freshness means a pipeline reported COMPLETE. A job that writes nothing and reports success keeps its dataset green.

Where it slows down. The weak spots are measured and listed rather than hidden. Listing all 1,000 datasets runs three correlated subqueries per row. A pass in which all 10,000 SLOs change status at once, as after a restart with stale statuses, takes seconds because each transition is its own transaction.

What I’d do differently

Check the schema facet against the contract. OpenLineage events can carry the output schema, and the registry currently ignores it. Comparing it on every run would catch a producer drifting from its contract without anyone proposing a change. That is the most useful thing it does not do.

Give change requests a clock. A request with no expiry and no escalation turns one unresponsive team into a veto. Notifying consumers when a request opens, and escalating after a deadline, belong in the first version, not the roadmap.

Measure the checker on a real corpus before trusting the false-alarm rate. Two in 51 is a number about my own labelled set. The figure that would change a decision is the rate on a few hundred real contract histories, and I do not have that yet.