data-platform-architecture: a reference design where every decision cites a measurement

Context

The public labs on this site each answer a narrow question: does exactly-once survive a broker kill, does a rollback restore a table, which incremental load survives a backfill. An architect’s job is the layer above them: which of those answers a platform should be built on, and how they fit together for one business.

data-platform-architecture is that layer, written for the same fictional SaaS business the data-stack-lab projects generate data for. It has four consumers with different needs: finance and analysts, customers looking at their own usage inside the product, the product’s own features, and models.

Constraints

Everything in it is assembled from public labs on synthetic data. It is not a description of any employer’s or client’s platform, and the workload tiers it prices are assumptions about a plausible business, not measurements of one.

The labs run multi-node clusters on one host, so their results show behaviour (what fails, what is lost, what recovers), not capacity. The managed platforms (Snowflake, Databricks, BigQuery) have not been run against this workload; they are compared from list prices and documentation only, and the decision that depends on them is marked Proposed.

Architecture

• Thirteen decision records: three ways in (events through Kafka, database changes through CDC, vendor files), Iceberg tables behind a catalog that vends scoped credentials, exactly-once with per-layer reconciliation, contracts that wait for consumers, Type 2 history, loads by data interval, freshness measured from the reader’s side, serving by access pattern, rehearsed recovery, three layers of tests, point-in-time features, canary releases, and the managed-platform choice.
• Views: data paths, zones and who may read each, deployment, and a failure-modes table where every row cites a measured recovery.
• Quality targets: freshness, completeness, correctness, recoverability, change safety and release safety, each with how it is measured and what has been measured so far, and a list of what has not.
• Governance and operations: ownership, classification, personal data and erasure, access, lineage; runbooks for backfills, late input, schema changes, failover, restore and tenant skew.
• A platform selection guide and cost model for Snowflake, Databricks SQL Serverless, BigQuery on-demand, Athena and Trino on EC2, priced from list prices quoted with their sources.

In the code

The evidence in each decision record is not written by hand. A build script reads the measured figures from one YAML file and writes them into every record that cites them, and into the decision table in the README. CI runs it in check mode, so a figure that changes in the file but not in a record fails the build, and so does an Accepted decision that cites nothing.

for a in adrs:
    for section in SECTIONS:
        if section not in a["text"]:
            problems.append(f"{a['path'].name}: missing '{section}'")
    for i in a["evidence"]:
        if i not in evidence:
            problems.append(f"{a['path'].name}: unknown evidence id '{i}'")
    if a["status"] == "Accepted" and not a["evidence"]:
        problems.append(f"{a['path'].name}: an Accepted decision must cite evidence")

A second check follows every source link to the lab repository’s file, weekly as well as on every push, so a renamed results file shows up before a reader finds it.

Decisions and trade-offs

Results

Twelve of thirteen decisions are backed by measured results. Among them: every record exact in 9 of 9 failure runs; 909,968 rows restored by a table rollback; and high-water-mark loads that lost 96.56% of a month when backfilled newest day first, which is why loads go by data interval. All 32 figures, with methods and sources, are in the repository’s EVIDENCE.md.

The platform cost turns on one ratio. From list prices, paying per scan (BigQuery on-demand, Athena) costs less than a Medium Snowflake or Databricks warehouse until one hour of compute scans 1.4–3.4 TB; against self-run Trino the line is about 0.4 TB. Which side a workload falls on is the measurement that decides its bill, and the guide shows how to take it from query history. The reasoning is in the write-up.

What went wrong

The first cost model ranked platforms by total. At every workload tier, the per-scan engines came out cheapest by a factor of ten or more. The ranking was my own assumption about terabytes scanned per engine-hour, printed back. The model now leads with the break-even and prints the assumed ratio next to every total.

Fact-checking caught overstatements before publishing. Drafts said a restore recovered rows “deleted after the target time” (it restored a dropped table), that managed Postgres services cannot run Citus (Azure has offered managed Citus; Amazon RDS and Aurora do not offer it), and that a full scan at the large tier reads 23 TB (that is all stored data; the event history is about 18 TB, so the scan costs about $101, not $131). Each was checked against its source and corrected.

The first diagram tried to show everything. Every cross-cutting service on one chart made a tangle of crossing lines. The README now draws the data path only, and names the registry, probes and canaries in a sentence below it.

What I’d do differently

Measure the managed platforms. Run the same dbt project and the same week of queries on Snowflake, Databricks and BigQuery, and replace the assumed compute equivalence with a measured one. Until then the cost guide is arithmetic.

Build the CDC path. Change data capture is in the design and not in any lab yet; replication slots across a Postgres failover are the first thing it has to prove.

Exercise erasure. The governance document designs deletion of one person’s data across zones, snapshots and serving copies; a test that proves nothing remains would turn that design into evidence.

The decisions, documents, cost model and checks are in data-platform-architecture.