Removing an optional field breaks consumers. Adding one breaks history.
Schema changes teams call safe still break someone. A JSON Schema contract checker, 51 labelled changes, an oracle, and no breaking change passed as safe.
12 min read
Topic
Data quality & contracts
Validation, reconciliation, quarantine and tolerant readers — practical boundaries for imperfect producers and changing schemas.
“Fix it upstream” stops being useful advice when producers update at different speeds or belong to another team. Then the boundary has to do real work: null, type, range, and referential checks; quarantine that keeps bad records inspectable; reconciliation against expected volume; and readers that can survive a field arriving late or changing shape. These notes are about making failure visible without turning every imperfect record into data loss.
12 posts
All writingExactly-once held through nine failures. The window counts did not.
Kafka, Flink, ClickHouse and Iceberg stayed exact through nine failures, yet 10-second window counts fell to 59.8% of on-time events. Why, and the fixes.
13 min read
Validation picked the fraud model that had read the future
A naively joined fraud model beat the point-in-time one on validation, 0.746 to 0.681 AUC-PR, then scored 0.363 to 0.941 on the next two weeks. Why it happened.
16 min read
A warehouse alert is a state machine. Slack is only where it renders.
Slack alerts with Acknowledge and Resolve buttons on warehouse metrics: why the state and its write order come first, and where a receiver can actually live.
14 min read
The classifier was 95% accurate. The trend it counted was a third too flat.
A 95%-accurate classifier counted a rise from 10% to 25% as 8% to 17%. Why counting labels flattens trends, and which estimators report them honestly.
14 min read
The best OCR model I tested does not ship. The scorer decided why.
Eight OCR engines on real clinic records, one fixed scorer. The strongest reader stays switched off: slower on the target box, and it cannot say when it is unsure.
15 min read
Label the string, not the row: a short-text triage lab
A synthetic 20,000-row multilingual lab: normalize to distinct keys, cluster them for one LLM call each, and why a cosine threshold needs a margin.
15 min read
The OCR plan said fix the photographs. Its first measurement withdrew it.
An OCR plan blamed skewed, creased phone photos for lost accuracy. Its first step, a per-defect cost table and a deskew test, showed capture cost nothing.
15 min read
The Spark 4 upgrade that succeeded and wrote nothing
Spark 4.2 on hadoop-aws 3.5 read 9,029 records and wrote zero objects, and no task failed. Why the S3A committer was bypassed, and the tests that now catch it.
17 min read
Ten gates, and the failure behind each one
Every gate in this repository exists because something got through. Three of them because a suite reported success against a broken artifact.
10 min read
Bronze, silver, gold on Iceberg: the dbt config that is silently ignored
Build a medallion pipeline with dbt-trino on Iceberg, orchestrate it with Airflow 3, and find the partition spec dbt accepts and discards without a warning.
~40 min, most of it waiting on two container builds
The data platform is part of the product
A practical operating model for reliable data and AI products: consumer-facing promises, typed decisions, idempotency, provenance, and repair.
11 min read