Ten CI gates, and the failure behind each one
Every gate in this repository exists because something got through. Three of them because a suite reported success against a broken artifact.
10 min read
Topic
Data quality & contracts
Validation, reconciliation, quarantine and tolerant readers — practical boundaries for imperfect producers and changing schemas.
“Fix it upstream” stops being useful advice when producers update at different speeds or belong to another team. Then the boundary has to do real work: null, type, range, and referential checks; quarantine that keeps bad records inspectable; reconciliation against expected volume; and readers that can survive a field arriving late or changing shape. These notes are about making failure visible without turning every imperfect record into data loss.
3 posts
All writingBronze, silver, gold on Iceberg: the dbt config that is silently ignored
Build a medallion pipeline with dbt-trino on Iceberg, orchestrate it with Airflow 3, and find the partition spec dbt accepts and discards without a warning.
~40 min, most of it waiting on two container builds
The data platform is part of the product
A practical operating model for reliable data and AI products: consumer-facing promises, typed decisions, idempotency, provenance, and repair.
10 min read