Removing an optional field breaks consumers. Adding one breaks history.
Schema changes teams call safe still break someone. A JSON Schema contract checker, 51 labelled changes, an oracle, and no breaking change passed as safe.
12 min read
Field notes
Notes from systems that had to keep working.
Field notes on data platforms, database lifecycle, distributed systems, production agentic AI, pipeline reliability, and self-hosted infrastructure.
-
Data platforms
11 pieces · Streaming, lakehouse, data contracts, and the upgrades that break them
-
-
Exactly-once held through nine failures. The window counts did not.
Kafka, Flink, ClickHouse and Iceberg stayed exact through nine failures, yet 10-second window counts fell to 59.8% of on-time events. Why, and the fixes.
13 min read
-
A warehouse alert is a state machine. Slack is only where it renders.
Slack alerts with Acknowledge and Resolve buttons on warehouse metrics: why the state and its write order come first, and where a receiver can actually live.
14 min read
-
Ten seconds per idle shard: why I maintain a Kinesis connector for Spark 4
The awslabs Kinesis connector for Spark 4 waits on every idle shard. Why I ported the synchronous connector to Spark 4.2 and AWS SDK v2, and maintain it in public.
14 min read
-
Validate said SUCCESS. The upgrade would have died halfway.
A Hive Metastore 4 upgrade rehearsed on a clone: zero constraints, a validator that passed anyway, and why the plan became blue/green.
14 min read
-
The Spark 4 upgrade that succeeded and wrote nothing
Spark 4.2 on hadoop-aws 3.5 read 9,029 records and wrote zero objects, and no task failed. Why the S3A committer was bypassed, and the tests that now catch it.
17 min read
-
A lakehouse spine: Trino, Iceberg and Polaris, with credentials that expire
Build Trino 483 on Iceberg with Apache Polaris as the REST catalog, then prove the vended S3 credentials cannot read the table next door.
~45 min to a working stack; the credential-vending section is the part that takes the time
-
Bronze, silver, gold on Iceberg: the dbt config that is silently ignored
Build a medallion pipeline with dbt-trino on Iceberg, orchestrate it with Airflow 3, and find the partition spec dbt accepts and discards without a warning.
~40 min, most of it waiting on two container builds
-
Iceberg maintenance: 20 files into 1, and a rollback I actually performed
Measure what compaction buys on a small-files partition, then delete 909,968 rows and get them back — and find where the safety net stops.
~25 min, and the rollback itself takes under a second
-
Two engines, one Iceberg catalog: StarRocks beside Trino, neither holding a key
Attach StarRocks to the same Polaris REST catalog Trino writes through, measure both on identical data, and check for standing S3 keys.
~20 min, mostly waiting for the StarRocks backend to register
-
The data platform is part of the product
A practical operating model for reliable data and AI products: consumer-facing promises, typed decisions, idempotency, provenance, and repair.
11 min read
-
AI systems
13 pieces · Evaluation, agents, retrieval and applied ML, measured
-
Validation picked the fraud model that had read the future
A naively joined fraud model beat the point-in-time one on validation, 0.746 to 0.681 AUC-PR, then scored 0.363 to 0.941 on the next two weeks. Why it happened.
16 min read
-
Filtered vector search returned four rows instead of ten, and nothing failed
A 10% filter cut pgvector's post-filtered answer to 3.9 of 10 rows and OpenSearch's filtered recall to 0.745 on 381,559 vectors. No error was raised.
12 min read
-
The agent got 48 of 54 right. Three of the wrong ones published themselves.
An analytics agent answered 48 of 54 questions correctly on a local model. What mattered was 3 of 37 auto-published answers wrong, and why no check saw them.
11 min read
-
The eval score fell 4.8 points. Twelve answers decided whether it meant anything.
A prompt change cost 4.8 points on an LLM judge, and 12 of 83 paired cases carried all the evidence. Why a release gate needs a paired test, not a delta.
13 min read
-
Agents read your definitions. Somebody has to own them.
Agents fail on stale or contradictory knowledge before they fail on reasoning. One concept per file, an owner, a date, a Disputed section, and generate the rest.
13 min read
-
The classifier was 95% accurate. The trend it counted was a third too flat.
A 95%-accurate classifier counted a rise from 10% to 25% as 8% to 17%. Why counting labels flattens trends, and which estimators report them honestly.
14 min read
-
The best OCR model I tested does not ship. The scorer decided why.
Eight OCR engines on real clinic records, one fixed scorer. The strongest reader stays switched off: slower on the target box, and it cannot say when it is unsure.
15 min read
-
Label the string, not the row: a short-text triage lab
A synthetic 20,000-row multilingual lab: normalize to distinct keys, cluster them for one LLM call each, and why a cosine threshold needs a margin.
15 min read
-
The user's laptop is not a server. Make it a queue consumer.
A cloud agent that drives a browser on a user's Mac kept reporting it offline. The fix was not sticky routing: model the laptop as a queue consumer.
13 min read
-
The OCR plan said fix the photographs. Its first measurement withdrew it.
An OCR plan blamed skewed, creased phone photos for lost accuracy. Its first step, a per-defect cost table and a deskew test, showed capture cost nothing.
15 min read
-
Ten gates, and the failure behind each one
Every gate in this repository exists because something got through. Three of them because a suite reported success against a broken artifact.
10 min read
-
Agents write the diff. The repo has to review it.
How to run Cursor, Claude Code, and Codex in an existing repo: AGENTS.md, CODEOWNERS, CI gates, and the AI-generated PR pitfalls that blow up review.
16 min read
-
A RAG pipeline on a Delta lakehouse, and the bug that survived being fixed
Scrape to bronze, silver, gold on Spark 4 and Delta, embed into Chroma, and serve answers that cite their sources or decline to answer.
~1 h, most of it pulling images and the first Ivy resolution
-
Databases and distributed systems
7 pieces · Sharding, failover, backup and restore on real clusters
-
Citus removed the node. Its cleanup records stayed behind.
Draining a Citus worker and removing it straight away left cleanup records for a node that no longer existed. A later rebalance retried one move 33 times.
11 min read
-
pgBackRest against a Patroni cluster: archive, rebuild, and restore
Wire pgBackRest into a running Patroni cluster over mutual TLS with Ansible, then prove it: full backup, replica rebuilt from the repo, and a point-in-time restore.
~90 min on an existing cluster; the restore drill itself is ~15 min
-
Patroni HA on PostgreSQL 18 with etcd, proven by killing the primary
Build a two-node Patroni cluster on Rocky 8 under QEMU, then SIGKILL the leader and measure the promotion, the timeline bump, and the pg_rewind rejoin.
~2–3 h first bring-up (TCG is slow); ~5 min for the failover itself
-
Dedicated pgBackRest TLS on RHEL 8 with Ansible—before Postgres
Stand up a dedicated pgBackRest TLS repo host on generic/rhel8 with Ansible-issued certs, then prove server-ping and mTLS from a client VM.
~90 min first bring-up (TCG + dnf)
-
PostgreSQL 18 won't take a volume at /var/lib/postgresql/data
The mount point moved in the PG 18 image, so every compose file copied from a 12–17 guide breaks. The corrected file, then replication, promotion and PITR.
~35 min hands-on
-
nodetool refresh only reads the live table's upload/ directory
Why a valid ScyllaDB snapshot restore can return zero rows with no error, the restore that works after a TRUNCATE, and the AIO limit that stops node three.
~40 min hands-on
-
A failover lab: six database engines, eight scenarios, every one run
Six database engines, eight failover scenarios, one 8-CPU Docker rig. The results matrix, the headline measurements, and how to run the first scenario.
~45 min hands-on
-
Delivery and infrastructure
2 pieces · Progressive delivery and the lab infrastructure underneath
-
My canary analysis aborted a good release and promoted a bad one
A Prometheus canary analysis aborted a good release and promoted a bad one before it caught 6 of 6. Judge against stable, refuse empty windows, count requests.
11 min read
-
amd64 Vagrant labs on Apple Silicon without broken host networking
Apple Silicon labs that boot still fail host and mesh ping when you use aarch64 boxes or park the VMs on your Wi‑Fi subnet.
~60–90 min first bring-up (TCG is slow)
-
Self-hosted runbooks
3 pieces · Follow-along setups for a VPS you own
-
A free mailbox for your domain with Stalwart
Want a free mailbox for your domain? Run Stalwart on Docker, skip ACME when 443 is taken, publish SPF/DKIM/DMARC, and prove a Gmail round-trip.
~60 min hands-on
-
A free VPN on your VPS with WireGuard
Want a free VPN you actually control? Pin wg-easy to :15, mount /lib/modules, keep the UI off 80/443, and prove traffic exits from your VPS.
~25 min hands-on
-
A free private remote desktop with RustDesk
Want a free private remote desktop? Self-host RustDesk on Docker, open UDP 21116, pin the key, and keep sessions past the public cutoff.
~30 min hands-on
No writing matches those filters.