My canary analysis aborted a good release and promoted a bad one
A Prometheus canary analysis aborted a good release and promoted a bad one before it caught 6 of 6. Judge against stable, refuse empty windows, count requests.
11 min read
Topic
Observability & SLOs
What to measure when 'uptime' means nothing for a data platform: freshness SLIs, burn-rate alerts, lineage, and alerts that should actually page someone.
Availability is a weak first signal for a data product: every service can be green while a table serves yesterday’s answer. I prefer consumer-facing signals such as freshness, completeness, decision latency, and safe replay, each with a named owner. These notes cover what deserves to wake a person, how to connect an alert to a repair action, and why deleting a noisy alert can be a reliability improvement.
4 posts
All writingA warehouse alert is a state machine. Slack is only where it renders.
Slack alerts with Acknowledge and Resolve buttons on warehouse metrics: why the state and its write order come first, and where a receiver can actually live.
14 min read
Validate said SUCCESS. The upgrade would have died halfway.
A Hive Metastore 4 upgrade rehearsed on a clone: zero constraints, a validator that passed anyway, and why the plan became blue/green.
14 min read
Iceberg maintenance: 20 files into 1, and a rollback I actually performed
Measure what compaction buys on a small-files partition, then delete 909,968 rows and get them back — and find where the safety net stops.
~25 min, and the rollback itself takes under a second