Apache SparkSpark in production: the Spark 4 upgrade traps (S3A committers, the Hive Metastore), a Kinesis connector for Spark 4, and the measurements worth collecting before changing a configuration.
3 posts
Kafka & streamingIngestion that survives replay, late events, duplicate deliveries and producers you do not control. Kafka, Flink, Kinesis, Structured Streaming, CDC.
2 posts
Iceberg & the lakehouseTable-format engineering on Iceberg: catalogs and vended credentials, medallion models, snapshot expiry, compaction cadence, and a rollback actually performed.
6 posts
PostgreSQLReplication, point-in-time recovery, promotion, connection limits, and the version-upgrade surprises that only show up in production.
8 posts
ClickHouseColumn-store analytics: MergeTree design, partition and ordering keys, materialised views, and keeping log tables from eating the disk.
1 post
Distributed databasesSharding, quorum, failover and backup across Citus, MongoDB, ScyllaDB, Redis, MariaDB/Galera and Solr — measured on real clusters rather than quoted from a docs page.
6 posts
KubernetesProgressive delivery and stateful workloads on Kubernetes: canary analysis that judges against stable, operators, disruption, and the operational price of explicit control.
2 posts
AI agents in productionProduction agents with typed, audited outputs — and the coding-agent control plane (AGENTS.md, Cursor, Claude Code, Codex, CODEOWNERS) that keeps AI-generated PRs reviewable. LangGraph, CrewAI, Google-ADK, MCP, contracts and fallbacks.
7 posts
RAGRetrieval over corpora that keep growing: chunking, metadata filters, index choice, and the TTL and compaction policy that stops a vector store becoming a landfill.
3 posts
Observability & SLOsWhat to measure when 'uptime' means nothing for a data platform: freshness SLIs, burn-rate alerts, lineage, and alerts that should actually page someone.
4 posts
Data quality & contractsValidation, reconciliation, quarantine, tolerant readers, and contracts that keep imperfect producers from silently corrupting downstream decisions.
12 posts
Career & craftNotes on ownership, mentoring, review culture, and the difference between contributing to a system and being on call for it.
1 post
Self-hosted opsFree, fully OSS setups you can run yourself: a WireGuard VPN, a RustDesk remote desktop, a Stalwart mailbox for your own domain, and amd64 lab VMs on Apple Silicon.
5 posts
Applied ML & evaluationModels judged by a scorer you wrote before you saw the output: LLM evaluation with paired tests, point-in-time training data, OCR extraction, short-text classification, and the measurements that withdrew a plan rather than confirming it.
7 posts
Writing
Browse by system, failure mode, or layer.
Published field notes organized by system, failure mode, and platform layer — from distributed databases and PostgreSQL to streaming, Iceberg, agents, and SLOs.