Case studies

Data platforms, AI systems, and the architecture that keeps them running.

Case studies across data platforms, AI systems and system architecture: employer practice, consulting, open source, independent products and measured labs.

  1. Data platforms 5 cases Ingestion, processing and serving at terabyte scale
  2. AI systems 7 cases Agents, evaluation, retrieval and applied ML
  3. Database lifecycle 3 cases Replication, failover, backup and version upgrades
  4. System architecture 2 cases Boundaries, contracts and recovery paths

Data platforms

5 cases · Ingestion, processing and serving at terabyte scale

01Professional practiceCase 2020 – present

Rebuilding a production data platform for scale and recovery

Data Engineer → Senior Data Engineer

A six-year platform story: faster processing, lower cost, safer change, and recovery designed before the incident.

50% faster core processingrole-level before/after wall-clock on the rebuilt Spark/Airflow path — not a lab sample · Data Engineer role, 2020–2024

40% lower infrastructure costrole-level before/after infrastructure run-rate on the Kubernetes migration — not a lab sample · Data Engineer role, 2020–2024

Python · SQL · Apache Spark · Apache Kafka · Apache Airflow · dbt

  • Data quality & contracts
  • Observability & SLOs
  • Iceberg & the lakehouse

02Open sourceCase 2023 – present

spark-sql-kinesis: a Kinesis source for Spark 4 without the idle-shard floor

Maintainer

Spark 4 dropped the AWS SDK the streaming connector depended on, and the maintained alternative paid seconds on every idle shard

≥ 6 s per idle shard, every batch, on the alternative connectorconnector source plus staging Spark UI stage timings; floor = empty-poll count × poll timeout, and the connector requires that product to exceed 5 s, so 6 s is the lowest it accepts · Aug 2026, staging

Scala · Apache Spark 4.2 · AWS SDK v2 · Amazon Kinesis · Kinesis Producer Library · Protobuf

  • Kafka & streaming
  • Apache Spark

03Open sourceCase 2026

streaming-platform: exactly-once clickstream, counted at every layer

Author

Exactly-once is a claim about counts, so every layer was counted against a ledger, through broker kills, TaskManager kills and stalls

9 of 9 failure runs with every event layer exactthree controller-broker kills, three TaskManager kills and three 150 s stalls under load; Kafka, ClickHouse, Iceberg and the dead-letter topic reconciled against the generator's ledger by sequence number · Oct 2026

59.8% of on-time events left in the window counts after one broker kill118,229 of 197,749 on-time events counted in their 10 s windows in the worst of three broker-kill runs; the rest went to the late side output because catch-up reordered partitions past the 5 s watermark bound · Oct 2026

Apache Kafka 4.3 · Apache Flink 1.20 · ClickHouse · Apache Iceberg · Schema Registry · Avro · Prometheus

  • Kafka & streaming
  • ClickHouse
  • Iceberg & the lakehouse

04Open sourceCase 2026

dataset-registry: data contracts that wait for the consumers they break

Author

A breaking schema change should be a question to the teams it breaks, asked by a registry that knows who they are

0 breaking schema changes passed as safe, in any mode51 hand-labelled JSON Schema changes, each scored under BACKWARD, FORWARD and FULL; every label cross-checked by an oracle that ran 8,092 generated probe documents through a real JSON Schema validator · measured Oct 2026

2 of 51 changes flagged as breaking that are safe, per directionthe same 51 cases; a widened and a narrowed regular expression, an added anyOf branch and an added multipleOf are flagged in both directions by design, and each truly breaks only one · measured Oct 2026

Go · PostgreSQL 18 · sqlc · OpenLineage · JSON Schema · Next.js

  • Data quality & contracts
  • PostgreSQL

05Measured labCase 2024 – 2026

A lakehouse, and the RAG pipeline that runs on it

Personal lab

The two halves of the thesis as one system: Trino and Iceberg underneath, retrieval and answers on top, both measured

88 ms one-month predicate over 1.5M Iceberg rowsmedian of 7 server-side runs from system.runtime.queries, after 2 warm-ups, on a month-partitioned Iceberg table · measured 2026 · 2024 stack rebuilt

909,968 rows recovered by snapshot rollbackan over-broad DELETE (a WHERE reaching years further back than intended), then ALTER TABLE EXECUTE rollback_to_snapshot, counted against the pre-incident total · measured 2026 · 2024 stack rebuilt

Trino · Apache Iceberg · Apache Polaris · StarRocks · Apache Spark · Delta Lake

  • Iceberg & the lakehouse
  • RAG
  • Distributed databases

AI systems

7 cases · Agents, evaluation, retrieval and applied ML

01Professional practiceCase 2024 – present

Turning model output into product state you can audit

Senior Data Engineer

A governed analytics workflow with typed state, provenance, evaluation, review, and an honest fallback path.

60% faster analytics deliveryrole-level before/after delivery time for recurring analytics requests — not a lab sample · Senior Data Engineer role, 2024–present

75% less analyst manual effortrole-level before/after manual effort per recurring analyst request — not a lab sample · Senior Data Engineer role, 2024–present

Python · Pydantic · LangGraph · LangChain · CrewAI · Google-ADK

  • AI agents in production
  • Data quality & contracts
  • RAG

02Consulting engagementCase 2026

Short-text classification, alerting and agent knowledge for a lakehouse team

Consultant — data and AI

A comment classifier that failed in production, rebuilt around the unit of work — plus the alerting, agent knowledge and trend measurement around it.

Databricks · Delta Lake · Python · multilingual-e5 · UMAP · HDBSCAN

  • Applied ML & evaluation
  • Data quality & contracts
  • AI agents in production

03Open sourceCase 2026

evalproof: LLM evaluation as measurement, gated on what the data can show

Author

An LLM judge's pass rate is a biased estimate, and a few-point drop on a small suite is mostly noise

p = 0.39 for a 4.8-point judge drop the gate declined to failexact McNemar on 83 paired cases — 8 worse, 4 better, 71 unchanged — qwen3:1.7b judged by qwen3:8b on Ollama, temperature 0 · measured Oct 2026

1,469 → 4,422 ms p95 latency, the regression that was realsystem latency over the same 83 cases, sequential, one Apple M5; mean answer length went from 19 to 77 tokens · measured Oct 2026

Python · numpy · Pydantic · httpx · Ollama · Qwen3

  • Applied ML & evaluation
  • AI agents in production

04Open sourceCase 2026

governed-agent: an analytics agent that can propose but never execute

Author

An agent that writes SQL and proposes refunds will sometimes be confidently wrong, and a system prompt is not a control

48/54 answers matched an answer key computed by SQLone live run of a 54-question golden set against qwen3:8b under Ollama, temperature 0, thinking off, over a seeded synthetic warehouse; 95% CI 78–95% · Oct 2026, one recorded run

3 of 37 answers published without review were wrongsame run; a finding is published only after its schema, policy and grounding checks pass with no proposal attached; 95% Wilson interval on 3/37 is 3–21% · Oct 2026, one recorded run

Python · LangGraph · MCP · sqlglot · DuckDB · FastAPI

  • AI agents in production
  • Data quality & contracts

05Open sourceCase 2026

realtime-ml: a fraud model whose leakage, skew and label delay are measured

Author

Most real-time models fail before they are served: in a join that reads the future, a backfill that disagrees with the stream, labels that arrive weeks late

0.363 vs 0.941 future AUC-PR, naive join against point-in-time joinsame rows, labels, model and hyperparameters; only the training join differs. Scored on the two weeks after training with point-in-time features and true labels, on seeded synthetic payments (874,446 transactions, 0.80% fraud) · measured 2026-10 · seeded synthetic data

82.5% of fraud cost saved on two unseen weeksa $5 review per flag plus the amount of every missed fraud, against approving everything; threshold chosen on the validation window, true labels. A hand-written rule saved 20.7% · measured 2026-10 · seeded synthetic data

Python · Apache Kafka · Feast · Redis · DuckDB · LightGBM

  • Applied ML & evaluation
  • Data quality & contracts

06Independent productCase 2026 – present

Camera-first clinic records, with OCR that knows when to refuse

Founder and sole engineer — built outside my day job

Clinicians photograph paper records; the system drafts structured fields for review and refuses what it cannot read.

95.8% of matched lab-result values read correctlyfield-level scorer on 97 labelled pages of real clinic captures; development set, not clinical validation · Aug–Sep 2026

8 OCR engines scored through one fixed scorersame extraction pipeline and field-level scorer; 11 + 26 labelled pages for classic engines, one 17-page packet for vision-language models · Aug 2026

Python · FastAPI · PostgreSQL · RapidOCR · ONNX Runtime · React

  • Applied ML & evaluation
  • Data quality & contracts
  • PostgreSQL

07Measured labCase 2026

retrieval-bench: three vector engines, one set of vectors

Author

Is Postgres enough, and is hybrid search worth it? One embedding pass, three engines under the same limits, and numbers that flatter none of them

3.9 of 10 rows returned by post-filtered HNSW at 10% selectivitypgvector 0.8.7 HNSW at ef_search 40 with iterative scan off, 1,000 queries over 381,559 vectors, a filter keeping 10% of rows; recall@10 0.391 against exact kNN over the rows that pass · measured 2026

0.745 recall@10 of OpenSearch's efficient filter at 10% selectivityfilter inside the knn clause, Faiss HNSW at ef_search 40, 1,000 queries over 381,559 vectors, against exact kNN over the rows that pass · measured 2026

pgvector · PostgreSQL 18 · Qdrant · OpenSearch · Faiss · Python

  • RAG
  • PostgreSQL

Database lifecycle

3 cases · Replication, failover, backup and version upgrades

01Measured labCase 2026

Sharding Postgres with Citus, measured against TimescaleDB and ClickHouse

Author

A schema that refused to distribute six ways, a tenant whose shard was 7.4× the median, and a rebalance under live writes

0 errors while a worker joined and 7 shard groups moved under load8 client threads for 600 s, 80% one-tenant reads and 20% single-row inserts against the Citus coordinator; every acknowledged write counted back afterwards · measured Oct 2026

4.2× faster join co-located than repartitionedthe same join over tenants 1–400, both returning 617,895 rows; p50 of 7 runs, 1,251.7 ms co-located on (tenant_id, user_id) vs 5,206.0 ms against a copy distributed by user_id · measured Oct 2026

Citus · PostgreSQL 17 · TimescaleDB · ClickHouse · ClickHouse Keeper · Docker Compose

  • Distributed databases
  • PostgreSQL
  • ClickHouse

02Measured labCase 2026

Multi-engine HA lab: eight failover scenarios

Personal lab

Six database engines, eight failover scenarios on real multi-node clusters — with the numbers

8 of 8 failover scenarios passedmy own test plan, executed on Docker with 8 CPU / 12.5 GB · 2026

383 µs replay lag measured on one hosta single pg_stat_replication reading with primary and hot standby on the same machine — loopback, no network, so it is a floor for the configuration rather than a latency anyone should expect across hosts

MongoDB · ScyllaDB · SolrCloud · MariaDB · Galera · Redis

  • Distributed databases
  • PostgreSQL
  • Observability & SLOs

03Measured labCase 2026

Orchestrated failover and a restore I actually performed

Personal lab

A production-shaped Postgres cluster on amd64 VMs: killed the primary, rebuilt the replica, recovered a dropped table

10–20 s observed promotion window after SIGKILLSIGKILL of the Patroni leader, then patronictl polled every 10 s until the survivor showed as Leader — so 10 s is the measurement resolution, not the precision of the result · 2026

~5 s replica rebuilt from backuppatronictl reinit; the pgBackRest restore itself logged 5,020 ms, then Patroni reported the replica created · 2026

PostgreSQL · Patroni · etcd · pgBackRest · Ansible · Rocky Linux

  • PostgreSQL
  • Distributed databases
  • Self-hosted ops

System architecture

2 cases · Boundaries, contracts and recovery paths

01Open sourceCase 2026

platform-deployments: a release is a commit, and a bad one stops itself

Author

Readiness probes cannot see a release that starts cleanly and then fails real requests; only the traffic can

6 of 6 bad releases aborted by canary analysisa release whose Envoy sidecar returns 500 on 30% of requests, under 20 req/s from k6; three runs with keep-alive clients, three with a new connection per request; Prometheus analysis every 15 s after a 30 s initial delay · measured Oct 2026 · kind, three nodes

0 of 18,003 requests failed across three good canary releasesk6 at 20 req/s for five minutes per run, GET /healthz and GET /review-queue with a 10 s timeout; canary 20% → 50% with 60 s pauses, background analysis and an evidence step before 100% · measured Oct 2026 · kind, three nodes

Kubernetes · kind · Argo CD · Argo Rollouts · Envoy · Prometheus

  • Kubernetes
  • Observability & SLOs

02Independent productCase 2024 – present

ClickHomes: an independent real-estate product

Founder and sole engineer — built outside my day job

A real-estate product I own end to end: PostgreSQL for truth, ClickHouse for speed, RESO as the only way in

90+ SQL migrationsversioned migration files in the repository

Python · FastAPI · PostgreSQL · ClickHouse · RESO · Next.js

  • PostgreSQL
  • Data quality & contracts
  • AI agents in production