Open source
Code you can run, with the numbers it produced.
Public repositories across data platforms, AI systems, databases and delivery, each with its method and measurements, plus contributions to projects I do not maintain.
Each repository builds a working system from scratch, measures how it behaves under failure or load, and keeps the method and the raw results next to the code. The case studies on /work/ explain the decisions; the write-ups on /writing/ explain the result that surprised me. Repositories are grouped by the same capabilities as both.
Data platforms
4 repositories
-
spark-sql-kinesis
Kinesis source and sink for Spark Structured Streaming on Spark 4.2, Scala 2.13 and AWS SDK v2, published to Maven Central.
An idle shard costs one GetRecords call per batch, where the alternative connector waits at least six seconds (staging; see the write-up).
-
streaming-platform
Real-time clickstream platform: Avro contracts, exactly-once through Kafka and Flink into ClickHouse and Iceberg, with a freshness SLO and burn-rate alerts.
Every event layer stayed exact through 9 failures; one broker kill pushed up to 40% of on-time events out of the windows.
-
dataset-registry
Data-ownership control plane: owners, freshness SLOs, versioned JSON Schema contracts, OpenLineage lineage, and breaking changes that wait for consumer acknowledgement.
The compatibility checker called no breaking change safe: 0 false negatives on 51 hand-labelled changes, 2 of 51 false alarms per direction.
-
lakehouse-lab
Iceberg on MinIO with Apache Polaris vending table-scoped credentials to Trino and StarRocks, plus a dbt and Airflow medallion pipeline.
rollback_to_snapshot recovered 909,968 deleted rows in 589 ms; expire_snapshots then destroyed that same safety net.
AI systems
7 repositories
-
evalproof
A local-first LLM evaluation tool that calibrates the judge, corrects its pass rate, and gates CI only on significant regressions.
Judge pass rate fell 4.8 pts, but p = 0.39 at n = 83, so the gate passed with a warning.
-
governed-agent
An analytics agent whose output becomes state only after schema, policy and grounding checks and, where required, human approval.
48/54 answers matched the key; 3 of 37 published (8%) were wrong, real numbers answering a different question.
-
realtime-ml
Real-time fraud scoring with Feast, LightGBM, MLflow and FastAPI shadow serving, built to measure leakage, training-serving skew and label delay.
The naive training join scored validation AUC-PR 0.746, then 0.363 on the next two weeks; point-in-time scored 0.681 then 0.941.
-
retrieval-bench
Vector and hybrid retrieval benchmark: pgvector, Qdrant and OpenSearch on the same embeddings, measuring recall, latency, filtering, build cost and nDCG.
At a 10% filter, pgvector HNSW post-filtering returns 3.9 rows instead of 10, and nothing errors.
-
short-text-triage
Seeded multilingual short-text triage lab: normalize to a key, embed each distinct key once, cluster with HDBSCAN, assign by margin.
A margin rule reached 95.2% precision against 66.2% for the best absolute threshold; labelling per cluster needed 125 LLM calls, not 20,000.
-
prevalence-shift
Seeded Monte Carlo showing why counting predicted labels flattens a trend, and how ACC, SLD and prediction-powered inference report prevalence honestly.
A 95%-accurate classifier counted a rise from 10% to 25% as 8% to 17%, a trend slope 0.628 of the true one.
-
rag-pipeline
Scrape-to-RAG lakehouse: Airflow, Spark Bronze/Silver/Gold on MinIO, DuckDB catalog, Chroma embeddings and a FastAPI RAG API on Ollama.
No published measurement yet.
Databases and distributed systems
3 repositories
-
distributed-db-lab
Scaling PostgreSQL out with Citus, time-series with TimescaleDB, and a 2×2 replicated ClickHouse cluster, with rebalances and failures measured.
Added a Citus worker under write load: 7 shard-group moves in 37.1 s, 977,368 operations, 0 errors.
-
db-failover-lab
Eight failover scenarios on multi-node clusters of PostgreSQL 18, Redis, MongoDB, ScyllaDB, SolrCloud and Galera, one container per node.
Concurrent UPDATEs of one row from two Galera masters converged: 300 / 300 / 300 on all three nodes.
-
postgres-ha-lab
PostgreSQL 18 HA on three amd64 Rocky 8 VMs: Patroni, etcd and pgBackRest over mutual TLS, with failover and restore performed.
After SIGKILL of the leader, Patroni promoted within a 10–20 s window; PITR returned 5 of 5 rows of a dropped table.
Delivery and infrastructure
1 repository
-
platform-deployments
GitOps platform for governed-agent on a three-node kind cluster: Argo CD, Argo Rollouts canaries gated by Prometheus, CloudNativePG, Kyverno.
Canary analysis aborted every bad release in six runs, 30–31 s after the bad pod became Ready (median).
Upstream
Contributions to projects I do not maintain · state as of October 14, 2026
-
awslabs/spark-sql-kinesis-connector #99
Pull request
Adds an opt-in option, kinesis.stopReadWhenMillisBehindLatestIsZero, so a shard's task stops once caught up instead of waiting out empty polls.
-
citusdata/citus #7333
Reproduction and offered patch
Confirmed the bug still reproduces on Citus 14.2.0 with PostgreSQL 17.11, adding a minimal repro, root cause and an offered patch.
-
citusdata/citus #4776
Issue
Reported that adding a primary key without a constraint name failed on a distributed table; Citus engineers later implemented support.
-
roncemer/spark-sql-kinesis #3
Pull request
Moved the connector's build to Scala binary 2.13, fixing conflicting spark-tags cross-version suffixes on Spark 3.2.1.