Basant Bhattarai

Navigate

Data systems, database reliability, and governed AI—documented from production and measured labs.

Topic

Distributed databases

Quorum, sharding, failover and backup across MongoDB, ScyllaDB, Redis, MariaDB/Galera and Solr — measured on a rig rather than quoted from a docs page.

Every distributed database advertises high availability, and the claim is usually true in a sense that will not help you at two in the morning. So I built a rig — MongoDB, ScyllaDB, SolrCloud, MariaDB with Galera, Redis in both Cluster and Sentinel modes, and PostgreSQL — and killed the primary in each while measuring what actually happened and how long it took. 8 of 8 failover scenarios passed; Redis Sentinel promoted a replica to writable in ~5 s. Everything in this topic is a number from that rig, published with the test plan attached so you can reproduce it or disagree with it.

6 posts

All writing

Essay Reference systems, measured Measured on my own hardware

Citus removed the node. Its cleanup records stayed behind.

Draining a Citus worker and removing it straight away left cleanup records for a node that no longer existed. A later rebalance retried one move 33 times.

  • Distributed databases
  • PostgreSQL

11 min read

Tutorial The lakehouse spine Measured on my own hardware

A lakehouse spine: Trino, Iceberg and Polaris, with credentials that expire

Build Trino 483 on Iceberg with Apache Polaris as the REST catalog, then prove the vended S3 credentials cannot read the table next door.

  • Iceberg & the lakehouse
  • Distributed databases

~45 min to a working stack; the credential-vending section is the part that takes the time

Tutorial RHEL 8 Postgres lab on Apple Silicon Follow-along runbook

pgBackRest against a Patroni cluster: archive, rebuild, and restore

Wire pgBackRest into a running Patroni cluster over mutual TLS with Ansible, then prove it: full backup, replica rebuilt from the repo, and a point-in-time restore.

  • PostgreSQL
  • Distributed databases

~90 min on an existing cluster; the restore drill itself is ~15 min

Tutorial RHEL 8 Postgres lab on Apple Silicon Follow-along runbook

Patroni HA on PostgreSQL 18 with etcd, proven by killing the primary

Build a two-node Patroni cluster on Rocky 8 under QEMU, then SIGKILL the leader and measure the promotion, the timeline bump, and the pg_rewind rejoin.

  • PostgreSQL
  • Distributed databases

~2–3 h first bring-up (TCG is slow); ~5 min for the failover itself

Tutorial The failover lab Measured on my own hardware

nodetool refresh only reads the live table's upload/ directory

Why a valid ScyllaDB snapshot restore can return zero rows with no error, the restore that works after a TRUNCATE, and the AIO limit that stops node three.

  • Distributed databases

~40 min hands-on

Tutorial The failover lab Measured on my own hardware

A failover lab: six database engines, eight scenarios, every one run

Six database engines, eight failover scenarios, one 8-CPU Docker rig. The results matrix, the headline measurements, and how to run the first scenario.

  • Distributed databases
  • PostgreSQL

~45 min hands-on

Other topics