Multi-engine HA lab: eight failover scenarios

Context

This is a lab, not a production system, and I would rather say that in the first sentence than bury it. It is a rig I built to re-earn hands-on failover knowledge across engines I do not run daily — by killing processes and reading what came back, rather than quoting a docs page.

Constraints

One host, 8 CPU and ~12.5 GB of RAM, which is why each cluster came up, got exercised and got torn down one at a time. No cross-engine load test. No real network partitions — a single Docker host cannot give you one. 2026 engine versions throughout, which meant meeting each version’s current surprises rather than the ones in older tutorials.

Architecture

Nine Docker Compose stacks: mongo-ha, mongo-shard, scylla-ha, solr-ha, galera-ha, redis-cluster, redis-sentinel, pg-replication, observability. Each stands up a real multi-node cluster, gets seeded, gets broken deliberately, and gets checked for what survived.

Decisions and trade-offs

Decision	Alternative considered	Why	What it cost us
Docker Compose on one host	A VM per node in the cloud	Quorum, election and promotion behavior is observable at this scale, at zero cost	Node-level network partitions and disk failure are not testable at all
One cluster at a time	Everything up simultaneously	12.5 GB of RAM	No contention testing; every result is an isolated run
docker kill rather than graceful stop	Graceful shutdown	Ungraceful loss is the case that actually breaks quorum, and the difference is the whole lesson	Slower resets; some engines needed manual recovery
Current 2026 versions	Older LTS releases	I wanted today’s failure modes	Version-specific traps cost real time (see below)

Results

8 of 8 scenarios in my own test plan passed. I wrote the plan, so the pass rate measures execution, not difficulty — owning that is more useful than implying an external benchmark. Which is also why the table below reports what each engine measured rather than a PASS column: a constant column carries about one bit, and the numbers are the part you can argue with.

Engine	Topology	What I did to it	Measured	Result	What it taught me
PostgreSQL18	Primary + hot standby	pg_promote()	383µsreplay lag	replay lag 383 µs	Timeline advanced 1 → 2 on promotion. write_lag measured 104 µs on the same run.
Redis8	Sentinel, quorum 2	SIGKILL primary	~5sto writable	promoted in ~5 s	Old primary was auto-demoted to replica on rejoin rather than fighting for the role.
Redis8	Cluster, 3 primaries + 3 replicas	SIGKILL primary	2 → 7config epoch	0 keys lost	All 16,384 slots stayed covered. No acknowledged write was lost.
MariaDB + Galera11.8	3-node multi-master	150 concurrent cross-master UPDATEs	300converged on all 3	300 / 300 / 300	Certification failures logged on two nodes — the counter still agreed everywhere afterwards.
MongoDB8.0	3-node replica set, 2 shards	100,000 docs on a hashed key	49,607 / 50,393shard split	49,607 / 50,393	A targeted query hit one shard; an unqualified filter scatter-gathered to both.
ScyllaDB2026.1	3-node, RF=3, 256 vnodes	Node loss at QUORUM	3 / 3replicas held key	reads served at RF=3	Restore needed the table's upload/ subdirectory; a TRUNCATE marker hides older SSTables.
SolrCloud9	ZooKeeper + 2 nodes, 2×2	Node loss	6numFound, still served	kept answering	Cluster went ORANGE, kept answering, and returned to GREEN unattended.
Prometheus + Grafana	Exporters per engine	Golden signals	500 / 500keyspace hits/misses	all engines observed	Every engine above was observed through the same dashboard rather than by hand.
Network partition	—	Asymmetric partition	not testableon one host	untested	One Docker host cannot produce a split network. Every failover above is process death, not partition — and partition is the case that actually causes split-brain. This row is the limit of the rig.

The last row is the limit of the rig, not a result. One Docker host cannot produce a split network, so every failover here is process death — and partition is the case that actually causes split-brain.

What went wrong

The passes are the boring part. These three cost the most time, and each one is a version-specific trap that no documentation page warned me about.

1. ScyllaDB

   The third node would not start, and the error was about the kernel

   Two nodes came up fine. The third died on system does not satisfy minimum AIO requirements — Seastar reserves async I/O contexts per reactor, and the Docker VM’s fs.aio-max-nr ceiling fit two of them. It is a VM-wide sysctl, and it does not survive a Docker Desktop restart.

2. ScyllaDB

   A valid snapshot restored zero rows, three times, with no error

   nodetool refresh only reads the table’s upload/ subdirectory. TRUNCATE writes a marker that hides every older SSTable even after you put them back. And recreating a table gives it a new UUID, so the directory you just copied into belongs to a dead table.

3. PostgreSQL 18

   Every compose file copied from a PG 12–17 guide breaks

   The image wants the volume at /var/lib/postgresql with PGDATA in an 18/docker subdirectory, not at /var/lib/postgresql/data. The change exists so pg_upgrade --link can work. The failure mode is a container that starts and then refuses the mount.

Running it

Nothing runs continuously — a stack goes up, gets exercised, comes down. The three traps that cost the most time: ScyllaDB’s third node refused to start until fs.aio-max-nr was raised on the Docker VM; Solr backups returned “access denied” until the location was in solr.allowPaths and owned by the solr user; and PostgreSQL 18 rejects a volume at /var/lib/postgresql/data, wanting the mount at /var/lib/postgresql with data in an 18/docker subdirectory.

What I’d do differently

No sustained load. Every failover happened on an idle cluster, which is the easy case. A background writer during the kill would make the numbers mean considerably more.

Partitions were simulated, not real. One host means I tested process death, not a split network. A second machine plus tc/netem would test the case that actually causes split-brain.

It should have been one script. Several runs were hand-driven, so reproducing a result means re-reading my own notes.

Each scenario is being written up properly, with commands and output, in the tutorial series.