I first built this stack in January 2024 for a client who wanted to see whether Trino over Iceberg could replace a warehouse they were outgrowing. That build pinned Trino 437, tabulario/iceberg-rest:0.2.0 as the catalog, and Hive Metastore underneath it.
Every one of those three has since been superseded. Trino is on 483. The tabulario image is archived. Apache Polaris graduated incubation and is on 1.7.0. So this post is the same architecture, rebuilt on current versions and measured, and the interesting part turned out to be the piece the 2024 build did not have at all: credentials that expire.
The claim I wanted to test is the one that justifies a REST catalog over a metastore. Hive Metastore tells you where a table is. Polaris decides whether you may touch it, and hands out short-lived S3 credentials scoped to that table’s prefix. If that works, Trino never holds a standing key. If it does not, you have added a service and gained a lookup.
\n\nIt works. Proving it took five failures, and four of them fail in a way that points somewhere else.
\n\nThree services before Trino enters the picture: MinIO for objects, Postgres for Polaris’s own metadata, and Polaris itself.
\n\nservices:\n minio:\n image: minio/minio:RELEASE.2025-09-07T16-13-09Z\n command: server /data --console-address \":9001\"\n environment:\n MINIO_ROOT_USER: lakehouse\n MINIO_ROOT_PASSWORD: lakehouse-dev-only\n ports: [\"9000:9000\", \"9001:9001\"]\n healthcheck:\n test: [\"CMD\", \"mc\", \"ready\", \"local\"]\n interval: 5s\n retries: 20\n\n polaris-db:\n image: postgres:18.1-alpine\n environment:\n POSTGRES_USER: polaris\n POSTGRES_PASSWORD: polaris-dev-only\n POSTGRES_DB: polaris\n healthcheck:\n test: [\"CMD-SHELL\", \"pg_isready -U polaris -d polaris\"]\n interval: 5s\n retries: 20\nPolaris defaults to in-memory persistence, which loses every namespace on restart. A lab that forgets its catalog when you reboot is not reproducible, so this one runs on Postgres.
\n\nPOLARIS_BOOTSTRAP_CREDENTIALS does not create the schema. Point Polaris at an empty Postgres with that variable set and it starts cleanly, passes its healthcheck, and then fails every single request:
Failed to retrieve polaris entity due to Failed due to\n'ERROR: relation \"polaris_schema.entities\" does not exist'\nThe schema is the admin tool’s job. And the admin tool does not read POLARIS_BOOTSTRAP_CREDENTIALS — that variable belongs to the server. Passing only -r exits 3 with Specify either --credentials or --print-credentials, so the same credential has to be repeated in the tool’s own form.
polaris-bootstrap:\n image: apache/polaris-admin-tool:1.7.0\n depends_on:\n polaris-db: { condition: service_healthy }\n environment:\n polaris.persistence.type: relational-jdbc\n quarkus.datasource.jdbc.url: jdbc:postgresql://polaris-db:5432/polaris\n quarkus.datasource.username: polaris\n quarkus.datasource.password: polaris-dev-only\n command: [\"bootstrap\", \"-r\", \"POLARIS\", \"-c\", \"POLARIS,root,s3cr3t-dev-only\"]\nBootstrap is idempotent — running it against an already-bootstrapped realm is a no-op — which is why it can sit in depends_on rather than in a script somebody has to remember not to run twice.
Verify. Realm 'POLARIS' successfully bootstrapped. on first run, and no error on the second.
Now the server itself:
\n\n polaris:\n image: apache/polaris:1.7.0\n depends_on:\n polaris-bootstrap: { condition: service_completed_successfully }\n ports: [\"8181:8181\", \"8182:8182\"]\n environment:\n POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t-dev-only\n polaris.realm-context.realms: POLARIS\n polaris.persistence.type: relational-jdbc\n quarkus.datasource.jdbc.url: jdbc:postgresql://polaris-db:5432/polaris\n quarkus.datasource.username: polaris\n quarkus.datasource.password: polaris-dev-only\n polaris.features.\"ALLOW_INSECURE_STORAGE_TYPES\": \"true\"\n polaris.features.\"SUPPORTED_CATALOG_STORAGE_TYPES\": \"[\\\"S3\\\"]\"\n polaris.readiness.ignore-severe-issues: \"true\"\n healthcheck:\n test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8182/q/health/ready\"]\n interval: 10s\n retries: 30\n/healthcheck 404s. The upstream Spark quickstart still uses http://localhost:8182/healthcheck. On 1.7.0 that path is gone — Quarkus serves health under /q/health. A healthcheck aimed at a 404 leaves the container in health: starting forever, and every service with a service_healthy dependency silently never starts. Nothing in the logs mentions it, because from Polaris’s point of view nothing is wrong.
Polaris is an authorisation boundary, and three objects have to line up before Trino can write a row:
\n\nprincipal → principal_role → catalog_role → catalog\nMiss the last link and Trino authenticates perfectly, then sees no namespaces at all — which reads exactly like a connection problem and is not one.
\n\nTOKEN=$(curl -s -X POST http://localhost:8181/api/catalog/v1/oauth/tokens \\\n -d grant_type=client_credentials -d client_id=root \\\n -d client_secret=s3cr3t-dev-only -d 'scope=PRINCIPAL_ROLE:ALL' \\\n | sed -n 's/.*\"access_token\":\"\\([^\"]*\\)\".*/\\1/p')\n\ncurl -X POST http://localhost:8181/api/management/v1/catalogs \\\n -H \"Authorization: Bearer $TOKEN\" -H 'Content-Type: application/json' -d '{\n \"catalog\": {\n \"name\": \"lakehouse\", \"type\": \"INTERNAL\",\n \"properties\": {\"default-base-location\": \"s3://warehouse/lakehouse\"},\n \"storageConfigInfo\": {\n \"storageType\": \"S3\",\n \"allowedLocations\": [\"s3://warehouse/lakehouse\"],\n \"endpoint\": \"http://minio:9000\",\n \"endpointInternal\": \"http://minio:9000\",\n \"pathStyleAccess\": true,\n \"stsUnavailable\": false,\n \"stsEndpoint\": \"http://minio:9000\",\n \"roleArn\": \"arn:aws:iam::123456789012:role/polaris\",\n \"kmsUnavailable\": true,\n \"region\": \"us-east-1\"\n }}}'\nFour things in that storage config are worth naming, because three of them cost me time.
\n\nThere is no S3_COMPATIBLE storage type. The enum is S3 | GCS | AZURE | FILE. Sending S3_COMPATIBLE — which is what the name suggests for MinIO — returns a bare HTTP 400 with an empty body. No message, no field name.
The S3 knobs are flat camelCase. endpoint and pathStyleAccess, not the dotted s3.endpoint keys you write in Iceberg client configuration. Same concepts, different layer, different spelling.
roleArn is required and ignored. MinIO returns an empty AssumedRoleUser.Arn, but the STS API shape demands one and Polaris will not call STS without it.
kmsUnavailable: true is the one that is in no quickstart. More on that in a moment.
Then the roles:
\n\ncurl -X POST .../catalogs/lakehouse/catalog-roles -d '{\"catalogRole\":{\"name\":\"engineer\"}}'\ncurl -X PUT .../catalog-roles/engineer/grants -d '{\"grant\":{\"type\":\"catalog\",\"privilege\":\"CATALOG_MANAGE_CONTENT\"}}'\ncurl -X POST .../principal-roles -d '{\"principalRole\":{\"name\":\"data_engineer\"}}'\ncurl -X PUT .../principal-roles/data_engineer/catalog-roles/lakehouse -d '{\"catalogRole\":{\"name\":\"engineer\"}}'\ncurl -X PUT .../principals/root/principal-roles -d '{\"principalRole\":{\"name\":\"data_engineer\"}}'\nVerify. Every call returns 201, and a re-run returns 409 Conflict rather than erroring — the script should be safe to run twice.
connector.name=iceberg\niceberg.catalog.type=rest\niceberg.rest-catalog.uri=http://polaris:8181/api/catalog\niceberg.rest-catalog.warehouse=lakehouse\niceberg.rest-catalog.security=OAUTH2\niceberg.rest-catalog.oauth2.credential=root:s3cr3t-dev-only\niceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL\niceberg.rest-catalog.vended-credentials-enabled=true\n\niceberg.file-format=PARQUET\n\nfs.native-s3.enabled=true\ns3.endpoint=http://minio:9000\ns3.region=us-east-1\ns3.path-style-access=true\nNote what is absent: any S3 access key. Trino authenticates to Polaris as an OAuth2 principal; Polaris does the rest.
\n\nDo not add iceberg.register-table-procedure.enabled=true. With vended credentials it makes Trino refuse to start:
Using the `register_table` procedure with vended credentials is currently not supported\nThat is a startup failure, not a query failure. The coordinator exits 100, every dependent container fails its healthcheck, and the offending catalog is named only inside the stack trace.
\nThis is the part worth reading twice, because the failure modes are actively misleading.
\n\nPolaris mints scoped credentials by calling sts:AssumeRole. Point it at MinIO with the root credentials and you get:
<Code>InvalidParameterValue</Code>\n<Message>Unsupported action AssumeRole</Message>\nThat message says the action is unsupported, so the obvious conclusion is that MinIO has no STS. It does. It refuses AssumeRole for the root account specifically. Create a MinIO user and the same call succeeds:
\n\nmc admin user add local polaris-svc polaris-svc-dev-only\nmc admin policy attach local readwrite --user polaris-svc\nThen point Polaris at that identity rather than root:
\n\n AWS_ACCESS_KEY_ID: polaris-svc\n AWS_SECRET_ACCESS_KEY: polaris-svc-dev-only\nVerify. aws sts assume-role --endpoint-url http://minio:9000 --role-arn arn:aws:iam::123456789012:role/polaris --role-session-name probe returns a Credentials block with an Expiration.
With STS reachable, CREATE TABLE AS SELECT succeeded. 1.5 million rows, 84 objects in the bucket. Then every SELECT failed:
Query ... failed: Failed to load table: orders in bronze namespace\nThe real error is four layers down, in Polaris’s log:
\n\ninvalid resource 'arn:aws:kms:us-east-1:123456789012:key/*'\n(Service: Sts, Status Code: 400)\nPolaris builds a session policy for the AssumeRole call and, by default, includes a KMS statement. MinIO’s policy engine does not understand KMS ARNs and rejects the whole request. kmsUnavailable: true on the catalog stops it being added.
This is the worst failure in the series. The data lands. The bucket fills. Only reads break, and the message Trino prints mentions neither KMS, nor STS, nor Polaris.
\n\nWhy writes survived and reads did not. The write path obtained credentials through a code path that did not need the offending policy; loadTable did. If you are debugging this, the tell is that SHOW TABLES works and SELECT does not.
Polaris returns a prefix field with each vended credential. That is a label, not evidence. The only thing that settles it is taking the credential and trying to read something it should not reach.
# credentials for bronze.orders only\ncurl -H \"Authorization: Bearer $TOKEN\" \\\n -H 'X-Iceberg-Access-Delegation: vended-credentials' \\\n \"$POLARIS/api/catalog/v1/lakehouse/namespaces/bronze/tables/orders\"\ncredential vended for bronze.orders\n prefix: s3://warehouse/lakehouse/bronze/orders-964137655cf046eb8a7eeabf4b97c257\n\nusing only that credential:\n own prefix ALLOW 84 objects\n sibling bronze.customers DENY AccessDenied\n whole warehouse bucket DENY AccessDenied\nAlongside the keys, Polaris returns s3.session-token and s3.session-token-expires-at-ms. The credential is short-lived as well as narrow.
Verify. Your own prefix lists objects; the sibling table and the bucket root both return AccessDenied.
Two things make this look broken when it is not. The vended prefix carries the table UUID Iceberg assigns at create time — bronze/orders-964137655cf0…, not bronze/orders. Probing the tidy path denies, and reads as a scoping failure. And aws s3 ls needs both a trailing slash and a region set; without them it fails in a way indistinguishable from AccessDenied at a glance.
A partitioned table, written through the whole chain:
\n\nCREATE TABLE lakehouse.bronze.orders\nWITH (partitioning = ARRAY['month(orderdate)']) AS\nSELECT orderkey, custkey, orderstatus, totalprice, orderdate, orderpriority, clerk\nFROM tpch.sf1.orders;\n1,500,000 rows in 5.4 s, landing as 80 Parquet files, 2 Avro manifests, one Puffin .stats file and one metadata.json — 16 MiB total.
Query timings, median of 7 server-side runs read from system.runtime.queries:
| query | \ntpch generator | \nIceberg on MinIO | \n
|---|---|---|
| scan + aggregate over 1.5M rows | \n238 ms | \n183 ms | \n
| one-month partition predicate | \n615 ms | \n88 ms | \n
| non-partition predicate | \n595 ms | \n202 ms | \n
tpch is a generator, not a storage format, and never touches the network — it is a floor rather than a fair “before”. The 7× gap on the partition predicate is pruning: Iceberg reads one manifest entry while tpch regenerates all 1.5 million rows.
Measure server-side. My first numbers were wall-clock around the Trino CLI: 603 ms for the query that actually takes 88 ms. The other 515 ms was docker compose exec plus the CLI’s own JVM startup — six times the figure I was trying to measure.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
relation \"polaris_schema.entities\" does not exist | \n Schema never created | \nRun polaris-admin-tool bootstrap before the server | \n
| Bootstrap exits 3 | \nAdmin tool ignores POLARIS_BOOTSTRAP_CREDENTIALS | \n Pass -c realm,clientId,clientSecret explicitly | \n
Container stuck health: starting | \n Healthcheck points at /healthcheck | \n Use /q/health/ready on port 8182 | \n
HTTP 400, empty body, on catalog create | \n storageType: S3_COMPATIBLE | \n Use S3 with flat endpoint / pathStyleAccess | \n
| Trino exits 100 at startup | \nregister-table-procedure + vending | \n Remove the procedure flag | \n
Unsupported action AssumeRole | \n Using MinIO root | \nCreate a MinIO user and use it | \n
| Writes succeed, all reads fail | \nKMS statement in the session policy | \nkmsUnavailable: true on the catalog | \n
Credential vending was requested … but no credentials are available | \n stsUnavailable: true | \n Enable STS, or disable vending and accept standing keys | \n
docker compose down -v\nThe operating rule this lab produced: a REST catalog is only worth the extra service if vending actually works. Turn vending off and Polaris becomes a metadata lookup with a permissions table in front of it, while every Trino user still holds a key to the whole bucket. That is not obviously better than Hive Metastore.
\n\nTurn it on and the boundary is real, but you inherit an STS dependency that must be exercised in whatever object store you actually run. Against MinIO that meant a service user and one undocumented flag. Against a different S3-compatible store it will mean something else — and the way to find out is to take a vended credential and try to read the table next door.
\n\nPart 2 builds bronze, silver and gold on top of this with dbt-trino, where the interesting failure is a partition spec that dbt accepts and silently discards.
\n", "summary": "Build Trino 483 on Iceberg with Apache Polaris as the REST catalog, then prove the vended S3 credentials cannot read the table next door.", "date_published": "2026-08-16T12:00:00+05:45", "date_modified": "2026-08-16T00:00:00+05:45", "tags": ["iceberg-lakehouse","distributed-databases"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/dbt-trino-iceberg-medallion-airflow/", "url": "https://basantbhattarai.com.np/writing/dbt-trino-iceberg-medallion-airflow/", "title": "Bronze, silver, gold on Iceberg: the dbt config that is silently ignored", "content_html": "Every lakehouse article asserts that Iceberg gives you row-level MERGE, so incremental models rewrite only what changed instead of replacing the table. That is the difference between a pipeline that scales and one that gets slower every day.
\n\nIt is also the kind of claim that is easy to state and rarely shown. dbt will happily print MERGE (25_000 rows) whether it rewrote one file or all eighty. The evidence is not in dbt’s output — it is in Iceberg’s $snapshots metadata table, which records exactly how many data files each commit added and deleted.
The stack is a rebuild of a January 2024 client project on current versions — Trino 437 became 483, Hive Metastore became Apache Polaris. This post builds bronze → silver → gold on the spine from part one, hands it to Airflow, and then goes and reads that table.
\n\nFour things went wrong on the way. Three of them failed silently.
\n\nlakehouse:\n target: dev\n outputs:\n dev:\n type: trino\n host: trino-coordinator\n port: 8080\n user: dbt\n catalog: lakehouse\n schema: silver\n http_scheme: http\n threads: 4\nNote what is not in this file: any S3 credential. dbt talks to Trino, Trino asks Polaris, Polaris vends per table. The whole chain from part one holds.
\n\nFROM python:3.12-slim\n# git is not decoration: dbt shells out to it for package installs and to stamp\n# the run manifest, and `dbt debug` fails a check without it.\nRUN apt-get update && apt-get install -y --no-install-recommends git \\\n && rm -rf /var/lib/apt/lists/*\nRUN pip install --no-cache-dir \"dbt-trino==1.10.3\"\nWORKDIR /dbt\nENTRYPOINT [\"dbt\"]\nVerify. dbt debug reports Connection test: OK connection ok.
models:\n lakehouse:\n +table_type: iceberg\n silver:\n +schema: silver\n +materialized: incremental\n gold:\n +schema: gold\n +materialized: table\nFirst run:
\n\n1 of 2 OK created sql incremental model silver_silver.orders_enriched\n2 of 2 OK created sql table model silver_gold.daily_revenue\nBoth models built. Both queries worked. Both namespaces in Polaris were wrong.
\n\ndbt’s default generate_schema_name returns <profile_schema>_<custom_schema>. That default exists so several developers can share a warehouse without colliding, and it is the right default for that. A lakehouse whose layer names are the contract does not want it:
{% macro generate_schema_name(custom_schema_name, node) -%}\n {%- if custom_schema_name is none -%}\n {{ target.schema }}\n {%- else -%}\n {{ custom_schema_name | trim }}\n {%- endif -%}\n{%- endmacro %}\nVerify. SHOW SCHEMAS FROM lakehouse lists bronze, silver, gold — and not silver_silver.
This is the one that matters.
\n\n{{\n config(\n materialized = 'incremental',\n incremental_strategy = 'merge',\n unique_key = 'orderkey',\n partitioning = [\"month(orderdate)\"],\n table_type = 'iceberg'\n )\n}}\nThat runs. dbt reports success. The model builds 1.5 million rows. And the table is not partitioned:
\n\nSHOW CREATE TABLE lakehouse.silver.orders_enriched;\nWITH (\n format = 'PARQUET',\n format_version = 2,\n location = 's3://warehouse/lakehouse/silver/orders_enriched-09054d82…'\n)\nNo partitioning property. dbt-trino expects it inside properties, as a literal Trino SQL fragment:
{{\n config(\n materialized = 'incremental',\n incremental_strategy = 'merge',\n unique_key = 'orderkey',\n table_type = 'iceberg',\n properties = {\n \"partitioning\": \"ARRAY['month(orderdate)']\"\n }\n )\n}}\nThe cost of getting it wrong, on the same rows and the same predicate:
\n\n| table | \ndata files | \npartitions | \nmedian query | \n
|---|---|---|---|
partitioning at config level (ignored) | \n 2 | \n0 | \n131 ms | \n
properties = {\"partitioning\": …} | \n 80 | \n80 | \n99 ms | \n
-- comments are a compilation error inside config(). That block is a Jinja expression, not SQL. A -- like this line inside it produces invalid syntax for function call expression, pointing at the config( line rather than the comment. Use {# … #} outside the call.
Verify. SHOW CREATE TABLE includes partitioning = ARRAY['month(orderdate)'], and SELECT count(*) FROM \"orders_enriched$partitions\" returns 80 rather than 0.
With partitioning fixed, dbt run --full-refresh failed:
TrinoExternalError(type=EXTERNAL, name=ICEBERG_CATALOG_ERROR,\n message=\"Failed to drop table 'orders_enriched__dbt_backup'\")\nNothing in that message is the cause. It is in Polaris’s log:
\n\nUnable to purge entity: orders_enriched__dbt_backup. To enable this feature, set\nthe Polaris configuration DROP_WITH_PURGE_ENABLED\n…\n\"DELETE /api/catalog/v1/lakehouse/namespaces/silver/tables/\n orders_enriched__dbt_backup?purgeRequested=true HTTP/1.1\" 403\n--full-refresh on an incremental model builds an <model>__dbt_backup table and then drops it. Trino asks for a purging drop; Polaris refuses by default, because purge deletes the data files and not merely the catalog entry. That is a defensible default. It is also one that leaves a stuck __dbt_backup table you cannot remove by hand either.
polaris.features.\"DROP_WITH_PURGE_ENABLED\": \"true\"\nVerify. DROP TABLE lakehouse.silver.orders_enriched__dbt_backup succeeds, and dbt run --full-refresh completes.
Now the actual claim. Baseline after a full refresh, then 25,000 new rows landed in bronze, then an incremental run:
\n\nSELECT operation,\n element_at(summary,'added-data-files') AS added,\n element_at(summary,'deleted-data-files') AS deleted,\n element_at(summary,'added-records') AS added_rows,\n element_at(summary,'total-data-files') AS total\nFROM lakehouse.silver.\"orders_enriched$snapshots\"\nORDER BY committed_at;\n operation | added | deleted | added_rows | total\n-----------+-------+---------+------------+-------\n append | 80 | NULL | 1500000 | 80\n append | 1 | NULL | 25000 | 81\ndbt reported MERGE (25_000 rows) in 2.66s. The commit added one data file out of eighty-one and deleted none. That is the claim, verified from the table’s own metadata rather than from the tool’s summary line.
Note the operation is append, not overwrite. Every incoming key was new, so Iceberg degraded the MERGE to an append — it had nothing to rewrite. A MERGE that genuinely matches existing rows shows a non-null deleted-data-files, because Iceberg rewrites the files containing the matched rows. If you are testing MERGE behaviour, feed it keys that already exist, or you will measure the wrong thing and conclude MERGE is free.
Use element_at(summary, …) rather than summary['…']: the bracket form throws Key not present in map on any snapshot where that key is absent, which is most of them.
@dag(\n dag_id=\"lakehouse_pipeline\",\n schedule=\"@daily\",\n start_date=pendulum.datetime(2026, 8, 1, tz=\"UTC\"),\n catchup=False,\n max_active_runs=1,\n)\ndef lakehouse_pipeline():\n @task\n def land(**context) -> int:\n run = context[\"dag_run\"]\n stamp = getattr(run, \"logical_date\", None) or run.run_after\n ...\n\n @task.bash(cwd=\"/opt/airflow/dbt\")\n def transform() -> str:\n return \"dbt build --profiles-dir /opt/airflow/dbt\"\n\n @task\n def maintain() -> dict[str, int]:\n ...\n\n land() >> transform() >> maintain()\ncontext[\"logical_date\"] raises KeyError on Airflow 3. AIP-83 made logical date optional: a manually triggered run has none at all, so the key is simply absent and the task dies before it reaches Trino. dag_run.run_after is always populated. This is a quiet migration break — the DAG parses, imports cleanly, and only fails when someone presses Trigger.
dbt lives inside the Airflow image rather than behind a DockerOperator. The alternative means mounting the host Docker socket into Airflow, which is a real privilege escalation for the sake of avoiding one pip install. Pinning the same dbt-trino==1.10.3 in both images means a DAG run and a manual dbt run cannot drift.
Verify. All three tasks report success, and land, transform, maintain appear in that order.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
Models built into silver_silver | \n dbt’s default schema concatenation | \nOverride generate_schema_name | \n
SHOW CREATE TABLE has no partitioning | \n partitioning at config level | \n Move it into properties as a SQL fragment | \n
invalid syntax for function call expression | \n -- comment inside config() | \n Use {# … #} outside the call | \n
Failed to drop table '…__dbt_backup' | \n Polaris refuses purging drops | \nDROP_WITH_PURGE_ENABLED: \"true\" | \n
KeyError: 'logical_date' | \n AIP-83 made it optional | \nFall back to dag_run.run_after | \n
Key not present in map on $snapshots | \n Bracket access to an absent summary key | \nelement_at(summary, '…') | \n
docker compose --profile orchestration --profile tools down -v\nThe rule this produced: verify the physical layout, not the tool’s summary line. Three of the four failures here reported success. dbt said the model built, and it had — unpartitioned. dbt said MERGE, and it was, but only because the keys happened not to collide.
SHOW CREATE TABLE and the $snapshots / $files metadata tables are the ground truth, and they are cheap to query. A CI check that asserts a silver table has the partition count you expect costs one query and catches a config key that everything else in the stack was willing to ignore.
Part 3 takes the same tables and asks the operational question: what happens after a few thousand of these runs, and can you get back a table somebody deleted.
\n", "summary": "Build a medallion pipeline with dbt-trino on Iceberg, orchestrate it with Airflow 3, and find the partition spec dbt accepts and discards without a warning.", "date_published": "2026-08-16T11:00:00+05:45", "date_modified": "2026-08-16T00:00:00+05:45", "tags": ["iceberg-lakehouse","data-quality"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/iceberg-compaction-snapshots-rollback/", "url": "https://basantbhattarai.com.np/writing/iceberg-compaction-snapshots-rollback/", "title": "Iceberg maintenance: 20 files into 1, and a rollback I actually performed", "content_html": "Parts one and two built a lakehouse and filled it. Both stopped at the point where it works, which is the point most lakehouse writing stops at.
\n\nThe question that decides whether the thing is still usable in a year is different: Iceberg appends a data file per write and a snapshot per commit. A table fed by an hourly pipeline accumulates both. Nobody notices for a month. Then queries get slow, and the cause is not the query.
\n\nThis post creates that problem deliberately, measures it, repairs it, and then does the other thing worth knowing: deletes most of a table and gets it back. The rig is the January 2024 client stack rebuilt on current versions; the maintenance questions are the ones that stack never got old enough to face.
\n\nTwenty small inserts into one partition — which is exactly what a five-minute micro-batch does over an hour and a half:
\n\n-- repeated 20x with a varying key offset\nINSERT INTO lakehouse.silver.orders_enriched\nSELECT orderkey + 200000000 + :i * 100000, custkey, 'micro-batch',\n CAST(1 AS bigint), 'O', totalprice, DATE '2001-06-15', 1, 'URGENT', clerk\nFROM tpch.sf1.orders LIMIT 5000;\nThe damage, read from Iceberg’s own metadata:
\n\nSELECT count(*) AS files,\n cast(sum(file_size_in_bytes)/1024 AS bigint) AS total_kb,\n cast(avg(file_size_in_bytes)/1024 AS bigint) AS avg_kb,\n sum(record_count) AS rows\nFROM lakehouse.silver.\"orders_enriched$files\"\nWHERE partition.orderdate_month = date_diff('month', DATE '1970-01-01', DATE '2001-06-01');\n files | total_kb | avg_kb | rows\n-------+----------+--------+--------\n 20 | 1006 | 50 | 100000\nTwenty files averaging 50 KB. Every query touching that month opens twenty objects, reads twenty sets of Parquet footers, and plans twenty splits — for 100,000 rows that would fit comfortably in one.
\n\nVerify. Median query time on that partition, seven server-side runs after two warm-ups: 184 ms.
\n\nALTER TABLE lakehouse.silver.orders_enriched EXECUTE optimize;\n1.4 seconds.
\n\n| \n | before | \nafter | \n
|---|---|---|
| data files in partition | \n20 | \n1 | \n
| total bytes | \n1,006 KB | \n430 KB | \n
| average file | \n50 KB | \n430 KB | \n
| median query on that partition | \n184 ms | \n69 ms | \n
The byte reduction is the part that surprised me. Compaction is usually described as a file-count fix, but total size fell 57% on identical rows — Parquet’s dictionary and run-length encodings work far better with more rows per row-group. You get the planning win and less to read.
\n\noptimize only merges files within a partition. Run against the full table — 82 partitions holding one file each — it completes successfully and changes nothing, producing no new snapshot at all. That looks exactly like a broken command. It is correct: there is nothing to combine, because compaction never merges across partition boundaries.
This is worth knowing before you schedule it, because it means a table partitioned too finely cannot be compacted out of a small-files problem. The repair there is the partition spec, not maintenance.
\nVerify. SELECT count(*) FROM \"orders_enriched$files\" WHERE partition.orderdate_month = … returns 1, and the $snapshots table has a new replace commit.
An unqualified DELETE — the kind that happens when a WHERE clause is pasted one line too high:
-- 1,645,000 rows before this\nDELETE FROM lakehouse.silver.orders_enriched WHERE orderdate < DATE '1996-01-01';\nDELETE: 909968 rows\nSELECT count(*) FROM lakehouse.silver.orders_enriched; -- 735,032\nJust over 55% of the table, gone, committed, with no transaction to roll back.
\n\nIceberg’s answer has two halves, and the first one is the one people forget. Before changing anything, you can read the old state:
\n\nSELECT count(*) FROM lakehouse.silver.orders_enriched\nFOR VERSION AS OF 307623642966580244;\n1645000\nThat is a read against a historical snapshot. Nothing is modified; you can diff the two, confirm the damage is what you think it is, and copy out a subset if a full rollback is too blunt.
\n\nThen the rollback:
\n\nALTER TABLE lakehouse.silver.orders_enriched\nEXECUTE rollback_to_snapshot(307623642966580244);\nreal 0m0.589s\nSELECT count(*) FROM lakehouse.silver.orders_enriched; -- 1,645,000\n909,968 rows recovered in 589 milliseconds.
\n\nThe speed is the point, and it is worth understanding why. No data moved. The deleted files were never removed from storage — the DELETE wrote a new snapshot that stopped referencing them. Rolling back rewrites the current-snapshot pointer in the catalog to an earlier manifest list. It is three orders of magnitude faster than the equivalent point-in-time restore in the pgBackRest lab, which had to physically restore a backup and replay WAL.
Verify. Row count matches the pre-incident total exactly, and $snapshots shows the rollback as the current snapshot.
The files survived the DELETE because a snapshot still referenced them. So the obvious question is what happens when that snapshot goes away.
ALTER TABLE lakehouse.silver.orders_enriched\nEXECUTE expire_snapshots(retention_threshold => '0s');\nSnapshot count went from 25 to 1. Then:
\n\nSELECT count(*) FROM lakehouse.silver.orders_enriched\nFOR VERSION AS OF 307623642966580244;\nQuery … failed: Iceberg snapshot ID does not exists: 307623642966580244\nThat is the whole lesson in one error message. expire_snapshots is the operation that destroys your ability to undo. Compaction leaves the old files behind, still referenced by history; expiry is what actually releases them, and it takes the rollback with it.
Which means the retention threshold is not a housekeeping setting. It is your recovery window, exactly as a pgBackRest retention policy is, and it should be chosen by the same conversation: how long before someone notices a bad load?
\n\nTrino guards this with a 7-day floor. Asking for less returns:
\n\nRetention specified (1.00m) is shorter than the minimum retention configured\nin the system (7.00d)\nOverriding it needs a session property — and the error message names the connector, iceberg.expire_snapshots_min_retention, while the property must be namespaced by your catalog. My catalog is called lakehouse, so:
trino --session lakehouse.expire_snapshots_min_retention=0s\nUsing the name from the error message gets Catalog 'iceberg' not found.
Both operations belong on a schedule, and the order is not arbitrary:
\n\n@task\ndef maintain() -> dict[str, int]:\n \"\"\"Compact, then expire.\n\n `optimize` writes new compacted files and leaves the originals behind,\n still referenced by older snapshots. Expiring afterwards is what actually\n releases them. Run expire first and compaction's own garbage stays until\n the next cycle.\n \"\"\"\n _run(\"ALTER TABLE … EXECUTE optimize\")\n _run(\"ALTER TABLE … EXECUTE expire_snapshots(retention_threshold => '7d')\")\nIn this lab maintain runs on every pass so the effect is visible quickly. In production it belongs on its own schedule: compaction rewrites files and competes with the pipeline for the same workers, and there is no reason for it to contend with the load it is cleaning up after.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
optimize succeeds, nothing changes | \n One file per partition already | \nNothing to fix; check the partition spec instead | \n
| Queries slowing with no query change | \nSmall files accumulating | \nSchedule optimize, measure $files count | \n
Retention specified … shorter than the minimum | \n Trino’s 7-day floor | \nSession property namespaced by catalog name | \n
Catalog 'iceberg' not found | \n Used the connector name from the error text | \nUse your catalog’s actual name | \n
Iceberg snapshot ID does not exists | \n Snapshot already expired | \nNothing to do — the window has closed | \n
Storage growing despite optimize | \n Old files still referenced by history | \nexpire_snapshots, after compaction | \n
docker compose down -v\nThree rules came out of this, and only one of them is about performance.
\n\nCompaction is a query-latency control, and it is cheap. 1.4 seconds to take a partition from 184 ms to 69 ms, and to shed 57% of its bytes. It should be scheduled from the day the table exists, not the day someone complains.
\n\noptimize cannot fix a bad partition spec. It never merges across partitions, so a table partitioned by day when it should be by month has a small-files problem that maintenance cannot reach.
Retention is a recovery policy, not housekeeping. The rollback in step 4 took 589 ms and recovered 909,968 rows — but only because the snapshot still existed. Whoever sets expire_snapshots(retention_threshold => …) is deciding how long you have to notice a mistake, and that is a decision worth making on purpose rather than inheriting from an example config.
Parts one to three built a lakehouse on Trino and never left it. That is a fine way to test Iceberg, and a poor way to test the catalog, because the argument for Polaris over Hive Metastore is not that it looks tables up faster.
\n\nThe argument is that the catalog is the system of record, so a second engine should see the tables the first one wrote — without an export, a copy, a CREATE EXTERNAL TABLE per table, or a second set of credentials.
That last clause is the part worth checking. Part one went to some trouble to make Trino hold no standing S3 key. If adding StarRocks means pasting the MinIO keys into a second config file, the boundary is gone and the whole exercise was decoration.
\n\nSo: attach StarRocks 4.1.4 to the same catalog, read the same tables, and see. The original 2024 build ran StarRocks 3.2 beside Trino 437 against Hive Metastore — same question, and the answer has changed.
\n\n starrocks:\n image: starrocks/allin1-ubuntu:4.1.4\n profiles: [\"engines\"]\n depends_on:\n polaris:\n condition: service_healthy\n ports:\n - \"9030:9030\" # MySQL protocol\n - \"8030:8030\" # FE HTTP\n healthcheck:\n test: [\"CMD-SHELL\", \"mysql -h127.0.0.1 -P9030 -uroot -e 'SELECT 1' || exit 1\"]\n interval: 15s\n timeout: 10s\n retries: 40\n start_period: 90s\nallin1 bundles the frontend and backend in one container. Production splits them; a lab that splits them spends 2 GB teaching nothing this post is about.
SELECT 1 succeeding does not mean StarRocks is ready. The FE accepts connections well before any BE registers, so the healthcheck above goes green and the first real query fails:
ERROR 1064 (HY000): Backend node not found. Check if any backend node is down.\nThe honest readiness signal is SHOW BACKENDS reporting Alive: true. If you are scripting a wait loop, poll that rather than a trivial query — otherwise you will write a retry that papers over a startup ordering bug you have not actually fixed.
Verify. SELECT current_version() returns 4.1.4-… rather than the backend error.
CREATE EXTERNAL CATALOG lakehouse PROPERTIES (\n 'type' = 'iceberg',\n 'iceberg.catalog.type' = 'rest',\n 'iceberg.catalog.uri' = 'http://polaris:8181/api/catalog',\n 'iceberg.catalog.warehouse' = 'lakehouse',\n 'iceberg.catalog.security' = 'oauth2',\n 'iceberg.catalog.oauth2.credential' = 'root:s3cr3t-dev-only',\n 'iceberg.catalog.oauth2.scope' = 'PRINCIPAL_ROLE:ALL',\n 'iceberg.catalog.vended-credentials-enabled' = 'true',\n 'aws.s3.endpoint' = 'http://minio:9000',\n 'aws.s3.enable_path_style_access' = 'true'\n);\nThat is the entire integration. No table DDL, no schema mirroring, no sync job.
\n\nRead it back:
\n\nSHOW DATABASES FROM lakehouse;\nbronze\ngold\ninformation_schema\nsilver\nsilver_gold\nsilver_silver\nEvery namespace Trino created, immediately. Including silver_gold and silver_silver — the debris from the schema-naming bug in part two, which I had fixed in dbt but never cleaned out of the catalog.
That is a small thing that says something real: the catalog is shared, so mistakes are shared too. A second engine is also a second pair of eyes on your namespace hygiene.
\n\nSELECT count(*) FROM lakehouse.silver.orders_enriched;\n1645100\nTrino reports the same number for the same table.
\n\nVerify. Row counts match between engines, and SHOW DATABASES FROM lakehouse lists the namespaces you created in Trino.
The catalog above declares vended-credentials-enabled and — deliberately — no aws.s3.access_key or aws.s3.secret_key at all. If StarRocks did not support vending, this would fail at first read.
It does not fail. 1,645,100 rows, with StarRocks obtaining short-lived credentials from Polaris exactly as Trino does: authenticate as an OAuth2 principal, receive a session token scoped to the table’s prefix, use it, discard it.
\n\nThis is the result that justifies the architecture. The authorisation boundary built in part one is engine independent. Adding a query engine does not add a place where a long-lived S3 key has to live, which is the failure mode that quietly undoes most “we use a catalog” stories.
\n\nCheck this on your own store rather than assuming it. Vending depends on the object store exposing STS and on the engine implementing the delegation header. Part one showed both failing against MinIO in different ways. The five-minute test is the one above: define a catalog with no keys in it and see whether a read succeeds.
\nSame query, same table, same catalog:
\n\nSELECT count(*), sum(totalprice)\nFROM lakehouse.silver.orders_enriched\nWHERE orderdate >= '1995-03-01' AND orderdate < '1995-04-01';\nGetting a comparable number took two attempts. Wall-clock through each engine’s CLI is useless: the Trino CLI starts a JVM, the MySQL client does not. Subtracting a SELECT 1 baseline from each was worse — it gave Trino a negative query time, because JVM startup varies by more than the query costs.
Both engines report their own timings, so use those. Trino from system.runtime.queries; StarRocks from its FE audit log:
grep 'orders_enriched' /data/deploy/starrocks/fe/log/fe.audit.log \\\n | grep -o 'Time=[0-9]*' | tail -7\n| engine | \nserver-side median | \n
|---|---|
| Trino 483 | \n99 ms | \n
| StarRocks 4.1.4 | \n34 ms | \n
This is a narrow test and the number should be read narrowly. One aggregate, one partition, 100,000 rows, on a laptop. That is close to the shape StarRocks’ vectorised MPP engine is built for, and it says nothing about the cases Trino is usually chosen for: federating across catalogs, joining a lakehouse table to a Postgres table, or long multi-table joins where the cost model matters more than the scan.
\n\nThe useful conclusion is not “StarRocks is 2.9× faster”. It is that you can now measure that question on your own data without migrating anything, because both engines read the same tables. Before a REST catalog, answering it meant a copy.
\nSince the catalog is shared, so is the tidying:
\n\nDROP TABLE IF EXISTS lakehouse.silver_silver.orders_enriched;\nDROP TABLE IF EXISTS lakehouse.silver_gold.daily_revenue;\nDROP SCHEMA IF EXISTS lakehouse.silver_silver;\nDROP SCHEMA IF EXISTS lakehouse.silver_gold;\nbronze\ngold\ninformation_schema\nsilver\nsystem\nNote this needs DROP_WITH_PURGE_ENABLED from part two, or the drops fail with a Trino error that never mentions purge.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
Backend node not found | \n FE up, BE not yet registered | \nPoll SHOW BACKENDS for Alive: true | \n
Catalog created, SHOW DATABASES empty | \n OAuth2 scope or grant chain missing | \nCheck PRINCIPAL_ROLE:ALL and the part-one grant chain | \n
| Reads fail with no S3 keys set | \nEngine or store lacks vending support | \nTest with a keyless catalog before committing to the design | \n
| Unexpected namespaces appear | \nShared catalog, shared history | \nThey are real; clean them from either engine | \n
| Engine timings wildly inconsistent | \nMeasuring client startup | \nUse system.runtime.queries and fe.audit.log | \n
docker compose --profile engines down\nThe rule: a catalog earns its place when the second engine costs one DDL statement and no new credentials.
\n\nThat is a testable bar, and it is worth applying before committing to a catalog rather than after. Attach a second engine to a throwaway namespace, define it with no storage keys, and read a table the first engine wrote. If that works, the catalog is doing the job the architecture diagram claims. If it needs a copy, a manual table registration, or its own set of long-lived keys, you have a metadata service — which may still be worth running, but not for the reasons usually given.
\n\nThe rest of this series is on GitHub as a single docker compose file: MinIO, Polaris, Trino, dbt, Airflow and StarRocks, with every figure in these four posts reproducible from it.
A small end-to-end system: scrape a bounded set of pages, land them in object storage, refine them through bronze, silver and gold Delta tables on Spark, embed the result, and serve answers that cite the passages they came from — or decline.
\n\nI built the first version in July 2025. Coming back to it in 2026, the most useful thing about it was not that it worked; it was why it had stopped.
\n\nThree reasons, and the first is not a version problem.
\n\nbitnami/spark:3.5 GONE\nmarquezproject/marquez:latest pulls\nchromadb/chroma:latest pulls\nbitnami/spark is gone. Broadcom moved the Bitnami catalog behind a paid tier during 2025. A bitnamilegacy namespace survives, but it publishes only sha256 digests — there is no 3.5, nothing you can write in a compose file and read six months later.
Nine images were pinned to :latest. Which means the stack that ran in July 2025 is not recoverable, and no assertion about it is checkable. That is the actual cost of :latest, and it is not a style preference.
Three JAR files, roughly 270 MB, had to be downloaded by hand — aws-java-sdk-bundle, hadoop-aws, hadoop-common — and they were in neither the repository nor my own working copy. The public repo failed on first run for everyone who cloned it, including me. They were also the AWS SDK v1 S3A stack, which Spark 4 has moved past regardless.
A vendor’s free tier is a dependency. The image was pinned, the tag was specific, and it still evaporated. When a base image comes from a commercial vendor’s community edition, the supported exit is worth knowing before you need it — here it was the official apache/spark, which also publishes arm64.
spark-master:\n image: apache/spark:4.0.4-scala2.13-java17-python3-r-ubuntu\n command: /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master\n volumes:\n - ./spark:/opt/spark-lab:ro\n - ./jobs:/opt/jobs:ro\n - ivy_cache:/tmp/ivy\nThe jars/ directory is gone entirely. Spark resolves S3A and Delta from Maven at submit time:
exec /opt/spark/bin/spark-submit \\\n --master \"${SPARK_MASTER:-spark://spark-master:7077}\" \\\n --conf spark.jars.ivy=\"${IVY_DIR:-/tmp/ivy}\" \\\n --packages \"io.delta:delta-spark_2.13:${DELTA_VERSION},org.apache.hadoop:hadoop-aws:${HADOOP_AWS_VERSION}\" \\\n --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \\\n --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \\\n --conf spark.hadoop.fs.s3a.endpoint=\"${S3_ENDPOINT}\" \\\n --conf spark.hadoop.fs.s3a.path.style.access=true \\\n \"$@\"\nTwo things in there are load-bearing and neither is obvious.
\n\nspark.jars.ivy is not optional on apache/spark. The spark user has no home directory, so spark-submit dies before resolving anything:
java.io.FileNotFoundException: /nonexistent/.ivy2.5.2/cache/resolved-org.apache.spark-…xml\nBitnami’s image set a home; the official one does not. This is the single most likely thing to bite anyone migrating off Bitnami.
\nDelta and Spark are pinned to each other, minor by minor. Same Spark, three Delta versions:
\n\ndelta-spark_2.13 | \n Spark 4.0.4 | \nresult | \n
|---|---|---|
| 4.3.1 (latest) | \n✗ | \nNoSuchMethodError: ParserInterface.$init$ | \n
| 4.1.0 | \n✗ | \nsame | \n
| 4.0.0 | \n✓ | \n50,000 rows written | \n
All three resolve from Maven without a murmur. Maven has no idea which Spark you are running, so a wrong pairing is a class-load failure on the first DataFrame call, several seconds after everything looked fine. “Use the latest” is the wrong instinct here. Matching is.
\n\nOne from the container runtime rather than Spark. Mounting ./jobs at /opt/spark-lab/jobs while /opt/spark-lab was itself a :ro mount left both Spark containers stuck in Created with no logs at all — the failure happens before any process starts, so there is nothing to read and docker compose logs returns empty. Use separate mount points.
Verify. spark: 4.0.4, rows_written: 50000, and a _delta_log/ prefix in the bucket.
def allowed(url: str) -> bool:\n \"\"\"Ask robots.txt per host, not once at the start.\"\"\"\n parts = urllib.parse.urlsplit(url)\n rp = robotparser.RobotFileParser()\n rp.set_url(f\"{parts.scheme}://{parts.netloc}/robots.txt\")\n try:\n rp.read()\n except Exception:\n return False # unreadable robots.txt is a no, not a yes\n return rp.can_fetch(UA, url)\nScraping is IO-bound and rate-limited by politeness. Putting it on the cluster buys nothing and makes the robots check harder to follow. It runs in a small Python container and lands append-only JSONL.
\n\nChunking is by <h2>/<h3> section rather than by page or by character count. A whole reference page is far too much context for one embedding; a fixed character window cuts sentences in half. A section is roughly one idea, which is the unit a question is usually about.
Verify. landed s3://lakehouse/raw/scrape/…jsonl sections=187 skipped=0 bytes=882663.
# bronze: pure function of the raw prefix, overwritten each run\nraw = spark.read.json(f\"{BUCKET}/raw/scrape/\")\nbronze = raw.withColumn(\"ingested_at\", F.current_timestamp()) \\\n .withColumn(\"text_len\", F.length(\"text\"))\nbronze.write.format(\"delta\").mode(\"overwrite\").save(f\"{BUCKET}/bronze/sections\")\n| layer | \nrows | \nwhat it owes you | \n
|---|---|---|
| bronze | \n187 | \nevery raw record, typed and stamped, nothing dropped | \n
| silver | \n170 | \ncleaned and deduplicated on (url, section) | \n
| gold | \n170 | \na deterministic chunk_id, the text, and enough provenance to cite | \n
Read the directory, not a glob. raw/scrape/*.jsonl returns the right rows and throws FileNotFoundException: No such file or directory into the log on the way past — S3A resolves the glob by listing, and the miss surfaces as an exception the job then recovers from. A stack trace in a successful run trains you to ignore stack traces.
chunk_id is sha256(url :: section) truncated to 16 characters. That makes it stable across runs, so re-running the pipeline updates a vector instead of inserting a second copy beside it.
read 170 chunks from delta in 0.21s\nembed 170 chunks in 2.62s (64.8/s, dim=384, model load 7.42s)\nupsert 170 vectors in 0.25s\nReading Delta here uses the Rust deltalake reader, not PySpark. Embedding 170 rows does not need a cluster, and a 400 MB JVM image to read a small table is the kind of default that makes a stack feel heavier than its problem.
kept = [(d, m, s) for d, m, s in zip(docs, metas, dists) if s <= MAX_DISTANCE]\nif not kept:\n return {\"answer\": \"I don't know.\", \"grounded\": False,\n \"nearest_distance\": round(min(dists), 4), \"sources\": []}\nThree questions against a corpus of Python standard-library documentation:
\n\n| question | \ngrounded | \nevidence | \n
|---|---|---|
| read and write JSON in Python | \nyes | \ncites the json sections | \n
| difference between a list and a tuple | \nyes | \nTuples d=0.234, Lists d=0.356 | \n
| capital city of Nepal | \nno | \nnearest passage d=0.923, above the 0.75 threshold | \n
The third is the one worth building for. The response carried a retrieve timing and no generate timing at all — the model was never called. Nothing in the corpus was close enough, so there was nothing to be fluent about.
That ordering matters. A retriever that hands four irrelevant passages to a model and asks it to be careful is asking the wrong component to hold the line. A refusal that costs no inference is the cheapest safety property in the system, and it is a threshold rather than a prompt.
\n\nVerify. An out-of-corpus question returns grounded: false, a nearest_distance above your threshold, and no generate timing.
This is the part I would keep if I could keep only one.
\n\nEvery section title came back with a stray  welded to the pilcrow — requests had guessed the charset from headers that did not commit to one. The fix is one line: hand BeautifulSoup r.content instead of r.text and let it sniff the meta charset.
That fixed the scrape. It fixed nothing downstream.
\n\nchunk_id is a hash of (url, section). Changing the title changed the identity of every chunk — and every store in the chain is upsert-only:
| store | \nrows after the “fix” | \nwhy | \n
|---|---|---|
| bronze — Delta, overwrite | \n187, correct | \nrewritten from raw every run | \n
| silver — Delta, MERGE | \n323 | \nMERGE inserts and updates. It never deletes a row the source stopped producing. | \n
| gold — Delta, overwrite | \n323 | \nderived from a silver that was already wrong | \n
| Chroma — upsert | \n323 | \nsame shape, same outcome | \n
Then the retriever answered a question with both copies:
\n\nsrc: 'Tuples' dist 0.2338\nsrc: 'TuplesÂ' dist 0.2350\nTwo near-identical passages competing for the same context window, from a bug that had already been fixed at the source. From the outside this reads as a model problem — the answer is repetitive, the citations are redundant. It is a pipeline problem, three layers upstream.
\n\nThe lesson is not “handle encodings”. It is that a natural key derived from scraped content is not stable, and an upsert-only layer has no way to forget. A merge that is meant to converge on its source needs a delete clause for rows the source no longer produces, or the layer needs a periodic full rebuild. I added the rebuild:
\n\nif os.environ.get(\"REBUILD\", \"\").lower() in (\"1\", \"true\", \"yes\"):\n client.delete_collection(\"sections\")\n| stage | \nfigure | \n
|---|---|
| scrape | \n187 sections, 10 pages, 883 KB | \n
| bronze → silver | \n187 → 170 (17 duplicates removed) | \n
| embed | \n170 chunks in 2.6 s — 64.8/s, 384 dimensions | \n
| upsert | \n0.16 s | \n
| retrieve | \n45–58 ms | \n
| generate | \n20–34 s, llama3.2:1b on CPU | \n
The generate figure is honest and unflattering: a 1B model on CPU. That is deliberate — the subject here is the pipeline around the model, and a larger model would make every run slower while changing none of the properties being tested.
\n\n| Symptom | \nCause | \nRepair | \n
|---|---|---|
bitnami/spark:3.5 not found | \n image moved to a paid catalog | \nuse apache/spark | \n
/nonexistent/.ivy2 FileNotFoundException | \n spark user has no home | \n set spark.jars.ivy | \n
NoSuchMethodError: ParserInterface.$init$ | \n Delta/Spark minor mismatch | \nmatch Delta to Spark, not to latest | \n
Containers stuck in Created, no logs | \n volume mounted inside a :ro mount | \n separate mount points | \n
FileNotFoundException in a successful job | \n S3A glob listing | \nread the directory | \n
| Duplicate near-identical citations | \nupsert-only stores, changed key | \nfull rebuild, or a delete clause | \n
docker compose --profile tools down -v\nTwo rules came out of this.
\n\nPin everything, and know your exit from every vendor image. The stack did not break because of a bad upgrade. It broke because a company changed its distribution model and :latest meant there was nothing to fall back to.
Ask which of your stores can forget. Every layer here could accept new data and none could drop stale data, which is fine until a key changes — and keys derived from source content change more often than anyone plans for. The question to ask of a pipeline is not “does a re-run add the right rows”, it is “does a re-run remove the wrong ones”. Bronze could. Nothing else could.
\n\nLineage is the obvious next piece: the original had Marquez wired in, and an OpenLineage graph would have shown the orphaned rows propagating across three stores in one picture rather than three separate counts.
\n", "summary": "Scrape to bronze, silver, gold on Spark 4 and Delta, embed into Chroma, and serve answers that cite their sources or decline to answer.", "date_published": "2026-08-16T08:00:00+05:45", "date_modified": "2026-08-16T00:00:00+05:45", "tags": ["rag","iceberg-lakehouse"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/pgbackrest-patroni-cluster-backup-pitr/", "url": "https://basantbhattarai.com.np/writing/pgbackrest-patroni-cluster-backup-pitr/", "title": "pgBackRest against a Patroni cluster: archive, rebuild, and restore", "content_html": "Backing up one PostgreSQL server is a solved problem with a well-worn runbook. Backing up a Patroni cluster is the same tool pointed at a target that moves: the primary is whichever node currently holds the leader key, and that can change between two backups without anyone touching a config file.
\n\nSo the repository is not configured against “the primary”. It is configured against every member, and pgBackRest works out which one is writable when it runs. That single decision is what makes the setup survive a failover, and it is visible in the very first command’s output.
\n\nEverything here is Ansible. Part 1 of this series established the rule and it holds: Vagrant boots the box and wires the lab NIC; packages, configuration, services and certificates belong to Ansible, because a lab you fixed by hand is a lab you cannot rebuild.
\n\nThe cluster is the one from part 2 — pgn1 and pgn2 on PostgreSQL 18.6 with etcd on etcd1. The repository lives on etcd1 as well, which is a lab economy and not a recommendation: in production the DCS and the backup repository are separate failure domains, and the repo does not sit on a database host at all.
The Ansible play installs pgBackRest on all three nodes, then writes a repository config that lists both cluster members:
\n\n[global]\nrepo1-path=\nrepo1-retention-full=2\nstart-fast=y\n\n[]\n\nTransport is mutual TLS, the same as part 1, and it is the better answer for a cluster: no shell account on the peers, and authorisation is an explicit CN allow-list rather than “whatever the postgres unix user can reach over ssh”. Each side names the other:
\n\ntls-server-address=*\ntls-server-port=8432\ntls-server-cert-file=/etc/pgbackrest/certs/etcd1.crt\ntls-server-key-file=/etc/pgbackrest/certs/etcd1.key\ntls-server-ca-file=/etc/pgbackrest/certs/ca.crt\ntls-server-auth=pgn1=pg18lab\ntls-server-auth=pgn2=pg18lab\nThe database hosts get the mirror image, plus asynchronous archiving so archive_command never becomes the cluster’s write ceiling. They run a TLS server too, because the repo reaches back to them when it takes a backup:
[global]\nrepo1-host=192.168.105.140\nrepo1-host-type=tls\nrepo1-host-port=8432\nrepo1-host-ca-file=/etc/pgbackrest/certs/ca.crt\nrepo1-host-cert-file=/etc/pgbackrest/certs/pgn1.crt\nrepo1-host-key-file=/etc/pgbackrest/certs/pgn1.key\narchive-async=y\nspool-path=/var/spool/pgbackrest\n\ntls-server-auth=etcd1=pg18lab\nA certificate signed by the CA but missing from tls-server-auth authenticates fine and is then refused. That is the behaviour you want, and it is worth knowing before you meet the error.
Name the stanza after the Patroni scope. pgBackRest thinks in stanzas and Patroni thinks in scopes; keeping them identical saves a future reader wondering whether they are the same thing.
\n\nVerify. The playbook is clean across all three hosts:
\n\netcd1 : ok=9 changed=4 unreachable=0 failed=0\npgn1 : ok=7 changed=3 unreachable=0 failed=0\npgn2 : ok=7 changed=3 unreachable=0 failed=0\nThis cost the most time of anything in the build, and the error points nowhere near the cause:
\n\nWARN: unable to check pg1: [ServiceError] TLS error [1:336151570] sslv3 alert bad certificate\nERROR: [027]: no database found\nThe certificates were fine. The CA fingerprint matched on every host. What did not match was the time:
\n\ncontrol node (Mac): 2026-08-14 16:24:33 UTC\n etcd1 2026-08-14 16:24:34 UTC\n pgn1 2026-08-14 15:07:59 UTC\n pgn2 2026-08-14 15:08:00 UTC\nCA notBefore: Aug 14 16:16:14 2026 GMT\nBoth database nodes were 76 minutes behind. The CA was issued on the Mac at 16:16, so from their point of view it was not yet valid, and openssl verify says so plainly once you ask it directly:
error 9 at 1 depth lookup: certificate is not yet valid\nTCG guests drift badly, and chronyd slews rather than steps, so it corrects a large offset far too slowly to help. Step the clock, and do it before anything issues a certificate:
\n\n- name: Force an immediate step\n ansible.builtin.command: chronyc -a makestep\n\n- name: Fail if still skewed from the control node\n ansible.builtin.assert:\n that: \"(guest_epoch.stdout | int - control_epoch | int) | abs < 120\"\n fail_msg: \" is too far from the control node; certificates will not validate.\"\nThe assertion matters more than the makestep. A silent 76-minute skew turns into a TLS error an hour later; a failed assertion names the problem at the point it exists.
Verify. With clocks stepped, the handshake works and check proves the whole path:
INFO: check repo1 (standby)\nINFO: switch wal not performed because this is a standby\nINFO: check repo1 configuration (primary)\nINFO: WAL segment 00000004000000000000000F successfully archived ... on repo1\nINFO: check command end: completed successfully (2560ms)\nThis is where most integrations quietly break, and the split is not arbitrary.
\n\narchive_mode and archive_command describe the whole cluster, so they belong in the DCS. Patroni regenerates postgresql.conf from the DCS every time it starts a node — so an archive_command typed straight into that file works perfectly until the first failover, then vanishes at exactly the moment you need it.
- name: Push archive settings into the DCS\n ansible.builtin.shell: |\n cat <<'YML' | patronictl -c edit-config --force --apply -\n postgresql:\n parameters:\n archive_mode: \"on\"\n archive_command: \"pgbackrest --stanza= archive-push %p\"\n archive_timeout: 60\n YML\n become_user: postgres\n run_once: true\ncreate_replica_methods is the opposite. It names a command on that host, so it cannot live in shared cluster state and goes into the local patroni.yml:
create_replica_methods:\n - pgbackrest\n - basebackup\npgbackrest:\n command: \"/usr/bin/pgbackrest --stanza=pg18lab --delta restore\"\n keep_data: true\n no_params: true\nOrder matters. With pgbackrest first, a new replica clones from the repository; basebackup stays as the fallback. Without it, every rebuild drags a full copy off the primary while that primary is serving traffic.
Apply the config and check whether PostgreSQL actually took it:
\n\npsql -tAc \"SELECT name||' = '||setting FROM pg_settings\n WHERE name IN ('archive_mode','archive_command','archive_timeout');\"\narchive_command = (disabled)\narchive_mode = off\narchive_timeout = 60\nThe DCS has the right values and the server does not. archive_timeout is reloadable so it applied; archive_mode is a postmaster-level parameter and needs a full restart. Patroni knows, and says so:
| Member | Role | State | TL | Pending restart | Pending restart reason |\n| pgn1 | Replica | streaming | 3 | * | archive_mode: off->on |\n| pgn2 | Leader | running | 3 | * | archive_mode: off->on |\nA rolling restart through patronictl handles the replica first and the leader last, so the write outage is one brief switchover rather than the whole cluster:
patronictl -c /etc/patroni/patroni.yml restart pg18lab --force\nVerify. The flag clears and archiving is live:
\n\narchive_command = pgbackrest --stanza=pg18lab archive-push %p\narchive_mode = on\nRun these from the repository host. stanza-create initialises the repo layout; check proves a WAL segment can actually make the trip.
sudo -u postgres pgbackrest --stanza=pg18lab stanza-create\nsudo -u postgres pgbackrest --stanza=pg18lab check\nVerify. Read the check output closely, because it contains the proof that the multi-host config works:
INFO: switch wal not performed because this is a standby\nINFO: check repo1 configuration (primary)\nINFO: WAL segment 000000030000000000000004 successfully archived to\n '/var/lib/pgbackrest/archive/pg18lab/18-1/...' on repo1\nINFO: check command end: completed successfully (6795ms)\npgBackRest probed both declared hosts, found one in recovery, skipped it, and did the WAL switch on the primary. Nothing told it which node was the leader.
\n\nsudo -u postgres pgbackrest --stanza=pg18lab --type=full backup\nINFO: backup start archive = 000000030000000000000006, lsn = 0/6000060\nINFO: new backup label = 20260814-141156F\nINFO: full backup size = 22.7MB, file total = 981\nINFO: backup command end: completed successfully (23910ms)\nThe repo stored 2.9 MB for a 22.7 MB database — compression is on by default and worth knowing before you size the volume.
\n\nThe first restore drill is the cheap one, and it is the one you will actually use: throw a replica away and let Patroni rebuild it.
\n\npatronictl -c /etc/patroni/patroni.yml reinit pg18lab pgn1 --force\nVerify. The Patroni journal on the rebuilt node names the method, which is the only way to know it did not silently fall back to pg_basebackup:
INFO: restore command begin 2.59.0: --delta --repo1-host=192.168.105.140 ... --stanza=pg18lab\nINFO: restore command end: completed successfully (5020ms)\nINFO: replica has been created using pgbackrest\nFive seconds, and the primary was never touched. On a database where a full copy would take hours across a busy link, that difference is the entire reason to wire the two tools together.
\n\nNow the drill people actually rehearse for. Note a timestamp, then break something:
\n\nSELECT now(); -- 2026-08-14 14:15:20.119249+00\nDROP TABLE failover_probe; -- the accident\nSELECT pg_switch_wal(); -- make sure the damage is archived\nPatroni must be told to stand down first, or it will fight the restore by restarting PostgreSQL underneath you:
\n\npatronictl -c /etc/patroni/patroni.yml pause --wait\nThen stop PostgreSQL — and this is the step that catches people. Pausing Patroni and stopping the service does not stop the database. That is what pause means: Patroni stops managing, PostgreSQL keeps serving.
\n\nERROR: [038]: unable to restore while PostgreSQL is running\nStop it properly on both nodes, then restore the leader to the moment before the drop:
\n\nsudo -u postgres /usr/pgsql-18/bin/pg_ctl -D /var/lib/pgsql/18/data -m fast stop\n\nsudo -u postgres pgbackrest --stanza=pg18lab --delta --type=time \\\n --target='2026-08-14 14:15:20' --target-action=promote restore\nINFO: repo1: restore backup set 20260814-141156F, recovery will start at 2026-08-14 14:11:56\nINFO: restore command end: completed successfully (4198ms)\nStart the server and let it replay forward to the target:
\n\nsudo -u postgres /usr/pgsql-18/bin/pg_ctl -D /var/lib/pgsql/18/data -w -t 300 start\nVerify. The table is back, on a new timeline, and out of recovery:
\n\nin recovery: f\ntimeline: 4\ntable back: t\nrow count: 5\n\n id | note\n----+---------------------------------\n 1 | written on pgn1 before kill\n 34 | written on pgn2 after promotion\n 35 | after etcd recovery\n 36 | before replica reinit\n 37 | the row we must not lose\nResume Patroni, start the other node, and rebuild it — and it hangs in starting forever:
LOG: fetching timeline history file for timeline 3 from primary server\nFATAL: could not receive timeline history file from the primary server:\n ERROR: could not open file \"pg_wal/00000003.history\": No such file or directory\nThe replica restored from a backup whose base is on timeline 3 and is asking the leader for that timeline’s history. After the point-in-time restore the leader is on timeline 4 and never had the file. archive-get cannot supply it either, because archiving was switched on after the cluster had already reached timeline 3 — so that history file was never archived at all.
The fix is not a flag. It is an ordering rule: after a point-in-time restore, take a fresh full backup before rebuilding any replica. The new backup’s base is on the current timeline, so nothing has to reach backwards.
\n\nsudo -u postgres pgbackrest --stanza=pg18lab --type=full backup\npatronictl -c /etc/patroni/patroni.yml reinit pg18lab pgn1 --force\nVerify. Both members on timeline 4, streaming, and both holding the recovered rows:
\n\n| Member | Host | Role | State | TL | Replay LSN | Lag |\n| pgn1 | 192.168.105.141 | Replica | streaming | 4 | 0/E000060 | 0 |\n| pgn2 | 192.168.105.142 | Leader | running | 4 | | |\n\n .141: 5 rows\n .142: 5 rows\n| Symptom | \nCause | \nRepair | \n
|---|---|---|
archive_command disabled after edit-config | \n archive_mode is postmaster-level | \n Rolling patronictl restart; watch the Pending restart column | \n
| archive settings vanish after a failover | \nWritten into postgresql.conf by hand | \n Put them in the DCS with edit-config | \n
unable to restore while PostgreSQL is running | \n pause and stop-service do not stop the database | \npg_ctl -m fast stop on every node first | \n
Replica stuck starting, wants 0000000N.history | \n Backup base predates the current timeline | \nFresh full backup after the PITR, then reinit | \n
| Replica rebuild is slow and loads the primary | \ncreate_replica_methods missing or wrong order | \n List pgbackrest before basebackup in the local yml | \n
check reports the wrong host as primary | \n Only one pgN-host declared | \n Declare every member; the primary moves | \n
sslv3 alert bad certificate, certs look correct | \n Guest clock behind the CA’s notBefore | \nchronyc -a makestep, then assert the offset | \n
| Peer authenticates then is refused | \nCN missing from tls-server-auth | \n Add tls-server-auth=<CN>=<stanza> on the receiving side | \n
The cluster now survives three separate things, and they are worth naming separately because they fail separately. A node dying is handled by Patroni, in about twenty seconds, from part 2. A replica needing to come back is handled by pgBackRest in five seconds without touching the primary. And an operator error — the dropped table — is handled by a point-in-time restore that recovered every row.
\n\nOnly the first of those is automatic. The other two are drills, and a drill you have not run is a plan, not a capability. The useful outcome of this post is not the config; it is that the restore was performed once, on purpose, while nothing was actually on fire.
\n", "summary": "Wire pgBackRest into a running Patroni cluster over mutual TLS with Ansible, then prove it: full backup, replica rebuilt from the repo, and a point-in-time restore.", "date_published": "2026-08-14T16:30:00+05:45", "date_modified": "2026-08-14T00:00:00+05:45", "tags": ["postgres","distributed-databases"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/patroni-postgresql-18-rocky8-etcd-failover/", "url": "https://basantbhattarai.com.np/writing/patroni-postgresql-18-rocky8-etcd-failover/", "title": "Patroni HA on PostgreSQL 18 with etcd, proven by killing the primary", "content_html": "Most Patroni walkthroughs stop at patronictl list showing a Leader and a Replica. That is the setup, not the result. The question worth answering is what happens when the primary dies badly — not a graceful systemctl stop, but a SIGKILL that leaves the cluster to work it out — and then whether the dead node can come back without a rebuild.
This post builds that on three amd64 guests and measures it. The numbers below came off the run, not off a docs page: promotion observed in a 10–20 second window — patronictl was polled every 10 s, so that interval is the instrument’s resolution and not the precision of the result — the timeline advancing 1 → 2, and the old primary rejoining as a streaming replica 15 seconds after it was restarted, via pg_rewind from the last common checkpoint.
It is also honest about the shape of the lab. etcd runs on one node, which is a single point of failure — so rather than wave that away, the last section stops etcd and measures what a DCS outage actually costs. The answer is worth knowing before you design one: the healthy leader demotes itself and the cluster goes read-only.
\n\nThe guests are amd64 under QEMU TCG on an Apple Silicon Mac, on the shared 192.168.105.0/24 fabric from the networking tutorial. Slow on purpose: a lab that lies about the instruction set makes every later packaging and timing story suspect.
I started this on generic/rhel8 and it does not work, for a reason worth stating plainly because it is invisible until the last command fails.
An unregistered RHEL 8 box has no BaseOS and no AppStream. Only EPEL and PGDG are enabled:
\n\nsudo subscription-manager status\nsudo dnf -q repolist\nOverall Status: Unknown\nepel pgdg-common pgdg14 pgdg15 pgdg16 pgdg17 pgdg18\nPGDG packages Patroni 4.1.5 as patroni-etcd, and that package needs a Python 3.12 interpreter:
nothing provides python(abi) = 3.12 needed by python3.12-etcd-0.4.5-49PGDG.rhel8.noarch\npython3.12 lives in AppStream. No subscription, no AppStream, no Patroni. Registering a RHEL box fixes it and is the right answer in an enterprise where entitlements exist. For a lab anyone should be able to reproduce, a RHEL-compatible rebuild is better: Rocky Linux 8 is binary-compatible, needs no account, and its $releasever of 8 serves current 8.10 content.
Verify. On a Rocky 8 guest the interpreter is there:
\n\nsudo dnf -q list available python3.12\npython3.12.x86_64 3.12.13-3.el8_10 appstream\nEvery command from here works unchanged on a registered RHEL 8. It is only the unregistered box that cannot get there.
\n\nThe Vagrantfile boots three nodes and does nothing else. Packages, configuration and services come later; Vagrant’s job is a booted box with a routable lab NIC.
\n\nLAB_PREFIX = \"192.168.105\"\n\nNODES = [\n { name: \"etcd1\", ip: \"#{LAB_PREFIX}.140\", ssh_port: 22140, memory: \"1024\", smp: \"1\", mac: \"52:54:00:12:01:40\" },\n { name: \"pgn1\", ip: \"#{LAB_PREFIX}.141\", ssh_port: 22141, memory: \"1536\", smp: \"2\", mac: \"52:54:00:12:01:41\" },\n { name: \"pgn2\", ip: \"#{LAB_PREFIX}.142\", ssh_port: 22142, memory: \"1536\", smp: \"2\", mac: \"52:54:00:12:01:42\" },\n].freeze\n\nVagrant.configure(\"2\") do |config|\n config.vm.box = \"generic/rocky8\"\n config.vm.synced_folder \".\", \"/vagrant\", disabled: true\n config.vm.box_check_update = false\n # TCG boots a RHEL-family guest slowly. The 300s default expires mid-cloud-init\n # and Vagrant reports a timeout for a VM that is still coming up.\n config.vm.boot_timeout = 900\n # ... provider block: q35, cpu max, the socket_vmnet wrapper, and\n # -netdev socket,id=lab0,fd=3 as in the networking post\nend\nTwo settings are load-bearing and both were learned the hard way. boot_timeout = 900 because TCG is slow enough that the default makes Vagrant declare failure on a healthy boot. And no_daemonize = true in the provider, because forking drops the inherited lab file descriptor and the lab NIC transmits into nothing.
Verify. Every node answers on the fabric, and every node can reach every other:
\n\nfor n in etcd1 pgn1 pgn2; do\n vagrant ssh $n -c \"hostname -s; ip -4 -o addr show eth1 | awk '{print \\$4}'\"\ndone\netcd1 192.168.105.140/24 mesh: .140 ok .141 ok .142 ok\npgn1 192.168.105.141/24 mesh: .140 ok .141 ok .142 ok\npgn2 192.168.105.142/24 mesh: .140 ok .141 ok .142 ok\nGuest ICMP to the vmnet gateway at .1 fails on macOS even when everything else is healthy. Host-to-guest and guest-to-guest are the pass criteria; do not chase the gateway ping.
etcd left the RHEL AppStream after RHEL 7. PGDG ships it in a repository that is present but disabled by default, and enabling it produces a message that sends people straight to a tarball:
\n\nError: Failed to download metadata for repo 'pgdg-rhel8-extras':\nrepomd.xml GPG signature verification error: Bad GPG signature\nThe signature is fine. Verifying it by hand against the key the repo already shipped proves it:
\n\ngpg --verify repomd.xml.asc repomd.xml\ngpg: Signature made Thu 13 Aug 2026 07:34:54 AM UTC\ngpg: using RSA key D4BF08AE67A0B4C7A1DBCCD240BCA2B408B40D20\ngpg: Good signature from \"PostgreSQL RPM Repository <pgsql-pkg-yum@lists.postgresql.org>\"\nWhat actually happens is that dnf wants to import that key, asks Is this ok [y/N]:, gets no answer in a non-interactive shell, and reports the refusal as a bad signature. The fix is -y:
sudo dnf -y --enablerepo=pgdg-rhel8-extras install etcd\nInstalled:\n etcd-3.7.0-1PGDG.rhel8.10.x86_64\nPoint it at the lab address, not localhost, or Patroni on the other two nodes cannot reach it:
\n\nname: etcd1\ndata-dir: /var/lib/etcd\nlisten-client-urls: http://192.168.105.140:2379,http://127.0.0.1:2379\nadvertise-client-urls: http://192.168.105.140:2379\nlisten-peer-urls: http://192.168.105.140:2380\ninitial-advertise-peer-urls: http://192.168.105.140:2380\ninitial-cluster: etcd1=http://192.168.105.140:2380\ninitial-cluster-state: new\ninitial-cluster-token: pg18-patroni-lab\nThe PGDG unit reads a different path, so override it rather than editing the shipped file:
\n\n[Service]\nExecStart=\nExecStart=/usr/bin/etcd --config-file=/etc/etcd/etcd.conf.yaml\nsudo systemctl daemon-reload\nsudo systemctl enable --now etcd\nsudo firewall-cmd --permanent --add-port=2379/tcp --add-port=2380/tcp && sudo firewall-cmd --reload\nVerify. The endpoint is healthy from the fabric address, not just from localhost:
\n\netcdctl --endpoints=http://192.168.105.140:2379 endpoint health\nhttp://192.168.105.140:2379 is healthy: successfully committed proposal: took = 472.211084ms\nOne deprecation worth noting: ETCDCTL_API=3 is now an unrecognised environment variable and etcd 3.7 warns about it. Every older Patroni guide still sets it. Drop it.
On both database nodes, install from PGDG and let it resolve its own Python:
\n\nsudo dnf -y install https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm\nsudo dnf -qy module disable postgresql\nsudo dnf -y install postgresql18-server postgresql18-contrib patroni-etcd\npostgres (PostgreSQL) 18.6\npatroni 4.1.5\nPython 3.12.13\nThat last line is the point of doing it this way. RHEL 8 and Rocky 8 ship Python 3.6 as python3, which current Patroni does not support. pip install patroni fails, and the usual workaround is a hand-built interpreter nobody maintains. The PGDG package pulls python3.12-etcd, python3.12-dns and the rest of a parallel 3.12 stack, and the system Python is never involved.
Do not run initdb. Patroni owns the data directory and will bootstrap it.
scope: pg18lab\nname: pgn1\n\nrestapi:\n listen: 192.168.105.141:8008\n connect_address: 192.168.105.141:8008\n\netcd3:\n hosts: 192.168.105.140:2379\n\nbootstrap:\n dcs:\n ttl: 30\n loop_wait: 10\n retry_timeout: 10\n postgresql:\n use_pg_rewind: true\n parameters:\n wal_level: replica\n max_wal_senders: 10\n max_replication_slots: 10\n pg_hba:\n - host replication replicator 192.168.105.0/24 scram-sha-256\n - host all all 192.168.105.0/24 scram-sha-256\n - local all all peer\n\npostgresql:\n listen: 192.168.105.141:5432\n connect_address: 192.168.105.141:5432\n data_dir: /var/lib/pgsql/18/data\n bin_dir: /usr/pgsql-18/bin\n authentication:\n replication: { username: replicator, password: replpass }\n superuser: { username: postgres, password: pgpass }\nUse etcd3, not etcd. The older key is the v2 API, which modern etcd no longer serves. And use_pg_rewind: true is what makes Step 6 work at all.
Start pgn1 first and let it win the bootstrap, then start pgn2:
sudo systemctl enable --now patroni\nsudo -u postgres patronictl -c /etc/patroni/patroni.yml list\nVerify. Two members, one leader, replica streaming with no lag:
\n\n+ Cluster: pg18lab (7673867291462481394) --------+----+-------------+-----+------------+-----+\n| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |\n+--------+-----------------+---------+-----------+----+-------------+-----+------------+-----+\n| pgn1 | 192.168.105.141 | Leader | running | 1 | | | | |\n| pgn2 | 192.168.105.142 | Replica | streaming | 1 | 0/302C4F8 | 0 | 0/302C4F8 | 0 |\n+--------+-----------------+---------+-----------+----+-------------+-----+------------+-----+\nWrite something first, so survival is a fact rather than an assumption:
\n\nPGPASSWORD=pgpass psql -h 192.168.105.141 -U postgres \\\n -c \"CREATE TABLE failover_probe(id serial primary key, note text, at timestamptz default now());\" \\\n -c \"INSERT INTO failover_probe(note) VALUES ('written on pgn1 before kill');\"\nConfirm it reached the replica, which should also report that it is in recovery:
\n\nPGPASSWORD=pgpass psql -h 192.168.105.142 -U postgres \\\n -tAc \"SELECT count(*), pg_is_in_recovery() FROM failover_probe;\"\n1|t\nNow kill it properly. Not systemctl stop — that is a handover, and a handover is the easy case:
sudo systemctl kill -s SIGKILL patroni\nsudo pkill -9 -f 'postgres -D'\nPoll the surviving node while it happens:
\n\n+10s pgn2 role=Replica\n+20s pgn2 role=Leader\nVerify. The survivor is a writable leader on a new timeline, and the pre-kill row is still there:
\n\nin recovery: f\nrows survived: 1\nINSERT 0 1\nrows now: 2\ntimeline: 2\nTwenty seconds is not a tuning result, it is arithmetic: ttl: 30 with loop_wait: 10 means the leader key expires somewhere inside a 30-second window and the next loop notices. Shortening the TTL shortens the outage and raises the chance a slow node gets demoted for a hiccup. That trade is the whole design conversation, and it belongs in your environment, not in a lab default.
A promoted replica is only half a failover. The old primary now holds writes that never reached the new leader — it diverged at the moment it died — so it cannot simply start following. Plenty of runbooks resolve this by wiping the data directory and re-cloning, which on a real database means hours and a lot of network.
\n\nuse_pg_rewind: true makes Patroni do the cheap thing instead. Start it and watch:
sudo systemctl start patroni\nsudo journalctl -u patroni -f\npg_rewind: rewinding from last common checkpoint at 0/20000B8 on timeline 1\npg_rewind: Done!\nINFO: no action. I am (pgn1), a secondary, and following a leader (pgn2)\nVerify. Fifteen seconds after the restart, the old primary is a streaming replica on the new timeline with zero lag:
\n\n| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |\n| pgn1 | 192.168.105.141 | Replica | streaming | 2 | 0/306F940 | 0 | 0/306F940 | 0 |\n| pgn2 | 192.168.105.142 | Leader | running | 2 | | | | |\npg_rewind needs either wal_log_hints = on or data checksums to work; the bootstrap above enables checksums via initdb, which is why this succeeded without extra configuration. Skip that and the rejoin silently falls back to a full re-clone, which looks identical in patronictl list and takes dramatically longer on a database with real data in it.
Almost, but the number belongs to a different component than most people attach it to.
\n\nPatroni does not require three Postgres nodes. Two — a primary and a standby — is a normal production topology and the one built above. The three-node minimum is a property of the DCS: etcd, Consul or ZooKeeper need an odd number of members to hold a quorum, and three is the smallest count that tolerates losing one. The two get conflated because the usual deployment co-locates etcd on the same three machines that run Postgres, so “a three-node Patroni cluster” ends up describing the DCS.
\n\nThat distinction is not academic, because a single-member DCS fails in a way that surprises people. It does not merely stop protecting you. Stop the one etcd node here and wait:
\n\nsudo systemctl stop etcd # on etcd1\npsql -h 192.168.105.142 -U postgres -tAc \"SELECT pg_is_in_recovery();\"\npsql -h 192.168.105.142 -U postgres -c \"INSERT INTO failover_probe(note) VALUES ('after etcd loss');\"\nVerify. Within about 45 seconds the healthy leader has demoted itself and the cluster is read-only:
\n\nt\nERROR: cannot execute INSERT in a read-only transaction\nNothing is wrong with Postgres. Patroni cannot renew the leader key, so it cannot prove it is still the leader, so it refuses to accept writes — which is the correct choice, because the alternative is two nodes both believing they are primary. A DCS outage converts into a full write outage, not a degraded one.
\n\nRestarting etcd restores service on its own:
\n\n| pgn1 | 192.168.105.141 | Replica | streaming | 3 |\n| pgn2 | 192.168.105.142 | Leader | running | 3 |\nINSERT 0 1\nNote the timeline: 3, not 2. The demote-and-repromote cycle is a new timeline, exactly like a failover, which is worth knowing before you go looking for the failover that “must have happened” in your logs.
\n\nSo: two Postgres nodes is a legitimate cluster. One etcd node is not a legitimate DCS. Put etcd on three hosts before this goes anywhere near production, and treat that as the availability floor for the whole system.
\n\n| Symptom | \nCause | \nRepair | \n
|---|---|---|
nothing provides python(abi) = 3.12 | \n Unregistered RHEL 8: no AppStream | \nRegister the box, or use Rocky/Alma 8 | \n
Bad GPG signature on pgdg-rhel8-extras | \n dnf refusing a non-interactive key import | \nAdd -y, or rpm --import the key first | \n
| Patroni starts, never joins | \netcd: instead of etcd3: in the config | \n Use etcd3; the v2 API is gone | \n
Replica stuck in creating replica | \n etcd unreachable from that node | \nOpen 2379/tcp; check advertise-client-urls is the lab IP | \n
| Vagrant reports boot timeout on a healthy VM | \nTCG slower than the 300s default | \nconfig.vm.boot_timeout = 900 | \n
| Old primary re-clones instead of rewinding | \nNo checksums and no wal_log_hints | \n initdb with data-checksums, or set the hint parameter | \n
| Healthy leader goes read-only, no failover in logs | \nDCS unreachable: Patroni cannot renew the leader key | \nRestore etcd; run three DCS members so one loss is survivable | \n
cd ~/labs/rocky8-patroni && vagrant destroy -f\nThe operating consequence is the part to carry forward. This cluster survives a dead primary in about twenty seconds without anyone being paged, and it repairs the dead node automatically instead of demanding a rebuild — but only because two settings were right before the incident: use_pg_rewind and data checksums. Neither can be added usefully while you are recovering.
The single etcd node is the honest limit of this lab. It proves the Postgres half. Whether the cluster stays available when the DCS itself goes down is a separate test, and it deserves a separate rig rather than an assumption.
\n", "summary": "Build a two-node Patroni cluster on Rocky 8 under QEMU, then SIGKILL the leader and measure the promotion, the timeline bump, and the pg_rewind rejoin.", "date_published": "2026-08-14T09:00:00+05:45", "date_modified": "2026-08-14T00:00:00+05:45", "tags": ["postgres","distributed-databases"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/pgbackrest-tls-ansible-rhel8/", "url": "https://basantbhattarai.com.np/writing/pgbackrest-tls-ansible-rhel8/", "title": "Dedicated pgBackRest TLS on RHEL 8 with Ansible—before Postgres", "content_html": "The usual pgBackRest lab starts with Postgres already running and SSH already trusted. That hides the failure you actually get in production: the backup host is up, port 8432 is open, and the first backup still dies on certificate CN, SAN, or a CLI option that does not mean what the docs from two versions ago said.
I wanted the opposite order. A dedicated backup1 is the TLS repo host. A tiny pg1 is only a TLS client. Ansible owns packages, pgbackrest.conf, firewalld, systemd, the private CA, and cert distribution. Vagrant boots the box and the lab NIC. Postgres is the next post in this series, not this one.
The measured package on this rig was pgBackRest 2.59.0 from PGDG on generic/rhel8. Your dnf may land a newer patch; record what you installed. This sits on the Apple Silicon amd64 networking tutorial. If host and mesh ping are not green, stop. A dead fabric looks like a pgBackRest bug and is not one.
There is no public repository to clone. The load-bearing files are reproduced below.
\n\nMac (Ansible control, Python 3.12 + ansible-core 2.15.13)\n │ SSH to lab IPs, not SLIRP hostfwd\n │\n ├─ backup1 192.168.105.130 generic/rhel8 1G / 1 CPU\n │ pgbackrest server :8432\n │ repo1-path /var/lib/pgbackrest\n │ certs /etc/pgbackrest/certs/backup1.{crt,key}\n │\n └─ pg1 192.168.105.131 generic/rhel8 1G / 1 CPU\n client cert /etc/pgbackrest/certs/pg1.{crt,key}\n proof server-ping + openssl s_client mTLS\nTwo NICs per guest, same as the fabric post: SLIRP for vagrant ssh, socket_vmnet for the addresses Ansible and pgBackRest will actually use. Private keys live on the control node under ansible/artifacts/certs/ and are gitignored. Guests get copies owned by the pgbackrest user, mode 0600 on keys.
NODES = [\n { name: \"backup1\", box: \"generic/rhel8\", ip: \"192.168.105.130\",\n ssh_port: 22130, memory: \"1024\", smp: \"1\", mac: \"52:54:00:12:01:30\" },\n { name: \"pg1\", box: \"generic/rhel8\", ip: \"192.168.105.131\",\n ssh_port: 22131, memory: \"1024\", smp: \"1\", mac: \"52:54:00:12:01:31\" },\n]\nEach provider block is arch = \"x86_64\", no_daemonize = true, and a second NIC on fd=3 through the socket_vmnet wrapper. The only shell provisioner is lab NIC + /etc/hosts. No packages, no pgBackRest.
./bin/vagrant-up backup1 pg1\nFirst TCG boot is slow. When provision finishes, eth1 holds the lab IP.
\n\nVerify.
\n\n$ ping -c 2 192.168.105.130\n2 packets transmitted, 2 packets received, 0.0% packet loss\n\n$ ping -c 2 192.168.105.131\n2 packets transmitted, 2 packets received, 0.0% packet loss\nHomebrew Ansible (core 2.21 on Python 3.14) fails against RHEL 8’s platform Python 3.6 with module deserialization errors (future feature annotations is not defined). Python 3.14 also breaks Ansible when: parsing (ast.Str). The control node for this lab is a Python 3.12 venv pinned to ansible-core 2.15.13.
python3.12 -m venv .venv\n.venv/bin/pip install 'ansible-core==2.15.13'\nInventory talks to lab IPs. vagrant ssh over SLIRP is fine for a rescue shell; backups must use the fabric.
all:\n hosts:\n localhost:\n ansible_connection: local\n ansible_python_interpreter: \"\"\n children:\n pgbackrest_servers:\n hosts:\n backup1:\n ansible_host: 192.168.105.130\n ansible_user: vagrant\n ansible_ssh_private_key_file: .vagrant/machines/backup1/qemu/private_key\n lab_ip: 192.168.105.130\n pgbackrest_clients:\n hosts:\n pg1:\n ansible_host: 192.168.105.131\n ansible_user: vagrant\n ansible_ssh_private_key_file: .vagrant/machines/pg1/qemu/private_key\n lab_ip: 192.168.105.131\nlab_ip is not decorative. The cert play uses it as a SAN. If the certificate only names backup1 and Ansible later connects by IP, OpenSSL will tell you the truth.
Verify.
\n\n$ .venv/bin/ansible -i ansible/inventory/lab.yml backup1,pg1 -m ping\nbackup1 | SUCCESS => { \"ping\": \"pong\" }\npg1 | SUCCESS => { \"ping\": \"pong\" }\nThe playbook is four jobs in order, and the order is load-bearing:
\n\npgbackrest and create the system user/dirs on every TLS node.ca.crt, <host>.crt, <host>.key to /etc/pgbackrest/certs/.8432/tcp on the repo host, start pgbackrest-server, then prove from pg1.Install the PGDG EL8 repo RPM, then dnf install the newest pgbackrest. On this run that resolved to 2.59.0. Disable the AppStream postgresql module if it exists; on this box the module was already absent, so that task is allowed to fail.
Certificates are created on the Mac so the CA private key never lives on a guest except as a local artifact you can delete:
\n\nopenssl genrsa -out ca.key 4096\nopenssl req -new -x509 -days 3650 -key ca.key -out ca.crt \\\n -subj \"/O=Lab/CN=lab-pgbackrest-ca\"\n\n# per host: CN = hostname, SAN = DNS:hostname,IP:lab_ip\nopenssl genrsa -out backup1.key 2048\nopenssl req -new -key backup1.key -out backup1.csr -subj \"/O=Lab/CN=backup1\"\n# extfile: serverAuth,clientAuth + subjectAltName=DNS:backup1,IP:192.168.105.130\nopenssl x509 -req -in backup1.csr -CA ca.crt -CAkey ca.key -CAcreateserial \\\n -out backup1.crt -days 3650 -extfile backup1.ext\nchmod 600 ca.key backup1.key\nBoth serverAuth and clientAuth are required. pgBackRest TLS is mutual: the repo host presents a server cert, the client presents a client cert, and tls-server-auth matches the client CN to a stanza list. A SAN-only cert with a wrong CN will handshake in openssl s_client and still fail pgBackRest auth.
Repo host config:
\n\n[global]\nlog-level-console=info\nlog-level-file=detail\nlog-path=/var/log/pgbackrest\nrepo1-path=/var/lib/pgbackrest\nrepo1-retention-full=2\nrepo1-bundle=y\nrepo1-block=y\ntls-server-address=*\ntls-server-port=8432\ntls-server-cert-file=/etc/pgbackrest/certs/backup1.crt\ntls-server-key-file=/etc/pgbackrest/certs/backup1.key\ntls-server-ca-file=/etc/pgbackrest/certs/ca.crt\ntls-server-auth=pg1=*\ntls-server-auth=pg2=*\n\n[demo]\n# pg1-path filled when Postgres is installed\nFour lines are load-bearing.
\n\ntls-server-address=*. Listen on every interface, including the lab NIC. localhost would make server-ping from pg1 hang.
tls-server-auth=pg1=*. Authorize the client certificate whose CN is pg1 for every stanza. Until Postgres exists the stanza is a placeholder; the auth line is still required or the later backup is refused as an unauthorized client.
repo1-bundle / repo1-block. Current PGDG defaults for a new repo. Harmless on an empty repo; leave them so the next post does not change file format mid-series.
Keys 0600, owned by pgbackrest. The systemd unit does not run as root. A root-owned key makes pgbackrest server fail after a restart that looks unrelated.
Client config (no local repo path — the repo is on backup1):
[global]\nlog-level-console=info\nlog-path=/var/log/pgbackrest\nrepo1-host=backup1\nrepo1-host-type=tls\nrepo1-host-user=pgbackrest\nrepo1-host-port=8432\nrepo1-host-cert-file=/etc/pgbackrest/certs/pg1.crt\nrepo1-host-key-file=/etc/pgbackrest/certs/pg1.key\nrepo1-host-ca-file=/etc/pgbackrest/certs/ca.crt\ntls-server-address=*\ntls-server-port=8432\ntls-server-cert-file=/etc/pgbackrest/certs/pg1.crt\ntls-server-key-file=/etc/pgbackrest/certs/pg1.key\ntls-server-ca-file=/etc/pgbackrest/certs/ca.crt\ntls-server-auth=backup1=*\nThe client also runs a TLS server so the repo host can call back once Postgres exists. Mutual TLS is the model, not “open 8432 and hope.”
\n\n[Unit]\nDescription=pgBackRest TLS server\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=simple\nUser=pgbackrest\nGroup=pgbackrest\nExecStart=/usr/bin/pgbackrest server\nRestart=on-failure\nRestartSec=3\n\n[Install]\nWantedBy=multi-user.target\nfirewalld on backup1 only: firewall-cmd --permanent --add-port=8432/tcp then --reload. If the unit is inactive, server-ping fails for a mundane reason: nothing is listening.
Verify.
\n\n$ systemctl is-active pgbackrest-server\nactive\n\n$ ss -lntp | grep 8432\nLISTEN 0 100 0.0.0.0:8432 0.0.0.0:* users:((\"pgbackrest\",pid=…,fd=4))\n\n$ pgbackrest version\npgBackRest 2.59.0\nserver-ping in 2.59 is an aliveness check. It does not take repo1-host-*. Ping address and port:
sudo -u pgbackrest pgbackrest server-ping \\\n --tls-server-address=backup1 \\\n --tls-server-port=8432\nThen prove the client certificate is accepted. server-ping does not authenticate; a green ping with a broken client cert is a false pass.
echo | openssl s_client -connect 192.168.105.130:8432 \\\n -cert /etc/pgbackrest/certs/pg1.crt \\\n -key /etc/pgbackrest/certs/pg1.key \\\n -CAfile /etc/pgbackrest/certs/ca.crt \\\n -servername backup1 2>/dev/null \\\n | grep -E 'Verify return code:|subject='\nVerify.
\n\n# from pg1 — measured\nserver-ping command begin 2.59.0: --tls-server-address=backup1 --tls-server-port=8432\nserver-ping command end: completed successfully (84ms)\n\nsubject=O = Lab, CN = backup1\nissuer=O = Lab, CN = lab-pgbackrest-ca\nVerify return code: 0 (ok)\nThat pair is the gate. A listening socket without mTLS success is not enough. Ansible runs both checks at the end of the playbook so a green recap means the handshake, not merely that packages installed.
\n\nFailure boundary. Passing repo1-host=… on server-ping fails with ERROR: [031]: option 'repo-host' not valid for command 'server-ping'. That is a CLI mismatch, not a dead server. Use --tls-server-address and --tls-server-port, then prove auth with openssl s_client. A later pgbackrest info / backup is the first command that actually exercises tls-server-auth.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
future feature annotations is not defined | \n Control Ansible too new for guest Python 3.6 | \nPython 3.12 venv + ansible-core 2.15.x | \n
Invalid conditional … ast.Str | \n Ansible running on Python 3.14 | \nSame venv; do not use Homebrew ansible | \n
| Lab IPs unreachable | \nFabric / subnet collision | \nFix socket_vmnet first | \n
option 'repo-host' not valid for command 'server-ping' | \n Wrong options for 2.59 | \n--tls-server-address + port | \n
Port open, Verify return code ≠ 0 | \n Wrong CA, missing SAN, or CN ≠ hostname | \nRe-issue certs; SAN must include DNS and lab IP | \n
server-ping hangs | \n tls-server-address=localhost or firewalld closed | \n Listen on *; open 8432/tcp on backup1 | \n
| Unit active, process cannot read key | \nKey not 0600 / not owned by pgbackrest | \n Fix ownership before restarting | \n
dnf cannot see pgbackrest | \n PGDG repo missing | \nRe-install pgdg-redhat-repo-latest | \n
vagrant halt backup1 pg1\n# or: vagrant destroy -f backup1 pg1\nLeave socket_vmnet running if you will rebuild. Rotate certs by deleting ansible/artifacts/certs/ and re-running the generate play. Do not commit *.key.
Next in this series: install Postgres on the clients, fill [demo] with pg1-path, and take the first backup over this TLS repo. The operating rule is dedicated repo host, Ansible-owned TLS, measured handshake before any pg_basebackup story. When that post lands, this one should still be a clean prerequisite, not a fossil inside a combined mega-role.
I wanted a small cluster on this Mac that behaved like the amd64 VMs I run everywhere else: Ubuntu next to a RHEL-family node, scriptable SSH, and a private network where the host can ping every guest and the guests can ping each other. That is the minimum before Patroni, Postgres 18, or any HA story is worth installing.
\n\nWhat I kept getting instead was the worst kind of almost-working lab. vagrant up finished. vagrant ssh worked. The guest claimed an address like 192.168.1.111. And then the host could not reach that address, peers could not reach each other, and half an afternoon disappeared into “networking is fine, something else must be wrong.”
Two mistakes caused most of it.
\n\nThe first is architecture. Apple Silicon makes aarch64 boxes feel natural. Production VMs are almost always amd64. The moment the lab lies about the instruction set, every later package, extension, and timing story is suspect. So the guests here are deliberate: qemu-system-x86_64 under TCG, slow on purpose, honest about the CPU.
The second is the subnet. Many older Vagrantfiles hard-code 192.168.1.x. On this Mac, Wi‑Fi (en0) was already 192.168.1.68/24. The lab and the LAN were the same /24. Port-forwarded SSH still worked through SLIRP. Everything that depended on a real L2 fabric—host ping, inter-node ping, later VIP tricks—did not. The repair is boring and non-negotiable: give the lab its own unused /24. Mine is 192.168.105.0/24.
This post is the path that survived measurement on one machine: two amd64 guests, Ubuntu plus CentOS 8 as the RHEL stand-in, socket_vmnet for the shared fabric, and proof that host and mesh pings actually return.
\n\nEach guest gets two interfaces on purpose.
\n\n127.0.0.1:<ssh_port>. This is how you get a shell when the lab NIC is still dark.192.168.105.1 and every peer. Static addresses sit above the DHCP end (.100): Ubuntu .111 / .112, Rocky-or-CentOS .121 / .122.The second NIC is not “another private_network line and hope.” It is wired as:
\n\n-device virtio-net-pci,netdev=lab0,mac=…\n-netdev socket,id=lab0,fd=3\nThat fd=3 has to be the unix pipe opened by socket_vmnet_client. When QEMU opened the unix socket itself with a native stream connect, the guest’s eth1 transmitted forever and received nothing. lsof on fd 3 is the cheap tell: you want TYPE=unix with a live peer, not a pipe left over from an IDE shell.
brew install qemu socket_vmnet\nvagrant plugin install vagrant-qemu\nvagrant plugin list | grep qemu\nYou want vagrant-qemu ≥ 0.6.3. The 0.3.x line does not give you a reliable path for this pattern.
\n\nsocket_vmnet has to stay up for the whole boot. A one-shot process that dies while TCG is still grinding through cloud-init will leave you debugging a dead bridge. Install it as a LaunchDaemon:
\n\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n<dict>\n <key>Label</key><string>lab.socket_vmnet</string>\n <key>ProgramArguments</key>\n <array>\n <string>/opt/homebrew/opt/socket_vmnet/bin/socket_vmnet</string>\n <string>--vmnet-gateway=192.168.105.1</string>\n <string>--vmnet-dhcp-end=192.168.105.100</string>\n <string>/opt/homebrew/var/run/socket_vmnet</string>\n </array>\n <key>RunAtLoad</key><true/>\n <key>KeepAlive</key><true/>\n <key>StandardOutPath</key><string>/opt/homebrew/var/log/socket_vmnet/stdout</string>\n <key>StandardErrorPath</key><string>/opt/homebrew/var/log/socket_vmnet/stderr</string>\n</dict>\n</plist>\nsudo mkdir -p /opt/homebrew/var/log/socket_vmnet\nsudo cp lab.socket_vmnet.plist /Library/LaunchDaemons/\nsudo launchctl bootstrap system /Library/LaunchDaemons/lab.socket_vmnet.plist\n# after edits: sudo launchctl kickstart -kp system/lab.socket_vmnet\nVerify.
\n\n$ ifconfig bridge100 | grep 'inet '\n\tinet 192.168.105.1 netmask 0xffffff00 broadcast 192.168.105.255\n$ ls -l /opt/homebrew/var/run/socket_vmnet\nsrwxr-xr-x ... /opt/homebrew/var/run/socket_vmnet\nFailure boundary. If your home LAN already owns 192.168.105.0/24, pick another free /24 and change both the daemon and every lab IP. Do not “just use 192.168.1.0/24” because a Patroni gist from 2019 did. That is how this whole mess started.
vagrant box add generic/ubuntu2204 --provider libvirt\nvagrant box add generic/centos8 --provider libvirt\ngeneric/centos8 is the RHEL-family stand-in that was measured here. Swap in Rocky when you have a libvirt box cached; the networking story does not change.
Vagrant’s ChildProcess likes to leave pipes on descriptors ≥ 3. socket_vmnet_client needs a clean fd 3 for the lab NIC. The host wrapper closes everything above stderr, then execs the client onto the real QEMU binary:
#!/bin/bash\nset -euo pipefail\nREAL_QEMU=\"${REAL_QEMU:-/opt/homebrew/opt/qemu/bin/qemu-system-x86_64}\"\nCLIENT=\"${SOCKET_VMNET_CLIENT:-/opt/homebrew/opt/socket_vmnet/bin/socket_vmnet_client}\"\nSOCK=\"${SOCKET_VMNET_SOCK:-/opt/homebrew/var/run/socket_vmnet}\"\n\n[[ -S \"$SOCK\" ]] || { echo \"socket_vmnet not running at $SOCK\" >&2; exit 1; }\n\nfor fd in $(ls /dev/fd 2>/dev/null | grep -E '^[0-9]+$' | sort -n); do\n [[ \"$fd\" -gt 2 ]] && eval \"exec ${fd}<&-\" 2>/dev/null || true\ndone\n\nexec \"$CLIENT\" \"$SOCK\" \"$REAL_QEMU\" \"$@\"\nPoint the provider at that wrapper. Do not daemonize QEMU here—forking drops the inherited lab fd. The load-bearing provider lines look like this:
\n\nconfig.vm.provider \"qemu\" do |qe|\n qe.arch = \"x86_64\"\n qe.machine = \"q35\"\n qe.cpu = \"max\"\n qe.smp = \"2\"\n qe.memory = \"1536\"\n qe.ssh_port = 22111\n qe.net_device = \"virtio-net-pci\"\n qe.no_daemonize = true\n qe.qemu_bin = File.join(File.dirname(__FILE__), \"bin\", \"qemu-system-x86_64\")\n qe.extra_qemu_args = [\n \"-device\", \"virtio-net-pci,netdev=lab0,mac=52:54:00:12:00:11\",\n \"-netdev\", \"socket,id=lab0,fd=3\",\n ]\nend\nA short provisioner then finds the non-SLIRP NIC (prefer the known MAC), writes a static address with netplan or NetworkManager, and records /etc/lab/ip. That is ordinary guest plumbing; the unusual part already happened on the host.
Verify.
\n\nAfter vagrant up has started QEMU:
$ lsof -p \"$(pgrep -n qemu-system-x86_64)\" -d 3\nCOMMAND PID USER FD TYPE DEVICE ... NAME\nqemu-syst ... 3u unix ... ->0x…\nIf fd 3 is a pipe or missing a peer, stop and fix the wrapper before you chase guest routes.
\n\nTCG first boot is slow. Several minutes of SSH retries is normal, not a hung VM. Raise boot_timeout (I use 900). If your shell—Cursor, VS Code, anything that injects pipes—still pollutes descriptors, close them before vagrant up:
( exec 3<&- 4<&- 5<&- 6<&- 7<&- 8<&- 9<&-\n vagrant up ubuntu1 rocky1 --provider=qemu\n)\nBring up one Ubuntu node and one RHEL-family node first. Four nodes fit the same Vagrantfile later; two are enough to prove L2.
\n\nVerify.
\n\nFrom the Mac host, after provision finishes:
\n\n$ ping -c 3 192.168.105.111\n3 packets transmitted, 3 packets received, 0.0% packet loss\n# ~1–2 ms on this rig\n\n$ ping -c 3 192.168.105.121\n3 packets transmitted, 3 packets received, 0.0% packet loss\n# ~2–3 ms\nVerify.
\n\nThen from inside each guest:
\n\n$ vagrant ssh ubuntu1 -c 'ping -c 3 192.168.105.121'\n3 packets transmitted, 3 received, 0% packet loss\n# ~2–4 ms\n\n$ vagrant ssh rocky1 -c 'ping -c 3 192.168.105.111'\n3 packets transmitted, 3 received, 0% packet loss\n# ~1–2 ms\nThose four checks are the gate. Guest ping to 192.168.105.1 may still show 100% loss while ip neigh shows the gateway REACHABLE. On this macOS vmnet setup the gateway answers ARP and carries L2, but often ignores ICMP. Do not fail the lab on gateway ping. Fail it when peers or the host cannot reach the lab IPs.
One more SSH gotcha under TCG: if your agent offers a long list of keys, you can burn through MaxAuthTries before the Vagrant key is tried. Prefer IdentitiesOnly=yes with the key Vagrant manages, or just use vagrant ssh.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
| Lab IPs look plausible but host/peers cannot reach them | \nLab subnet equals Wi‑Fi (192.168.1.0/24 here) | \n Move lab to an unused /24; change daemon + Vagrant IPs together | \n
| eth1 TX rises, RX stays 0 | \nfd 3 is not the vmnet client pipe, or QEMU used stream | \n Wrapper + socket_vmnet_client; confirm lsof … -d 3 is unix | \n
| socket_vmnet vanishes mid-boot | \nOne-shot daemon | \nLaunchDaemon with KeepAlive | \n
Too many authentication failures | \n Agent offers many keys while TCG is slow | \nIdentitiesOnly=yes / vagrant ssh | \n
| First boot “hangs” for ages | \nx86_64 under TCG | \nRaise boot_timeout; wait; watch SSH retries, not wall-clock panic | \n
vagrant halt ubuntu1 rocky1\n# or: vagrant destroy -f ubuntu1 rocky1\nLeave lab.socket_vmnet installed while you are iterating. Stop it only when you want bridge100 gone.
The operating rule for the Postgres 18+ / Patroni write-ups that sit on top of this Mac: amd64 guests, a dedicated lab subnet, socket_vmnet_client on fd 3, and a green host-plus-mesh check before a single package install. Architecture fidelity without L2 fidelity is just a slow container with extra ceremony. Get the fabric honest first; then the database work can mean what it says.
\n", "summary": "Apple Silicon labs that boot still fail host and mesh ping when you use aarch64 boxes or park the VMs on your Wi‑Fi subnet.", "date_published": "2026-08-13T10:00:00+05:45", "date_modified": "2026-08-13T00:00:00+05:45", "tags": ["self-hosted","postgres"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/building-data-platforms-and-ai-products/", "url": "https://basantbhattarai.com.np/writing/building-data-platforms-and-ai-products/", "title": "The data platform is part of the product", "content_html": "I used to draw data platforms underneath the product: neat boxes for ingestion, storage, transformation, and serving, with the “real” application sitting safely above them. The diagram was tidy and the boundary was wrong.
\n\nA user does not experience those boxes. They experience one answer. If a source arrives late, a transformation accepts an impossible value, a cache serves an old projection, and an AI workflow confidently acts on it, every individual service can be green while the product is wrong.
\n\nThat changed the question I ask in design reviews. I no longer begin with, “What should process this data?” I begin with, “What promise is the product making, and how will we repair that promise when one layer lies?”
\n\nThis is the operating model I now use. It is intentionally independent of any employer architecture or vendor stack.
\n\n“The pipeline runs every hour” is a schedule, not a reliability promise. The consumer cares whether the expected fact is queryable before a decision becomes stale.
\n\nFor each product-facing dataset or endpoint, I write down five things:
\n\nThat short contract changes the implementation. Freshness becomes a property of a published fact rather than a scheduler. Quarantine becomes a visible state instead of a forgotten dead-letter queue. A backfill becomes part of the interface, because a correction that never reaches the product is not a correction.
\n\nI also separate availability from fitness. A service can answer every request and still serve a stale or semantically invalid record. For data products, I want at least these signals beside ordinary uptime:
\n\n| Signal | \nQuestion it answers | \n
|---|---|
| Freshness | \nIs the newest expected fact available inside its decision window? | \n
| Completeness | \nDid the expected population arrive, not merely some records? | \n
| Validity | \nDo values satisfy the contract at the publication boundary? | \n
| Decision latency | \nHow long from source event to a product action? | \n
| Repair time | \nHow long from detection to a corrected product state? | \n
Whenever automated output can change another workflow, I persist a decision record before I persist the effect. The record needs enough information to explain what happened without reconstructing it from logs scattered across services.
\n\nThis small standard-library example is deliberately runnable. It validates confidence, requires source references, fixes the set of allowed outcomes, and emits a JSON-ready record. The type hints help the editor; the runtime checks protect the boundary.
\n\nfrom __future__ import annotations\n\nfrom dataclasses import asdict, dataclass\nfrom datetime import UTC, datetime\nfrom typing import Literal\n\nOutcome = Literal[\"approve\", \"reject\", \"needs_review\"]\nALLOWED_OUTCOMES = {\"approve\", \"reject\", \"needs_review\"}\n\n\n@dataclass(frozen=True, slots=True)\nclass DecisionRecord:\n subject_id: str\n outcome: Outcome\n confidence: float\n policy_version: str\n source_refs: tuple[str, ...]\n created_at: datetime\n\n def __post_init__(self) -> None:\n if not self.subject_id.strip():\n raise ValueError(\"subject_id must not be blank\")\n if self.outcome not in ALLOWED_OUTCOMES:\n raise ValueError(f\"unsupported outcome: {self.outcome}\")\n if not 0.0 <= self.confidence <= 1.0:\n raise ValueError(\"confidence must be between 0 and 1\")\n if not self.policy_version.strip():\n raise ValueError(\"policy_version must not be blank\")\n if not self.source_refs or any(not ref.strip() for ref in self.source_refs):\n raise ValueError(\"at least one non-blank source reference is required\")\n if self.created_at.tzinfo is None:\n raise ValueError(\"created_at must include a timezone\")\n\n def to_dict(self) -> dict[str, object]:\n payload = asdict(self)\n payload[\"source_refs\"] = list(self.source_refs)\n payload[\"created_at\"] = self.created_at.isoformat()\n return payload\n\n\nrecord = DecisionRecord(\n subject_id=\"case-1042\",\n outcome=\"needs_review\",\n confidence=0.71,\n policy_version=\"risk-policy-2026-08\",\n source_refs=(\"snapshot:6b8f\", \"rule:missing-history\"),\n created_at=datetime.now(UTC),\n)\n\nprint(record.outcome)\nprint(record.to_dict()[\"policy_version\"])\nRunning the file should produce the outcome and policy version:
\n\n$ python3 contracts/decision_record.py\nneeds_review\nrisk-policy-2026-08\nThe database should defend the same boundary. Application validation gives a useful error close to the producer; database constraints protect every write path, including scripts and future services.
\n\nCREATE TABLE decision_records (\n decision_id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n idempotency_key text NOT NULL UNIQUE,\n subject_id text NOT NULL CHECK (btrim(subject_id) <> ''),\n outcome text NOT NULL\n CHECK (outcome IN ('approve', 'reject', 'needs_review')),\n confidence numeric(5,4) NOT NULL\n CHECK (confidence BETWEEN 0 AND 1),\n policy_version text NOT NULL CHECK (btrim(policy_version) <> ''),\n source_refs jsonb NOT NULL\n CHECK (jsonb_typeof(source_refs) = 'array'\n AND jsonb_array_length(source_refs) > 0),\n created_at timestamptz NOT NULL,\n recorded_at timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE INDEX decision_records_subject_time_idx\n ON decision_records (subject_id, created_at DESC);\nNotice what is absent: a raw prompt as the only evidence, a free-form verdict, and an overwritten “current answer.” Those make a demo easy and a correction almost impossible.
\n\nProduct reads often need a shape that the source of truth should not have. A low-latency endpoint may want one precomputed row per subject; an audit may need the full history; an analyst may need a columnar projection. Those are different read paths, not competing definitions of truth.
\n\nI use three layers of responsibility:
\n\nThe third layer is disposable by design. If a projection is corrupted or its model changes, I should be able to rebuild it from the first two layers without inventing history. That repair path is more valuable than choosing a fashionable serving engine.
\n\nThis distinction also makes corrections honest. A new policy should append a new decision; it should not mutate the old one until the audit trail agrees with the present. The product can point to the active decision while the history explains how it got there.
\n\nAI does not need a separate philosophy of reliability. It needs the same producer boundary as any source we do not fully control, with stricter attention to provenance and evaluation.
\n\nI model the workflow as explicit states:
\n\nproposed -> validated -> applied\n | |\n | +-> needs_review\n +--------------> rejected\n“Proposed” is important. Model output has not become a product fact merely because inference returned successfully. It still has to pass structural validation, policy checks, evidence requirements, and—where the consequence justifies it—human review.
\n\nConfidence is not a universal truth meter either. A threshold only has meaning against an evaluated task, versioned data, and a defined cost for false approval versus false rejection. If those are missing, 0.93 is decoration.
For every automated decision, I want to answer:
\n\nIf the system cannot answer those questions, it is not ready to make a consequential change unattended.
\n\nRetries happen everywhere: queues redeliver, clients time out after a successful write, operators replay a date range, and backfills overlap live traffic. “Exactly once” is often a property claimed by one component while the business effect crosses several.
\n\nI prefer an idempotency key derived from the stable inputs to the decision, such as:
\n\nfrom hashlib import sha256\n\n\ndef decision_key(\n subject_id: str,\n policy_version: str,\n source_snapshot: str,\n) -> str:\n canonical = \"\\x1f\".join((subject_id, policy_version, source_snapshot))\n return sha256(canonical.encode(\"utf-8\")).hexdigest()\nThe write path then treats a duplicate as a known result, not a second decision:
\n\nINSERT INTO decision_records (\n idempotency_key,\n subject_id,\n outcome,\n confidence,\n policy_version,\n source_refs,\n created_at\n)\nVALUES (\n :idempotency_key,\n :subject_id,\n :outcome,\n :confidence,\n :policy_version,\n CAST(:source_refs AS jsonb),\n :created_at\n)\nON CONFLICT (idempotency_key) DO NOTHING\nRETURNING decision_id;\nThere is a trade-off: the key encodes what I consider the identity of a decision. Changing that definition is a contract change. I document it and version the policy rather than hiding it in a helper function.
\n\nReplay needs a product rule too. When a corrected source record produces a different outcome, does the system apply it automatically, open a review, or leave the prior effect in place? That answer depends on consequence, not throughput. The platform can provide the mechanism; the product must own the policy.
\n\nEnd-to-end ownership does not mean one person builds every service. It means the seams share one definition of correctness and one repair story.
\n\nThis is the checklist I use before a data-backed or AI-assisted workflow ships:
\n\nThe principle underneath all of this is modest: the platform is part of the product because it decides which facts the product may trust. Once that is explicit, architecture discussions get less theatrical. The useful questions become concrete: What is the promise? Where is it recorded? How does it fail? Who can repair it? And can they do so while the rest of the system keeps moving?
\n", "summary": "A practical operating model for reliable data and AI products: consumer-facing promises, typed decisions, idempotency, provenance, and repair.", "date_published": "2026-08-11T10:00:00+05:45", "date_modified": "2026-08-11T16:00:00+05:45", "tags": ["ai-agents","data-quality"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/postgresql-18-docker-volume-path-change/", "url": "https://basantbhattarai.com.np/writing/postgresql-18-docker-volume-path-change/", "title": "PostgreSQL 18 won't take a volume at /var/lib/postgresql/data", "content_html": "There is one line that appears in every PostgreSQL docker-compose file written between 2017 and 2025:
\n\nvolumes:\n - pgdata:/var/lib/postgresql/data\nOn postgres:18 it does not work. The container starts, complains about data in an unused mount or volume, and your data does not end up where you think it does.
The image changed where it wants the volume. In 12 through 17 the data directory was the mount point. In 18 the mount point is the parent, /var/lib/postgresql, and the cluster lives in an 18/docker subdirectory beneath it. The reason is pg_upgrade --link: an in-place major upgrade needs the old and new data directories side by side on the same filesystem, and that is impossible when the data directory is itself the mount point. Moving the mount up one level makes room for 18/docker and 19/docker to coexist during an upgrade.
volumes:\n- - pgdata:/var/lib/postgresql/data\n+ - pgdata:/var/lib/postgresql\nThat is the whole fix, and if that is all you came for you can stop here. The rest of this post builds a primary and a streaming hot standby on top of the corrected file, measures the replication lag, promotes the standby, and sets up WAL archiving for point-in-time recovery — which is where the second half of the change bites, because a replica cloned by pg_basebackup needs PGDATA set explicitly.
The compose file and the init script are reproduced inline. There is no public repository to clone.
\n\nGotcha. The message you get contains the phrase data in unused mount/volume, which reads like a warning about a stray volume rather than what it is — the image telling you the data directory is not where the mount is. Searching for the phrase is how most people find this; the fix is not in the message.
Two services: a primary with the replication settings turned on, and a replica that clones itself from the primary on first boot and then streams.
\n\nname: pg-replication\n\nservices:\n pgprimary:\n image: postgres:18\n environment:\n POSTGRES_USER: admin\n POSTGRES_PASSWORD: admin123\n POSTGRES_DB: poc\n command:\n - postgres\n - -c\n - wal_level=replica\n - -c\n - max_wal_senders=10\n - -c\n - max_replication_slots=10\n - -c\n - hot_standby=on\n ports: [\"5434:5432\"]\n # PG18 image wants the mount at /var/lib/postgresql (data lands in an 18/docker\n # subdir) — NOT /var/lib/postgresql/data, which the image now rejects.\n volumes:\n - pgp:/var/lib/postgresql\n - ./primary-init.sh:/docker-entrypoint-initdb.d/primary-init.sh:ro\n\n pgreplica:\n image: postgres:18\n environment:\n PGDATA: /var/lib/postgresql/18/docker\n depends_on: [pgprimary]\n ports: [\"5435:5432\"]\n # If the data dir is empty, clone the primary with pg_basebackup, then start as\n # a hot standby. Runs the clone as the postgres user so file ownership is correct.\n command:\n - bash\n - -c\n - |\n set -e\n if [ ! -s \"$$PGDATA/PG_VERSION\" ]; then\n until pg_isready -h pgprimary -p 5432 -U admin -d poc; do\n echo \"waiting for primary...\"; sleep 2;\n done\n export PGPASSWORD=replpass\n gosu postgres pg_basebackup -h pgprimary -p 5432 -U replicator \\\n -D \"$$PGDATA\" -Fp -Xs -R -P -S replica1_slot\n fi\n exec docker-entrypoint.sh postgres\n volumes:\n - pgr:/var/lib/postgresql\n\nvolumes:\n pgp:\n pgr:\nFour details in there are load-bearing.
\n\nPGDATA: /var/lib/postgresql/18/docker on the replica. The primary gets this path from the entrypoint’s own default because it runs initdb. The replica does not run initdb — it runs pg_basebackup, and pg_basebackup writes wherever -D points. Without PGDATA set, the clone lands in the wrong place and the server then starts against an empty directory. This is the same mount-point change, showing up a second time in a shape that does not mention mounts at all.
$$PGDATA, with two dollar signs. Compose does variable interpolation on the file before Docker ever sees it. $$ escapes to a literal $, so the shell inside the container gets $PGDATA. A single $ here means Compose substitutes an empty string and the test becomes [ ! -s /PG_VERSION ].
gosu postgres around pg_basebackup. The command runs as root. A data directory full of root-owned files makes the server refuse to start, with an ownership error that gives no hint about which command created them.
-Xs -R -S replica1_slot on pg_basebackup. -Xs streams WAL alongside the base backup, so the backup is consistent without needing archived WAL. -R writes standby.signal and a primary_conninfo line so the clone comes up as a standby with no further editing. -S binds it to a named replication slot.
Verify. Nothing to run yet — but read the volume lines once more. Both are :/var/lib/postgresql, not :/var/lib/postgresql/data.
The entrypoint runs anything in /docker-entrypoint-initdb.d/ once, on the first boot, after initdb and before the server is opened to the network. That is the right place to create the replication role, the slot, and the pg_hba.conf line that lets the replica in.
#!/bin/bash\n# Runs once on the primary during initdb. Creates a replication role and a slot,\n# and allows replication connections in pg_hba.conf.\nset -e\n\npsql -v ON_ERROR_STOP=1 -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\" <<-EOSQL\n CREATE ROLE replicator WITH REPLICATION LOGIN PASSWORD 'replpass';\n SELECT pg_create_physical_replication_slot('replica1_slot');\nEOSQL\n\n# Allow the replica to connect for streaming replication.\necho \"host replication replicator all scram-sha-256\" >> \"$PGDATA/pg_hba.conf\"\nThe pg_hba.conf line is easy to forget because replication is a pseudo-database — a normal host all all ... rule does not cover replication connections, so the role can log in for queries and still be refused for streaming.
A physical replication slot makes the primary keep WAL that the replica has not consumed yet. Without one, a replica that falls far enough behind gets its WAL recycled out from under it and has to be rebuilt. With one, you get the opposite failure mode: a replica that is down forever makes the primary’s WAL grow without bound. Pick your problem knowingly and monitor for it.
\n\nchmod +x pg-replication/primary-init.sh\nVerify.
\n\n$ ls -l pg-replication/primary-init.sh\n-rwxr-xr-x 1 basant staff 500 Aug 11 09:00 pg-replication/primary-init.sh\ndocker compose -f pg-replication/docker-compose.yml up -d\nThe replica polls pg_isready until the primary answers, then clones. On this rig that took a few seconds; -P prints basebackup progress into the container log while it happens.
Verify. The replica’s log walks through four states, in this order. Timestamps and LSNs are elided here — yours will differ:
\n\n$ docker compose -f pg-replication/docker-compose.yml logs pgreplica\n... entering standby mode\n... consistent recovery state reached\n... database system is ready to accept read-only connections\n... started streaming WAL ... on timeline 1\nAll four have to appear. “Entering standby mode” alone means the clone worked but streaming has not started. “Ready to accept read-only connections” without “started streaming WAL” means the standby is replaying from disk but has not connected back to the primary — usually the pg_hba.conf line or the slot.
Two separate promises, and they are worth checking separately: the standby has your data, and the standby will not accept a write that would diverge from the primary.
\n\nOn the primary:
\n\ndocker exec -it pg-replication-pgprimary-1 psql -U admin -d poc\nCREATE TABLE t (id int);\nINSERT INTO t (id) VALUES (1);\nOn the replica:
\n\ndocker exec -it pg-replication-pgreplica-1 psql -U admin -d poc\nVerify. The row is there, and the write is refused:
\n\npoc=# SELECT * FROM t;\n id\n----\n 1\n(1 row)\n\npoc=# INSERT INTO t (id) VALUES (2);\nERROR: cannot execute INSERT in a read-only transaction\nThat error is the standby doing its job. hot_standby=on buys you read traffic on the replica, not a second writable node — the moment two nodes accept conflicting writes on the same timeline you have a reconciliation problem that PostgreSQL will not solve for you.
“Replication lag” is three different numbers and people quote whichever one is smallest. pg_stat_replication on the primary gives all of them: write_lag is time until the standby wrote the WAL, flush_lag until it fsynced, replay_lag until it applied and the data became visible to a reader. replay_lag is the one that matters for read-after-write, and it is always the largest.
SELECT application_name, state, sync_state, write_lag, replay_lag,\n pg_wal_lsn_diff(sent_lsn, replay_lsn) AS lag_bytes\n FROM pg_stat_replication;\nVerify.
\n\n application_name | state | sync_state | write_lag | replay_lag | lag_bytes\n------------------+-----------+------------+-----------------+-----------------+-----------\n walreceiver | streaming | async | 00:00:00.000104 | 00:00:00.000383 | 0\n(1 row)\nwrite_lag ≈ 104 µs, replay_lag ≈ 383 µs, and zero bytes of outstanding WAL. Both containers are on the same host with no network in between, so treat these as the protocol’s floor rather than anything you will see across a WAN — but they do establish that when production lag is measured in seconds, the cost is network and disk, not PostgreSQL.
sync_state reads async, which is the default and the right default for most setups: the primary does not wait for the standby to acknowledge, so a stalled standby cannot stall your writes. It also means a hard primary failure can lose the last few transactions. If that is unacceptable, synchronous_standby_names trades write latency for that guarantee.
Check the slot too:
\n\nSELECT slot_name, slot_type, active, wal_status FROM pg_replication_slots;\n slot_name | slot_type | active | wal_status\n---------------+-----------+--------+------------\n replica1_slot | physical | t | reserved\n(1 row)\nactive = t means a replica is attached. wal_status = reserved means the WAL the slot needs is still within max_slot_wal_keep_size. When that column reads extended, unreserved or lost, you are on the path to either a full disk or a replica that can never catch up. It is the single best column to alert on for this topology.
There is no automatic failover here, and that is not an oversight. PostgreSQL leaves the decision to something that can tell a dead primary apart from an unreachable one — Patroni with etcd and a watchdog, or repmgr. What those tools ultimately call is this:
\n\ndocker exec pg-replication-pgreplica-1 psql -U admin -d postgres -c \"SELECT pg_promote();\"\nVerify. Promotion returns true, recovery ends, and writes are accepted:
\n\npostgres=# SELECT pg_promote();\n pg_promote\n------------\n t\n(1 row)\n\npostgres=# SELECT pg_is_in_recovery();\n pg_is_in_recovery\n-------------------\n f\n(1 row)\npoc=# INSERT INTO t (id) VALUES (2);\nINSERT 0 1\nAnd the timeline advances:
\n\npoc=# SELECT timeline_id FROM pg_control_checkpoint();\n timeline_id\n-------------\n 2\n(1 row)\nThat bump from 1 to 2 is the most important thing on this page after the volume path. A timeline is PostgreSQL’s record of a divergence: the new primary is now writing history that the old primary knows nothing about.
\n\nDo not restart the old primary and point traffic at it. After a promotion, the old primary is on timeline 1 and the new one is on timeline 2. They are two different databases that agree about the past. Bringing the old one back as a writable node is how you get split brain. It has to be rewound onto the new timeline with pg_rewind, or rebuilt from a fresh pg_basebackup, before it can rejoin as a standby.
Replication protects you from a node dying. It does not protect you from DELETE without a WHERE clause, because the standby replays that faithfully in 383 microseconds. Point-in-time recovery is the other half, and it is two pieces: a base backup, plus every WAL segment written since.
Add archiving to the primary’s command list and give it somewhere to write:
\n\n command:\n - postgres\n - -c\n - wal_level=replica\n - -c\n - max_wal_senders=10\n - -c\n - max_replication_slots=10\n - -c\n - hot_standby=on\n - -c\n - archive_mode=on\n - -c\n - archive_command=test ! -f /archive/%f && cp %p /archive/%f\n volumes:\n - pgp:/var/lib/postgresql\n - pgarchive:/archive\n - ./primary-init.sh:/docker-entrypoint-initdb.d/primary-init.sh:ro\n\nvolumes:\n pgp:\n pgr:\n pgarchive:\narchive_mode needs a restart, not a reload. The test ! -f guard makes the command refuse to overwrite an existing segment: archive_command must fail rather than silently clobber, because a segment overwritten is a recovery that stops early.
Take the base backup as the postgres user, for the same ownership reason as the replica clone:
docker exec pg-replication-pgprimary-1 sh -c '\n PGPASSWORD=replpass gosu postgres pg_basebackup \\\n -h localhost -p 5432 -U replicator -D /archive/base -Fp -Xs -P'\nForce a segment switch so there is something in the archive to look at:
\n\ndocker exec pg-replication-pgprimary-1 \\\n psql -U admin -d poc -c \"SELECT pg_switch_wal();\"\nVerify. A base backup directory and at least one archived segment:
\n\n$ docker exec pg-replication-pgprimary-1 ls /archive\nbase\n000000010000000000000003\n000000010000000000000003.00000028.backup\nTo restore to a point in time you start from a copy of /archive/base, add a recovery.signal file to its data directory, and set two parameters in postgresql.conf:
restore_command = 'cp /archive/%f \"%p\"'\nrecovery_target_time = '2026-08-11 09:15:00+05:45'\nThe server then replays archived WAL up to that timestamp and pauses. SELECT pg_wal_replay_resume(); completes the recovery and opens the cluster for writes on a new timeline.
What I measured and what I did not. The replication, lag, promotion and timeline numbers above came off this rig. For PITR I verified the base backup and the archive — the restore_command and recovery_target_time recipe is the documented path, and I have not shown output for a full point-in-time rewind because I did not run one here. Test yours on real data before you rely on it; an untested restore is a hypothesis.
docker compose -f pg-replication/docker-compose.yml down -v\nVerify.
\n\n$ docker volume ls --filter name=pg-replication\nDRIVER VOLUME NAME\nNote that a promoted replica plus a full base backup is three copies of the same cluster on disk. down without -v keeps all three.
| Symptom | \nCause | \nFix | \n
|---|---|---|
Container complains about data in unused mount/volume and the data is not persisted | \n The volume is mounted at /var/lib/postgresql/data, which PG 18 no longer uses as the data directory | \n Mount at /var/lib/postgresql; the cluster lands in 18/docker beneath it | \n
| Replica starts against an empty cluster after a clean clone | \nPGDATA is unset on the replica, so pg_basebackup -D \"$PGDATA\" wrote to the wrong path | \n Set PGDATA: /var/lib/postgresql/18/docker on the replica service | \n
| The replica’s clone test always fires, even on a warm volume | \nSingle $ in the compose command, so Compose interpolated $PGDATA to an empty string | \n Write $$PGDATA in the compose file | \n
| Server refuses to start: data directory has invalid ownership | \npg_basebackup ran as root | \n Run it through gosu postgres | \n
pg_basebackup is rejected with a pg_hba.conf entry error | \n replication is a pseudo-database and is not covered by a host all all rule | \n Append host replication replicator all scram-sha-256 in the init script | \n
pg_basebackup fails saying the replication slot does not exist | \n The slot was never created, or the init script did not run because the volume already had data | \nCreate it with SELECT pg_create_physical_replication_slot('replica1_slot');, or down -v and start clean | \n
| Log shows “ready to accept read-only connections” but never “started streaming WAL” | \nThe standby is replaying from disk and has not connected to the primary | \nCheck primary_conninfo in the standby’s config, the pg_hba.conf line, and the replicator password | \n
pg_stat_replication is empty on the primary | \n No standby is attached | \nConfirm the replica container is running and that the slot shows active = t | \n
| Primary’s disk fills with WAL | \nA replication slot is holding WAL for a replica that is gone | \nWatch wal_status in pg_replication_slots; set max_slot_wal_keep_size, or drop the slot if the replica is not coming back | \n
Writes on the standby fail with cannot execute INSERT in a read-only transaction | \n It is still a standby | \nThis is correct. Promote it with SELECT pg_promote(); only if you actually mean to fail over | \n
| Both nodes accept writes after a failover | \nThe old primary was restarted on timeline 1 while the new one is on timeline 2 | \nNever do this. Rewind the old node with pg_rewind or rebuild it from pg_basebackup before it rejoins | \n
| Recovery stops earlier than the target time | \narchive_command overwrote or skipped a segment | \n Keep the test ! -f guard so archiving fails loudly instead of clobbering | \n
Earlier in this series: A failover lab you can run on one laptop for the rig and the full results matrix, and nodetool refresh only reads upload/ for the ScyllaDB restore that returns zero rows three different ways.
\n", "summary": "The mount point moved in the PG 18 image, so every compose file copied from a 12–17 guide breaks. The corrected file, then replication, promotion and PITR.", "date_published": "2026-08-11T09:00:00+05:45", "date_modified": "2026-08-11T10:20:00+05:45", "tags": ["postgres"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/scylladb-nodetool-refresh-upload-truncate-marker/", "url": "https://basantbhattarai.com.np/writing/scylladb-nodetool-refresh-upload-truncate-marker/", "title": "nodetool refresh only reads upload/ — and TRUNCATE hides your SSTables", "content_html": "I had a good snapshot. I had the SSTable files on disk. I ran nodetool refresh on all three nodes, got no errors, and SELECT count(*) returned zero.
It took three separate discoveries to get the rows back, and none of them produced an error message at any point. That is what makes this expensive: every step in the obvious restore procedure succeeds, and the data still is not there.
\n\nThe three behaviors, stated up front so you can stop reading if one of them is your problem:
\n\nnodetool refresh only loads SSTables from the table’s upload/ subdirectory. Files dropped into the table directory itself are ignored — silently, with a zero exit code.TRUNCATE writes a truncation marker, and that marker hides every SSTable written before it. You can put the old files back and they stay invisible. Recovering from a TRUNCATE needs DROP TABLE and a recreate, not another restore attempt.There is a fourth thing, which happens before any of this and is the reason the cluster would not start in the first place. It gets Step 1.
\n\nThe compose file is reproduced inline below. There is no public repository to clone.
\n\nThe first two Scylla nodes came up fine. The third exited during startup with:
\n\nsystem does not satisfy minimum AIO requirements\nScyllaDB’s Seastar runtime reserves asynchronous I/O contexts per reactor when it starts. fs.aio-max-nr is a kernel-wide ceiling on those contexts, and the Linux VM behind Docker Desktop ships with a low default. Two nodes fit under it. Three did not.
The limit belongs to the VM, not to a container, so raise it from a privileged throwaway container. It applies to every container on that daemon, including ones you start later.
\n\ndocker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576\nVerify.
\n\n$ docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576\nfs.aio-max-nr = 1048576\nGotcha. This sysctl lives inside the Docker VM, and the VM is rebuilt when Docker Desktop restarts. The fix does not persist. If a node that started yesterday refuses to start today with the same AIO message, re-run the privileged container before you look at anything else.
\nThree nodes, each pinned to a single shard and 1200 MB, which is what makes three of them fit on one laptop. scylla1 is the seed; the other two join it.
name: scylla-ha\n\nservices:\n scylla1:\n image: scylladb/scylla:2026.1\n command: [\"--seeds=scylla1\", \"--smp=1\", \"--memory=1200M\", \"--overprovisioned=1\", \"--api-address=0.0.0.0\"]\n ports: [\"9042:9042\"]\n volumes: [\"s1:/var/lib/scylla\"]\n\n scylla2:\n image: scylladb/scylla:2026.1\n command: [\"--seeds=scylla1\", \"--smp=1\", \"--memory=1200M\", \"--overprovisioned=1\", \"--api-address=0.0.0.0\"]\n volumes: [\"s2:/var/lib/scylla\"]\n depends_on: [scylla1]\n\n scylla3:\n image: scylladb/scylla:2026.1\n command: [\"--seeds=scylla1\", \"--smp=1\", \"--memory=1200M\", \"--overprovisioned=1\", \"--api-address=0.0.0.0\"]\n volumes: [\"s3:/var/lib/scylla\"]\n depends_on: [scylla1]\n\nvolumes:\n s1:\n s2:\n s3:\ndepends_on only controls start order, not readiness, so bring the nodes up by hand. Start the seed, wait for it to report Up/Normal, then add one joiner at a time.
docker compose -f scylla-ha/docker-compose.yml up -d scylla1\ndocker exec scylla-ha-scylla1-1 nodetool status\nOnce scylla1 shows UN, add the second, wait again, then the third:
docker compose -f scylla-ha/docker-compose.yml up -d scylla2\ndocker compose -f scylla-ha/docker-compose.yml up -d scylla3\nVerify. nodetool status lists three nodes at UN — Up and Normal — each holding 256 tokens:
UN 172.21.0.2 ... rack1\nUN 172.21.0.3 ... rack1\nUN 172.21.0.4 ... rack1\nA node stuck at UJ is still joining and streaming; give it time rather than restarting it. A node at DN never made it — check that its AIO problem is fixed and that the seed was UN before it started.
RF=3 on a three-node cluster means every node holds every row, which is what makes the QUORUM behavior in the next step easy to reason about.
\n\nCREATE KEYSPACE poc WITH replication = {\n 'class': 'NetworkTopologyStrategy',\n 'datacenter1': 3\n};\n\nCREATE TABLE poc.sensor_by_day (\n sensor_id text,\n day date,\n ts timestamp,\n value double,\n PRIMARY KEY ((sensor_id, day), ts)\n) WITH CLUSTERING ORDER BY (ts DESC);\nThe partition key is (sensor_id, day), not sensor_id alone. Bucketing a time series by day bounds the partition: a sensor that reports for two years does not accumulate one enormous partition that has to be read, repaired and compacted as a unit. This is the single most common modeling mistake in Cassandra-family stores, and it does not show up until the partition is already too big to fix comfortably.
Three rows are enough for the whole exercise:
\n\ndocker exec scylla-ha-scylla1-1 cqlsh -e \"\nINSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:00:00',42.0);\nINSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:01:00',43.5);\nINSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:02:00',41.2);\"\nVerify.
\n\ncqlsh> SELECT count(*) FROM poc.sensor_by_day;\n\n count\n-------\n 3\n\n(1 rows)\nBefore testing the failure, confirm the replication actually landed where you think it did. nodetool getendpoints answers “which nodes hold this partition” for a specific partition key.
docker exec scylla-ha-scylla1-1 nodetool getendpoints poc sensor_by_day \"s1:2026-06-07\"\nVerify. Three addresses, one per node, because RF=3 on a three-node ring means every node is a replica:
\n\n172.21.0.2\n172.21.0.3\n172.21.0.4\nNow stop one node and read at QUORUM. QUORUM of RF=3 is 2, so two survivors are exactly enough.
\n\ndocker stop scylla-ha-scylla2-1\nVerify.
\n\ncqlsh> CONSISTENCY QUORUM;\nConsistency level set to QUORUM.\ncqlsh> SELECT count(*) FROM poc.sensor_by_day;\n\n count\n-------\n 3\n\n(1 rows)\nThat is the whole availability claim, checked. Stop a second node and the same query starts failing with an unavailable exception, which is also correct — at that point the cluster cannot prove it is returning current data, so it refuses rather than guessing.
\n\nBring the node back before continuing:
\n\ndocker start scylla-ha-scylla2-1\nnodetool snapshot flushes memtables and then hardlinks the current SSTables into a snapshots/<tag> directory under the table. It costs almost nothing at the moment you take it, because hardlinks; it starts costing disk later as compaction replaces the live files and the snapshot keeps the old ones alive.
A snapshot is per node. There is no cluster-wide snapshot command — you run it everywhere.
\n\nfor n in 1 2 3; do\n docker exec scylla-ha-scylla$n-1 nodetool snapshot -t poc_bkp poc\ndone\nVerify. The snapshot directory exists under the table directory:
\n\n$ docker exec scylla-ha-scylla1-1 sh -c 'ls -d /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp'\n/var/lib/scylla/data/poc/sensor_by_day-6f1c8ad0-6373-11f1-9c4d-3a17e2c3b901/snapshots/poc_bkp\nThe UUID in that path is the table’s identity, and it will be different on your ring. Remember that it is there. It becomes the third gotcha in Step 7.
\n\nThe loss scenario. TRUNCATE rather than a dropped table, because truncate is the mistake people actually make — it is one word away from a SELECT in a terminal that is pointed at the wrong environment.
docker exec scylla-ha-scylla1-1 cqlsh -e \"TRUNCATE poc.sensor_by_day;\"\nVerify.
\n\ncqlsh> SELECT count(*) FROM poc.sensor_by_day;\n\n count\n-------\n 0\n\n(1 rows)\nThe obvious restore is: copy the snapshot’s SSTables back next to the live ones, then tell the node to pick them up.
\n\ndocker exec scylla-ha-scylla1-1 sh -c '\n cp /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp/* \\\n /var/lib/scylla/data/poc/sensor_by_day-*/'\ndocker exec scylla-ha-scylla1-1 nodetool refresh poc sensor_by_day\nThat command exits 0. It prints nothing. And the count is still zero.
\n\nGotcha 1 — refresh only reads upload/. nodetool refresh does not scan the table directory. It scans the table’s upload/ subdirectory, loads whatever valid SSTables it finds there, and returns. Files sitting directly in the table directory are not an error and not a warning; they are simply not looked at. Create upload/ if it does not exist and put the files inside it.
Move the files into upload/ and refresh again, and the count is still zero. That is the second behavior.
Gotcha 2 — TRUNCATE leaves a marker that hides older SSTables. Truncate does not just delete files. It records a truncation record for the table, and every SSTable whose data predates that record is treated as truncated away, no matter how it arrived on disk afterwards. Loading a pre-truncate snapshot into a table that has been truncated gets you nothing. The marker belongs to the table, so the way out is DROP TABLE and recreate — which discards the marker along with the table.
So: drop, recreate, copy into upload/, refresh. And it is still zero, for a third reason.
Gotcha 3 — a recreated table gets a new UUID and a new directory. The on-disk path is <table_name>-<table_id>, and DROP + CREATE mints a fresh id. The old directory survives on disk with your snapshot inside it, which is why nothing looks broken. You are loading files into a directory that no longer belongs to any live table. The live one is the directory whose suffix matches system_schema.tables.id for that table.
Three behaviors, three silent no-ops, one restore that has to be done in a specific order to work.
\n\nThe ordering matters: stage the snapshot files out of the table directory first, because the drop in the next command takes the directory context with it.
\n\nStage on every node:
\n\nfor n in 1 2 3; do\n docker exec scylla-ha-scylla$n-1 sh -c '\n mkdir -p /tmp/stage &&\n cp /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp/* /tmp/stage/'\ndone\nDrop and recreate the table, which discards the truncation marker:
\n\ndocker exec scylla-ha-scylla1-1 cqlsh -e \"DROP TABLE poc.sensor_by_day;\"\ndocker exec scylla-ha-scylla1-1 cqlsh -e \"\nCREATE TABLE poc.sensor_by_day (\n sensor_id text, day date, ts timestamp, value double,\n PRIMARY KEY ((sensor_id, day), ts)\n) WITH CLUSTERING ORDER BY (ts DESC);\"\nConfirm which directory is now the live one. The id here is what the directory suffix has to match:
\n\ndocker exec scylla-ha-scylla1-1 cqlsh -e \"\nSELECT id FROM system_schema.tables\n WHERE keyspace_name='poc' AND table_name='sensor_by_day';\"\ndocker exec scylla-ha-scylla1-1 sh -c 'ls -d /var/lib/scylla/data/poc/sensor_by_day-*'\nYou will see at least two directories: the stale one from before the drop, and the new live one. Copy into the new one’s upload/, on every node:
for n in 1 2 3; do\n docker exec scylla-ha-scylla$n-1 sh -c '\n LIVE=$(ls -dt /var/lib/scylla/data/poc/sensor_by_day-* | head -1)\n echo \"$LIVE\"\n mkdir -p \"$LIVE/upload\"\n cp /tmp/stage/* \"$LIVE/upload/\"\n chown -R scylla:scylla \"$LIVE/upload\"'\ndone\nCheck the directory, do not trust the sort. ls -dt | head -1 picks the most recently modified directory, which is the newly created one on a clean run. If you have dropped and recreated the table more than once, or touched the stale directory in between, that heuristic will pick the wrong one. Compare the printed path against the id you just read from system_schema.tables before you trust it.
Now refresh, on every node:
\n\nfor n in 1 2 3; do\n docker exec scylla-ha-scylla$n-1 nodetool refresh poc sensor_by_day\ndone\nVerify.
\n\ncqlsh> CONSISTENCY QUORUM;\nConsistency level set to QUORUM.\ncqlsh> SELECT count(*) FROM poc.sensor_by_day;\n\n count\n-------\n 3\n\n(1 rows)\nThree rows, read at QUORUM, restored from a snapshot taken before a truncate. nodetool tablestats will confirm the loaded files are being served — on my run it reported a read count of 40 and an SSTable count of 6 for this table after the exercise.
docker exec scylla-ha-scylla1-1 nodetool tablestats poc.sensor_by_day\nTwo things worth saying about what this restore is and is not. First, snapshots are per node and this ring is RF=3, so every node had a complete copy and loading all three was straightforward; on a larger ring with RF smaller than the node count, each node’s snapshot is a different subset and the staging has to keep them apart. Second, refresh is a load, not a repair. Real disaster recovery on this engine also involves nodetool repair for anti-entropy and node rebuild to restream from replicas, and gc_grace_seconds governs when tombstones can actually be collected — get that wrong and you resurrect deleted rows.
For monitoring, ScyllaDB exposes Prometheus metrics on port 9180 — scylla_storage_proxy_* for coordinator-side latency and errors, and reactor utilization for the shard-per-core saturation signal that is specific to this engine.
docker compose -f scylla-ha/docker-compose.yml down -v\nVerify.
\n\n$ docker volume ls --filter name=scylla-ha\nDRIVER VOLUME NAME\nThree nodes’ worth of SSTables and snapshots come to real disk. down without -v keeps all of it.
| Symptom | \nCause | \nFix | \n
|---|---|---|
Node exits at startup: system does not satisfy minimum AIO requirements | \n fs.aio-max-nr in the Docker VM is below what Seastar reserves per reactor | \n docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576, then restart the node | \n
| That AIO fix worked yesterday and does not today | \nDocker Desktop restarted and rebuilt its VM, discarding the sysctl | \nRe-run the privileged sysctl container after every restart | \n
nodetool refresh exits 0 and nothing is loaded | \n SSTables are in the table directory, not in upload/ | \n Copy them into <table-dir>/upload/ and refresh again | \n
Files are in upload/, refresh is clean, rows still missing | \n A TRUNCATE marker is hiding every SSTable older than the truncate | \n DROP TABLE and recreate, then load the snapshot into the new table | \n
| Restore works on one node, count is short at QUORUM | \nrefresh was only run on some nodes | \n Run nodetool snapshot and nodetool refresh on every node | \n
Refresh finds nothing right after a DROP + CREATE | \n The recreate minted a new table UUID, so the directory you loaded is stale | \nRead system_schema.tables.id and load into the directory whose suffix matches it | \n
| Refresh logs a permission error | \nFiles were copied into upload/ as root, and the server runs as scylla | \n chown -R scylla:scylla <table-dir>/upload before refreshing | \n
A joiner sits at UJ for a long time | \n Bootstrap streaming on a one-shard, 1200 MB node is slow, not stuck | \nWait, and watch progress with nodetool netstats | \n
A joiner never appears in nodetool status | \n All three nodes were started at once, before the seed was UN | \n Start scylla1, wait for UN, then start scylla2 and scylla3 one at a time | \n
SELECT fails with an unavailable exception at QUORUM | \n Two of three nodes are down, so QUORUM of RF=3 cannot be met | \nBring a node back; do not lower the consistency level to hide it | \n
The next post in this series is PostgreSQL 18 won’t take a volume at /var/lib/postgresql/data. The rig itself, and the full eight-scenario results matrix, is in A failover lab you can run on one laptop.
\n", "summary": "Three ScyllaDB behaviors that make a perfectly valid snapshot restore return zero rows, with the repro, the fix, and the AIO limit that stops node three.", "date_published": "2026-08-07T09:00:00+05:45", "date_modified": "2026-08-11T10:20:00+05:45", "tags": ["distributed-databases"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/free-mailbox-stalwart-your-domain/", "url": "https://basantbhattarai.com.np/writing/free-mailbox-stalwart-your-domain/", "title": "A free mailbox for your domain with Stalwart", "content_html": "Want you@yourdomain.com without paying Google Workspace or Fastmail, and without running a four-gigabyte Mailcow stack? Stalwart is a single Rust binary: IMAP, SMTP, JMAP, a real admin UI, and about 100–200 MB idle. Community edition (AGPL) is enough for a personal or product domain.
You need three things the SaaS plans hide from you:
\n\nIf the same VPS already serves a website on 80/443, that is normal. Do not let Stalwart’s ACME grab 443. The wizard has a switch for this. Turn it off. Admin HTTPS goes through the nginx (or Caddy) you already run; IMAP and SMTP get a copy of the same Let’s Encrypt cert.
\n\nReplace example.com / mail.example.com with your domain below. Do not commit mailbox passwords.
What “done” means. Not the admin UI loading. Done is: you can receive a message from Gmail in IMAP, reply, and Gmail → Show original shows SPF, DKIM, and DMARC passing. First deliveries may land in spam. That is reputation, not a broken server.
\nPoint the mail hostname at the VPS and publish the records that do not depend on the wizard yet. DKIM comes later, from Stalwart’s zone file.
\n\n| Type | \nName | \nValue | \n
|---|---|---|
| A | \nmail | \n your VPS public IPv4 | \n
| MX | \n@ | \n mail.example.com priority 10 | \n
| TXT | \n@ | \n v=spf1 mx a:mail.example.com ~all | \n
| TXT | \n_dmarc | \n v=DMARC1; p=none; rua=mailto:admin@example.com | \n
Start DMARC at p=none. Tighten to p=quarantine after a few clean weeks.
At the VPS provider (not the DNS host), set reverse DNS / PTR for the public IP to mail.example.com. Large receivers still weigh PTR.
dig +short mail.example.com A\ndig +short example.com MX\ndig -x YOUR_VPS_IP +short\n# expect: mail.example.com.\nOpen the mail ports. Keep the admin port on loopback.
\n\nsudo ufw allow 25/tcp\nsudo ufw allow 465/tcp\nsudo ufw allow 587/tcp\nsudo ufw allow 993/tcp\nsudo ufw reload\nVerify. dig returns your A and MX. PTR is mail.example.com. If PTR still shows a generic srv….hstgr.cloud (or equivalent), fix that before you chase DKIM failures.
This compose file is standalone. Admin listens on 127.0.0.1:8088 only. It does not bind 80 or 443.
mkdir -p stalwart-mail/docker-data/{etc,data}\ncd stalwart-mail\ncat > .env <<'EOF'\nMAIL_HOSTNAME=mail.example.com\nMAIL_DOMAIN=example.com\nSTALWART_IMAGE_TAG=v0.16\nSTALWART_ADMIN_PORT=8088\nTZ=UTC\nEOF\nservices:\n stalwart:\n image: stalwartlabs/stalwart:${STALWART_IMAGE_TAG:-v0.16}\n container_name: stalwart\n hostname: ${MAIL_HOSTNAME:-mail.example.com}\n restart: unless-stopped\n environment:\n - TZ=${TZ:-UTC}\n - STALWART_RECOVERY_ADMIN=${STALWART_RECOVERY_ADMIN:-}\n ports:\n - \"25:25\"\n - \"465:465\"\n - \"587:587\"\n - \"143:143\"\n - \"993:993\"\n - \"4190:4190\"\n - \"127.0.0.1:${STALWART_ADMIN_PORT:-8088}:8080\"\n volumes:\n - ./docker-data/etc:/etc/stalwart\n - ./docker-data/data:/var/lib/stalwart\n mem_limit: 1g\n cpus: \"1.0\"\ndocker compose up -d\ndocker compose logs stalwart --tail=80\nOn first boot the log prints a bootstrap admin. It exists only until the wizard finishes. Open an SSH tunnel if you are not on the VPS:
\n\nssh -L 8088:127.0.0.1:8088 user@your-vps\n# then browse http://127.0.0.1:8088/admin\nWizard choices that matter:
\n\nmail.example.com, domain example.com.:443.docker compose restart stalwart.In Account Manager create at least:
\n\nyou@example.com — the inbox you will usenoreply@example.com — if an app will send mailTwo TLS places. nginx (or Caddy) can terminate https://mail.example.com for the admin UI. Phones and Thunderbird talking to 993/465 talk to Stalwart directly. If you only cert the proxy, Android and Gmail’s IMAP setup see a self-signed listener. Upload the same Let’s Encrypt fullchain + key in Stalwart → Settings → Server → TLS, and set it as the default certificate.
If you already have nginx on the host, add a vhost that proxies to the container on the Docker network (join Stalwart to that network, or proxy to 127.0.0.1:8088 from the host). Issue mail.example.com with the same certbot you use for the site. Then copy fullchain.pem / privkey.pem into Stalwart after every renew.
Verify. docker compose ps shows stalwart running, ss -lnt lists 25/465/587/993, and http://127.0.0.1:8088/admin accepts the permanent admin login.
In the admin UI: Management → Domains → example.com → ⋮ → View DNS zone file. Paste the DKIM TXT records (and any SPF/DMARC extras it suggests) at your registrar. Selectors are often named like v1-ed25519-YYYYMMDD and v1-rsa-YYYYMMDD. Copy them exactly.
Optional but useful:
\n\n| Type | \nName | \nValue | \n
|---|---|---|
| SRV | \n_imaps._tcp | \n 0 1 993 mail.example.com. | \n
| SRV | \n_submissions._tcp | \n 0 1 465 mail.example.com. | \n
| CNAME | \nautoconfig | \n mail.example.com. | \n
| CNAME | \nautodiscover | \n mail.example.com. | \n
Wait for TTL, then point a real client at the box. Thunderbird, Apple Mail, and FairEmail behave. The Gmail Android app often authenticates on SMTP and never sends MAIL FROM, so mail sits in Outbox — prefer FairEmail on Android.
| Setting | \nValue | \n
|---|---|
| IMAP | \nmail.example.com port 993 SSL/TLS | \n
| SMTP | \nmail.example.com port 465 SSL/TLS (or 587 STARTTLS) | \n
| Username | \nthe full address, you@example.com | \n
| Password | \nthe mailbox password, not the admin password | \n
| Port 25 | \ndo not use this on a phone; it is MX only | \n
Send from Gmail to you@example.com and confirm it arrives. Reply. In Gmail, open the reply → three dots → Show original. You want SPF: PASS, DKIM: PASS, DMARC: PASS.
For an app:
\n\nSMTP_HOST=mail.example.com\nSMTP_PORT=465\nSMTP_USER=noreply@example.com\nSMTP_PASSWORD=the-mailbox-password\nVerify. A message you sent from you@example.com arrives at a Gmail address, and Show original reports SPF, DKIM, and DMARC all PASS. Inbox versus spam is a reputation question; PASS on the auth trio is the server question.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
| Wizard wants 443 and the site dies | \nACME left on | \nRe-run setup with auto-TLS off; keep 80/443 on the existing proxy | \n
| IMAP/SMTP clients warn about the cert | \nCert only on nginx, not in Stalwart | \nUpload the LE chain as a Stalwart Certificate object | \n
| Nothing inbound | \nProvider blocks 25, or MX/A wrong | \nnc -vz mail.example.com 25 from outside; fix SG / MX | \n
| Outbound stuck / greylisted | \nNo PTR, or SPF does not name this host | \nSet PTR to mail.example.com; dig -x must match | \n
| DKIM fail | \nZone file not published, or wrong selector | \nRe-copy the TXT from the domain’s DNS zone view | \n
| Gmail app Outbox never sends | \nGmail IMAP SMTP quirk | \nUse FairEmail or Thunderbird; 465 + full username | \n
| Bootstrap password gone | \nWizard already completed | \nUse the permanent admin from the last wizard screen | \n
Back up both bind mounts. They are the mailboxes and the config.
\n\ntar czf stalwart-backup-$(date +%F).tgz -C stalwart-mail docker-data\nPin v0.16 (or a newer minor after reading the upgrade notes). After certbot renews mail.example.com, copy the new fullchain and key into Stalwart again. Move DMARC from p=none to p=quarantine once Show original has been clean for a while.
To leave: docker compose down, revert MX to your previous host, and keep the tarball if you might restore.
You now have three free OSS services that usually cost a subscription: remote desktop, VPN, and mail for your own domain. They fit on one small VPS if you keep 80/443 for the website, 51821 and 8088 off the public internet, and treat DNS plus one backup of keys/mail data as the real state.
\n", "summary": "Want a free mailbox for your domain? Run Stalwart on Docker, skip ACME when 443 is taken, publish SPF/DKIM/DMARC, and prove a Gmail round-trip.", "date_published": "2026-08-04T09:00:00+05:45", "date_modified": "2026-08-11T11:30:00+05:45", "tags": ["self-hosted"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/failover-lab-six-engines-eight-scenarios/", "url": "https://basantbhattarai.com.np/writing/failover-lab-six-engines-eight-scenarios/", "title": "A failover lab you can run on one laptop", "content_html": "Every high-availability document ends with the same sentence in a different typeface: the cluster tolerates the loss of a node. It almost never says which node, how long the gap is, whether the write you just acknowledged is still there afterwards, or what the survivors do about writes while an election runs.
\n\nSo I wrote a test plan with eight scenarios and ran all of them on one machine, one stack at a time, inside a Docker daemon with 8 CPU and 12.5 GB of RAM. Every cluster was brought up, exercised, and torn down before the next one started, because 12.5 GB does not hold two of these at once.
\n\n8 of 8 scenarios passed — in my own test plan. That qualifier is the honest one. There is no external conformance suite behind this. I decided what “passed” meant for each engine, wrote it down, and then went and checked it. What follows is the rig, the plan, the numbers it produced, and one scenario run end to end so you can reproduce the method on the other seven.
\n\nThe compose files are reproduced inline in this post and in the rest of the series. There is no public repository to clone.
\n\nSix engines, eight scenarios. Redis appears twice because it ships two entirely different failure models, and the last row is the observability stack that watches the others.
\n\nw: \"majority\" earns its keep.Every stack in this series is deliberately shaped to fit an 8 CPU / 12.5 GB daemon with exactly one stack running. ScyllaDB nodes are pinned to one shard and 1200 MB each; the Galera nodes and the six Redis nodes are similarly small. If Docker has less than this, containers will start and then be OOM-killed halfway through a bootstrap, which looks exactly like a cluster bug and is not one.
\n\nOn Docker Desktop this lives in Settings → Resources. Set CPUs to 8 and memory to 12.5 GB, then apply and restart.
\n\nVerify.
\n\n$ docker info | grep -E \"^ CPUs|^ Total Memory\"\n CPUs: 8\n Total Memory: 12.5GiB\nIf Total Memory reads lower than that, the daemon did not pick up the change — apply and restart it again before continuing.
ScyllaDB’s Seastar runtime allocates asynchronous I/O contexts per reactor at startup. The Linux VM behind Docker Desktop ships with a conservative fs.aio-max-nr, and it is a kernel-wide limit, so the first two Scylla nodes started fine and the third one died with:
system does not satisfy minimum AIO requirements\nThe limit belongs to the VM, not to any one container, so you raise it from a privileged throwaway container.
\n\ndocker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576\nVerify.
\n\n$ docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576\nfs.aio-max-nr = 1048576\nGotcha. This setting lives inside the Docker VM, not on your machine. Restarting Docker Desktop resets it, and the symptom comes back looking like a fresh problem. If a Scylla node refuses to start on a day when it worked yesterday, re-run the privileged container before you debug anything else.
\nThree mongod processes, one replica set, no authentication. Auth is skipped deliberately: the keyfile and RBAC setup is a separate exercise, and mixing it into a failover test means every failure has two possible causes.
name: mongo-ha\n\nservices:\n mongo1:\n image: mongo:8.0\n command: [\"mongod\", \"--replSet\", \"rs0\", \"--bind_ip_all\", \"--port\", \"27017\"]\n ports: [\"27021:27017\"]\n volumes: [\"m1:/data/db\"]\n mongo2:\n image: mongo:8.0\n command: [\"mongod\", \"--replSet\", \"rs0\", \"--bind_ip_all\", \"--port\", \"27017\"]\n ports: [\"27022:27017\"]\n volumes: [\"m2:/data/db\"]\n mongo3:\n image: mongo:8.0\n command: [\"mongod\", \"--replSet\", \"rs0\", \"--bind_ip_all\", \"--port\", \"27017\"]\n ports: [\"27023:27017\"]\n volumes: [\"m3:/data/db\"]\n\nvolumes:\n m1:\n m2:\n m3:\ndocker compose -f mongo-ha/docker-compose.yml up -d\nThen initiate the set. mongo1 gets priority 2 so that the set has a preferred primary — that is what makes the failback at the end of Step 5 observable rather than random.
docker exec mongo-ha-mongo1-1 mongosh --quiet --eval '\nrs.initiate({\n _id: \"rs0\",\n members: [\n { _id: 0, host: \"mongo1:27017\", priority: 2 },\n { _id: 1, host: \"mongo2:27017\", priority: 1 },\n { _id: 2, host: \"mongo3:27017\", priority: 1 }\n ]\n})'\nI use a three-line status helper throughout, because rs.status() in full is several hundred lines and the three fields that matter are name, state and health.
docker exec mongo-ha-mongo1-1 mongosh --quiet --eval '\nconst s = rs.status();\nprint(\"set: \" + s.set);\ns.members.forEach(m => print(\" \" + m.name + \" \" + m.stateStr + \" health=\" + m.health));'\nVerify. Within about ten seconds of rs.initiate():
set: rs0\n mongo1:27017 PRIMARY health=1\n mongo2:27017 SECONDARY health=1\n mongo3:27017 SECONDARY health=1\nIf all three read SECONDARY, the election has not finished — wait and run it again. If one reads STARTUP, that member never got the config; check that the hostnames in rs.initiate() match the compose service names.
This first pass uses a controlled loss, not a crash: docker stop sends SIGTERM and lets mongod shut down cleanly. That isolates election behavior from recovery after an unclean process exit. A separate crash test would use docker kill --signal=KILL and should be recorded as a different scenario. The distinction matters enormously for Galera; for this MongoDB election, both paths should leave the same two-member majority able to elect.
docker stop mongo-ha-mongo1-1\nVerify. Re-run the status helper against a surviving member:
\n\nset: rs0\n mongo1:27017 (not reachable/healthy) health=0\n mongo2:27017 PRIMARY health=1\n mongo3:27017 SECONDARY health=1\nmongo2 and mongo3 are two of three members, which is a majority, so they can hold an election and elect one of themselves. If you had killed two of three, you would see all survivors sitting at SECONDARY forever, refusing writes — that is the correct behavior, not a hang.
Gotcha. The election is not instantaneous, and during it there is no primary. A client that treats a connection error as fatal will surface a failure to the user even though the set recovers on its own moments later. This is why retryable writes and a full replica-set connection string exist. Testing failover against a single-host URI tests nothing.
\nA failover that loses acknowledged writes is not a failover, it is data loss with good manners. w: \"majority\" is the setting that makes the promise: the write is acknowledged only once a majority of members have it, so any node that can win a subsequent election already has it.
docker exec mongo-ha-mongo2-1 mongosh --quiet --eval '\ndb.getSiblingDB(\"poc\").orders.insertOne(\n { n: 1 },\n { writeConcern: { w: \"majority\", j: true } }\n);\nprint(\"write after failover: OK\");'\nVerify.
\n\nwrite after failover: OK\nOn the full run I did the same thing at volume — 1000 documents written with w: \"majority\", j: true, then read back from a secondary, which returned 1000 documents replicated. The single insert above is the fast version of the same check.
Now bring the old primary back:
\n\ndocker start mongo-ha-mongo1-1\nVerify. It rejoins as a secondary, catches up from the oplog, and then reclaims the primary role because its priority is 2:
\n\nset: rs0\n mongo1:27017 PRIMARY health=1\n mongo2:27017 SECONDARY health=1\n mongo3:27017 SECONDARY health=1\nThat failback is a second election, with a second window where there is no primary. Priorities are useful for keeping a preferred node in front of traffic; they are not free.
\n\nThis is the step that keeps the rig usable. down without -v leaves the volumes behind, and nine stacks of orphaned volumes is a lot of disk.
docker compose -f mongo-ha/docker-compose.yml down -v\nVerify.
\n\n$ docker volume ls --filter name=mongo-ha\nDRIVER VOLUME NAME\nAn empty listing under the header is what you want. To clear the whole series in one go:
\n\nfor d in mongo-ha mongo-shard scylla-ha solr-ha galera-ha redis-cluster \\\n redis-sentinel pg-replication observability; do\n docker compose -f $d/docker-compose.yml down -v 2>/dev/null\ndone\n| Engine | \nHA / cluster | \nSharding | \nBackup / restore | \nMonitoring | \nResult | \n
|---|---|---|---|---|---|
| MongoDB 8.0 | \n3-node replica set, auto-failover | \nhashed-key sharded cluster (2 shards) | \nmongodump / mongorestore | \n serverStatus, repl lag, oplog | \nPASS | \n
| ScyllaDB 2026.1 | \n3-node, RF=3, QUORUM survives node loss | \ntoken ring, 256 vnodes/node | \nnodetool snapshot + refresh | \n nodetool info / tablestats, :9180 | \n PASS | \n
| SolrCloud 9 | \nZK + 2 nodes, 2×2 shards/replicas | \nnumShards=2 | \n collection BACKUP / RESTORE | \nmetrics API, solr-exporter | \nPASS | \n
| MariaDB 11.8 + Galera 4 | \n3-node multi-master, quorum, IST rejoin | \nn/a (full replicas) | \nmariabackup SST | \nwsrep_* status vars | \n PASS | \n
| Redis 8 Cluster | \n3 primaries + 3 replicas, auto-failover | \n16,384 hash slots, hash tags | \nAOF per node | \nCLUSTER INFO / SHARDS | \n PASS | \n
| Redis 8 Sentinel | \n1 primary + 2 replicas + 3 sentinels | \nn/a | \nAOF / RDB | \nSENTINEL master / replicas | \n PASS | \n
| PostgreSQL 18 | \nprimary + hot standby, pg_promote() failover | \n n/a | \npg_basebackup + WAL / PITR | \n pg_stat_replication, lag | \n PASS | \n
| Observability | \nexporters → Prometheus → Grafana | \nn/a | \nn/a | \ngolden-signals dashboard | \nPASS | \n
Four columns per engine, because “does it fail over” is only a quarter of the question. A cluster that survives a node loss but cannot be restored from a backup is not highly available, it is briefly lucky.
\n\nThese are the measurements, not the marketing. All of them came off this rig.
\n\nPostgreSQL 18 streaming replication reported write_lag ≈ 104 µs and replay_lag ≈ 383 µs in pg_stat_replication, with replica lag of 0 bytes and the slot showing active=t, wal_status=reserved. pg_promote() returned t, pg_is_in_recovery() flipped to f, and the timeline went from 1 to 2. Sub-millisecond replay on a single host is not a WAN number, but it establishes that the lag you see in production is network and disk, not the protocol.
Redis Sentinel promoted a replica in ~5 s with quorum 2 and down-after-milliseconds 5000, and demoted the old primary to a replica of the new one when it came back. Redis Cluster promoted a replica after I killed a master owning slots 5461–10922, bumped the config epoch from 2 to 7, returned to cluster_state:ok with cluster_slots_ok:16384, and lost nothing.
MongoDB sharding on a hashed userId split 100,000 documents 49,607 / 50,393 across two shards. A query carrying the shard key touched 1 shard; a filter on a non-key field scatter-gathered to 2.
Galera took 150 concurrent updates of a single row from two masters at once. Node 2 logged 3 certification failures, node 3 logged 5 certification failures and 2 brute-force aborts, and the counter still converged to 300 on all three nodes. Optimistic concurrency caught every conflict at commit time and the arithmetic came out right.
\n\nSolrCloud went from GREEN to ORANGE when I stopped a node — 4 active replicas, 2 down — and kept answering *:* with numFound 6 the whole time, then recovered to GREEN on its own when the node came back.
ScyllaDB returned 3 rows at CONSISTENCY QUORUM with one of three nodes stopped, exactly as RF=3 promises.
The observability stack scraped a Postgres exporter and a Redis exporter into Prometheus with all targets up, and reported rate(redis_commands_processed_total[1m]) of 27.7 and a 500/500 hit/miss split — a 50% cache hit ratio — through to an auto-provisioned Grafana dashboard.
Two of the eight are written up in full, and both are the write-ups I went looking for and could not find while I was stuck:
\n\nQueued from the same lab notes: the Galera wsrep_provider_options read-only surprise and the graceful-versus-ungraceful quorum behavior; Redis Cluster against Sentinel with the timings side by side; SolrCloud’s ORANGE state and the solr.allowPaths backup failure; and the MongoDB shard-key explain plan. They will appear under /writing/ as they are finished.
| Symptom | \nCause | \nFix | \n
|---|---|---|
| Containers start, then get OOM-killed mid-bootstrap | \nTwo stacks running at once, or the daemon has less than 12.5 GB | \nTear the previous stack down with docker compose down -v first; run exactly one stack at a time | \n
A ScyllaDB node exits with system does not satisfy minimum AIO requirements | \n fs.aio-max-nr in the Docker VM is too low for the number of reactors | \n docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576, then restart the node | \n
| That AIO fix stops working after a restart | \nThe sysctl lives in the Docker VM, which is recreated on restart | \nRe-run the privileged sysctl container after every Docker Desktop restart | \n
rs.initiate() fails with “already initialized” | \n A volume survived an earlier run | \ndocker compose -f mongo-ha/docker-compose.yml down -v, then bring it up again | \n
All members sit at SECONDARY and writes are refused | \n No majority is reachable — two of three members are down | \nRestart a member; a three-node set needs two to elect | \n
not primary errors on a write after a failover | \n The client is pinned to one host instead of the set | \nUse a full replica-set connection string and enable retryable writes | \n
| Disk keeps growing between labs | \ndocker compose down without -v keeps named volumes | \n Always down -v; then docker volume prune | \n
| A joiner never leaves the joining state | \nEvery node was started at once, before the seed was ready | \nStart the seed, wait until it reports healthy, then start the joiners | \n
Want a VPN without paying Mullvad, Tailscale, or a random “free VPN” app that sells your traffic? WireGuard is the modern OSS default: small, fast, audited crypto. wg-easy puts a web UI on top so you add a phone by scanning a QR instead of editing [Peer] blocks.
A $5 VPS is enough. Idle RAM is about 50 MB. You will spend more time on DNS and the firewall than on WireGuard itself.
\n\nOne trap, stated early so you do not copy the wrong blog post:
\n\nimage: ghcr.io/wg-easy/wg-easy:latest\nenvironment:\n - WG_HOST=vpn.example.com\n - PASSWORD_HASH=...\n - WG_ALLOWED_IPS=0.0.0.0/0\n:latest still points at v14. v15 (May 2025) dropped those env vars. Config now lives in a first-run wizard (or INIT_* for unattended setup). Only PORT, HOST, and INSECURE remain as ordinary runtime env. Follow a v14 file on v15 and the hostname never sticks. Follow a v15 file on :latest and the wizard never appears. Pin :15.
This setup is a full-tunnel exit node: your laptop and phone leave the internet from the VPS IP. Great on café Wi-Fi. It also means the VPS acceptable-use policy now applies to your browsing. Split tunnel is one Allowed-IPs change if you do not want that.
\n\nPoint vpn.example.com at the VPS. Open UDP 51820 for the tunnel. Open TCP 51821 only from your current public IP. Do not put this UI on 80 or 443 if those already serve a website.
sudo ufw allow 51820/udp\nsudo ufw allow from YOUR_PUBLIC_IP to any port 51821 proto tcp\nsudo ufw reload\n\nsudo sysctl -w net.ipv4.ip_forward=1\necho 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-wireguard.conf\nsudo sysctl --system\nmkdir -p wireguard-vpn/config\ncd wireguard-vpn\nservices:\n wg-easy:\n image: ghcr.io/wg-easy/wg-easy:15\n container_name: wg-easy\n restart: unless-stopped\n environment:\n - INSECURE=true\n - PORT=51821\n - HOST=0.0.0.0\n volumes:\n - ./config:/etc/wireguard\n - /lib/modules:/lib/modules:ro\n ports:\n - \"51820:51820/udp\"\n - \"51821:51821/tcp\"\n cap_add:\n - NET_ADMIN\n - SYS_MODULE\n sysctls:\n - net.ipv4.ip_forward=1\n - net.ipv4.conf.all.src_valid_mark=1\n - net.ipv6.conf.all.disable_ipv6=0\n - net.ipv6.conf.all.forwarding=1\n mem_limit: 256m\n cpus: \"0.50\"\nThree lines are load-bearing.
\n\nimage: ...:15. Not :latest. Confirm after pull:
docker compose pull\ndocker compose config | grep image\n# image: ghcr.io/wg-easy/wg-easy:15\n/lib/modules:/lib/modules:ro. Required in v15 so the container can load the WireGuard kernel module. Omit it and the interface never comes up in a way that looks like a permissions problem.
INSECURE=true plus port 51821. That combination is HTTP on a high port, which is acceptable only because UFW has already limited 51821 to you. For TLS later, add a vpn.example.com vhost to the existing reverse proxy and set INSECURE=false. Do not start a second process on 80/443.
docker compose up -d\ndocker compose logs --tail=40 wg-easy\ndocker compose exec wg-easy wg show\nVerify. docker compose ps shows wg-easy up, wg show lists wg0 listening on 51820, and nc -vzu vpn.example.com 51820 from your laptop reports that UDP is reachable.
Open http://vpn.example.com:51821 from the IP you allowed. Complete the wizard:
| Setting | \nValue | \n
|---|---|
| Admin username | \nyours | \n
| Admin password | \n16+ characters from a password manager | \n
| Host / Endpoint | \nvpn.example.com | \n
| Port | \n51820 | \n
| DNS | \n1.1.1.1 or 1.1.1.1,8.8.8.8 | \n
| Allowed IPs | \n0.0.0.0/0, ::/0 | \n
0.0.0.0/0, ::/0 is the full tunnel. For split tunnel use only the VPN subnet, usually 10.8.0.0/24. Enable TOTP on the admin account before you add clients.
Then New client → name the device → download the .conf or show the QR. One client per device. Reusing a single config on a laptop and a phone will fight over the same internal IP.
Unattended alternative, if you refuse the wizard: uncomment an INIT_* block (INIT_ENABLED, INIT_USERNAME, INIT_PASSWORD, INIT_HOST, INIT_PORT, INIT_DNS, INIT_ALLOWED_IPS). That is the only remaining place those values belong. They are not WG_* anymore.
Do not copy a password into git. If you use INIT_PASSWORD, keep it in a gitignored .env. The compose file above uses the wizard so the repo never holds a credential.
Verify. The UI lists the new client, and the downloaded config contains Endpoint = vpn.example.com:51820 plus AllowedIPs = 0.0.0.0/0, ::/0.
Before connecting:
\n\ncurl -s https://api.ipify.org && echo\n# your ISP address\nImport the tunnel:
\n\n.conf → enable “Activate on system startup” if you want it after reboot.sudo cp client.conf /etc/wireguard/wg0.conf && sudo systemctl enable --now wg-quick@wg0After connecting:
\n\ncurl -s https://api.ipify.org && echo\n# must now be the VPS public IPv4\nnslookup example.com\nIf the second curl still shows the ISP address, you are on split tunnel or the client did not activate. If DNS fails while the IP changed, the wizard DNS field is wrong.
docker compose exec wg-easy wg show\nYou should see a recent handshake and transfer counters moving.
\n\nVerify. curl https://api.ipify.org after connect returns the VPS IP, DNS still resolves, and wg show on the server shows a handshake in the last two minutes.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
| No wizard, old password env | \nImage is v14 (:latest) | \n Pin :15, recreate the container | \n
Wizard present, WG_HOST ignored | \n You are on v15 with a v14 file | \nDelete the old env; set hostname in the wizard | \n
wg show empty / module errors | \n Missing /lib/modules mount | \n Add the read-only mount; confirm kernel WireGuard | \n
| UDP 51820 closed from outside | \nUFW or provider SG | \nufw allow 51820/udp plus the cloud rule | \n
| UI open to the world | \n51821 allowed from 0.0.0.0/0 | \n Restrict to your IP; enable TOTP | \n
| IP does not change | \nAllowed IPs is the VPN subnet only | \nSet 0.0.0.0/0, ::/0 and re-download the client | \n
| Banks / streaming break | \nFull-tunnel egress looks like a VPS | \nSplit-tunnel those destinations, or accept the trade | \n
| Every device dies after a wipe | \n./config was deleted | \n Restore wg0.json / the bind mount from backup | \n
State lives in ./config. Losing it means re-issuing every peer.
tar czf wg-backup-$(date +%F).tgz -C wireguard-vpn/config .\ndocker compose pull # still pinned to :15\ndocker compose up -d\nTo leave: docker compose down, delete the tunnel on each device. Wiping server state is rm -rf wireguard-vpn/config.
If curl shows the VPS IP, you have a free VPN you control. Add a phone, a tablet, a second laptop — one client each in the UI, same server.
Previous: a free remote desktop with RustDesk. Next: a free mailbox for your domain with Stalwart.
\n", "summary": "Want a free VPN you actually control? Pin wg-easy to :15, mount /lib/modules, keep the UI off 80/443, and prove traffic exits from your VPS.", "date_published": "2026-06-03T09:00:00+05:45", "date_modified": "2026-08-11T11:30:00+05:45", "tags": ["self-hosted"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] }, { "id": "https://basantbhattarai.com.np/writing/self-host-rustdesk-relay-udp-21116/", "url": "https://basantbhattarai.com.np/writing/self-host-rustdesk-relay-udp-21116/", "title": "A free private remote desktop with RustDesk", "content_html": "Want a TeamViewer-style remote desktop without a subscription, and without handing connection metadata to someone else’s relay? RustDesk is fully open source. The clients are free. The public servers will connect you — and then drop the session at about thirty seconds unless you pay.
\n\nYou can skip that tax. Run the official hbbs (rendezvous) and hbbr (relay) containers on any small Linux VPS, point relay.example.com at it, and paste one public key into each client. Idle cost is tens of megabytes of RAM. Bandwidth is whatever your VPS already has.
Two things usually waste the first hour, so we will treat them as part of the setup rather than surprises:
\n\nhbbr. You have not tested the relay until ALWAYS_USE_RELAY=Y and you see traffic on 21117.Substitute your domain below. Do not publish the private key.
\n\nWhat “done” means. Not docker compose ps saying healthy. Done is: UDP 21116 answers from outside, the client shows Ready (relay.example.com), and a session through your relay lasts past sixty seconds.
Create a directory and an A record for the relay hostname, pointing at the VPS. IPv6 is optional; if you add AAAA, keep using the hostname as the client setting so you are not locked to one stack.
\n\nmkdir -p rustdesk-server/data\ncd rustdesk-server\nOpen the ports. UDP 21116 is the one people skip. 21118/21119 are only for the web client; leave them closed if you are not using it.
\n\nsudo ufw allow 21115/tcp\nsudo ufw allow 21116/tcp\nsudo ufw allow 21116/udp\nsudo ufw allow 21117/tcp\nsudo ufw reload\nsudo ss -lntu | grep -E '21115|21116|21117' || true\nWrite .env and the compose file. RUSTDESK_PUBLIC_HOST must be the name you put in DNS. hbbs -r tells clients which host:port is the relay.
cat > .env <<'EOF'\nRUSTDESK_PUBLIC_HOST=relay.example.com\nRUSTDESK_ALWAYS_USE_RELAY=Y\nEOF\nservices:\n hbbs:\n image: rustdesk/rustdesk-server:latest\n container_name: rustdesk_hbbs\n restart: unless-stopped\n environment:\n - ALWAYS_USE_RELAY=${RUSTDESK_ALWAYS_USE_RELAY:-Y}\n command: hbbs -r ${RUSTDESK_PUBLIC_HOST}:21117\n ports:\n - \"21115:21115\"\n - \"21116:21116\"\n - \"21116:21116/udp\"\n volumes:\n - ./data:/root\n depends_on:\n - hbbr\n networks: [rustdesk_net]\n mem_limit: 256m\n cpus: \"0.25\"\n\n hbbr:\n image: rustdesk/rustdesk-server:latest\n container_name: rustdesk_hbbr\n restart: unless-stopped\n command: hbbr\n ports:\n - \"21117:21117\"\n volumes:\n - ./data:/root\n networks: [rustdesk_net]\n mem_limit: 256m\n cpus: \"0.50\"\n\nnetworks:\n rustdesk_net:\n driver: bridge\nExplicit port maps beat host networking here. The relay then sits beside an existing app stack without claiming the host network namespace. Idle cost is small: tens of megabytes of RAM and almost no CPU until a session is relayed.
\n\ndocker compose up -d\ndocker compose ps\ndocker compose logs hbbs --tail=30\ndocker compose logs hbbr --tail=20\nLook for ALWAYS_USE_RELAY=Y in the hbbs log and a listen line on 21117 in hbbr. Treat a healthcheck of unhealthy as untrusted if the image has no shell for CMD-SHELL. Ports listening plus an external nc are the real signals.
Verify. From a machine that is not the VPS, all four of these should report open. If TCP works and UDP fails, the firewall or provider security group is incomplete.
\n\nnc -vz relay.example.com 21115\nnc -vz relay.example.com 21116\nnc -vz relay.example.com 21117\nnc -vzu relay.example.com 21116\nThe first start writes an Ed25519 pair into ./data. Clients pin the public key. Without that pin they will not talk to your server, which is the whole anti-MITM design.
cat ./data/id_ed25519.pub\n# example only: 5fGu2T8aQrXkPnMwZ...kP9vNw=\nTreat ./data/id_ed25519 as a secret. Back it up off the VPS. If that file is lost, the next start mints a new key and every existing client refuses to connect until you paste the new public key.
On each RustDesk client:
\n\nbrew install --cask rustdesk on a Mac).relay.example.com.hbbs -r advertises the relay; the official docs discourage setting this on every client.id_ed25519.pub, no quotes, no trailing space.The status line should read Ready (relay.example.com), not the public-server message.
Relay is server-side. ALWAYS_USE_RELAY=Y is an hbbs environment variable. Putting a relay hostname in the client does not force relay. Two clients on the same LAN may still take a direct path on some RustDesk builds; test across two networks if you need to prove hbbr.
On the machine you want to control unattended: Settings → Security → enable a permanent password (16+ random characters, password manager), optionally 2FA, and grant the OS permissions RustDesk asks for (Accessibility and Screen Recording on macOS). That first permission grant needs a local screen. It cannot be done over SSH.
\n\nVerify. The controlled client shows Ready (relay.example.com) and docker compose logs hbbs --tail=50 shows the client ID registering, with no permission denied or key errors.
From the controlling laptop, enter the nine-digit ID of the remote machine, use the permanent password, and wait for the desktop. Then do the only check that matters for this post: leave the session up for more than sixty seconds while moving the mouse.
\n\nWhile it is connected:
\n\ndocker compose logs -f hbbr\nsudo ss -tunap | grep 21117\nWith ALWAYS_USE_RELAY=Y you should see hbbr activity and established sockets on 21117. If the session is up and hbbr is silent, you are on P2P and have not tested the relay.
That is also why I default to relay-on. It costs server bandwidth. It makes the failure mode visible. Set RUSTDESK_ALWAYS_USE_RELAY=N later if you want hole-punched P2P and only fall back to hbbr.
Verify. The session is still connected at 60 seconds, hbbr logged traffic, and ss shows ESTABLISHED on 21117. That is the proof you are not on the public 30-second relay.
| Symptom | \nCause | \nRepair | \n
|---|---|---|
| Client stays on the public server message | \nKey has spaces/quotes, or ID Server is wrong | \nPaste id_ed25519.pub exactly; restart the app | \n
TCP nc works, UDP 21116 fails | \n UFW or cloud SG missing UDP | \nufw allow 21116/udp and the matching provider rule | \n
Session starts, hbbr stays quiet | \n P2P / same LAN, or ALWAYS_USE_RELAY not Y | \n Confirm the env in hbbs logs; test from a different network | \n
| Containers “unhealthy” but ports listen | \nImage healthcheck needs a shell the image may lack | \nIgnore the health label; trust ss and nc | \n
| Every client dies after a recreate | \n./data/id_ed25519 was deleted | \n Restore the backup, or distribute the new public key | \n
| IPv4 hangs after you add AAAA | \nClient tries A first and never falls back | \nKeep the hostname in the client; do not switch RUSTDESK_PUBLIC_HOST to a raw address unless you want one stack | \n
Quarterly:
\n\ndocker compose pull\ndocker compose up -d\nBackup recipe — the keys are under a kilobyte:
\n\ntar czf rustdesk-keys-$(date +%F).tgz -C rustdesk-server data/id_ed25519 data/id_ed25519.pub\nTo walk away from the private server, docker compose down and clear ID Server / Key on each client. They return to the public infrastructure in about thirty seconds per device.
If you can keep a session up for a minute through hbbr, you have a free private remote desktop. Add more devices the same way: same hostname, same public key, no server change.
Next, the other thing people usually want on the same VPS: a free VPN with WireGuard.
\n", "summary": "Want a free private remote desktop? Self-host RustDesk on Docker, open UDP 21116, pin the key, and keep sessions past the public cutoff.", "date_published": "2026-06-02T09:00:00+05:45", "date_modified": "2026-08-11T11:30:00+05:45", "tags": ["self-hosted"], "authors": [ { "name": "Basant Bhattarai", "url": "https://basantbhattarai.com.np" } ] } ] }