nodetool refresh only reads the live table's upload/ directory

What went wrong

I had a good snapshot. I had the SSTable files on disk. I ran nodetool refresh, got no error, and SELECT count(*) returned zero.

That is what makes a restore on this engine expensive to debug: the obvious procedure succeeds at every step, and the data is still not there. Three behaviors decide whether the rows come back, stated up front so you can stop reading if one of them is your problem:

1. nodetool refresh only loads SSTables from the table’s upload/ subdirectory. Files placed in the table directory itself are ignored, silently, with a zero exit code.
2. The upload/ has to belong to the live table. A dropped and recreated table gets a new UUID and a new directory, and the old one survives on disk. Loading into the old one also exits 0 and loads nothing.
3. The files must be owned by the scylla user. This one is loud: refresh exits 4 and says so.

There is a fourth thing, which happens before any of this and is the reason the cluster would not start in the first place. It gets Step 1.

The compose file is reproduced inline below. It is also in the public db-failover-lab repository.

Step 1 — Raise fs.aio-max-nr before the third node

The first two Scylla nodes came up fine. The third exited during startup with:

system does not satisfy minimum AIO requirements

ScyllaDB’s Seastar runtime reserves asynchronous I/O contexts per reactor when it starts. fs.aio-max-nr is a kernel-wide ceiling on those contexts, and the Linux VM behind Docker Desktop ships with a low default. Two nodes fit under it. Three did not.

The limit belongs to the VM, not to a container, so raise it from a privileged throwaway container. It applies to every container on that daemon, including ones you start later.

docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576

Verify.

$ docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576
fs.aio-max-nr = 1048576

Step 2 — Bring the ring up one node at a time

Three nodes, each pinned to a single shard and 1200 MB, which is what makes three of them fit on one Docker host. scylla1 is the seed; the other two join it.

name: scylla-ha

services:
  scylla1:
    image: scylladb/scylla:2026.1
    command: ["--seeds=scylla1", "--smp=1", "--memory=1200M", "--overprovisioned=1", "--api-address=0.0.0.0"]
    ports: ["9042:9042"]
    volumes: ["s1:/var/lib/scylla"]

  scylla2:
    image: scylladb/scylla:2026.1
    command: ["--seeds=scylla1", "--smp=1", "--memory=1200M", "--overprovisioned=1", "--api-address=0.0.0.0"]
    volumes: ["s2:/var/lib/scylla"]
    depends_on: [scylla1]

  scylla3:
    image: scylladb/scylla:2026.1
    command: ["--seeds=scylla1", "--smp=1", "--memory=1200M", "--overprovisioned=1", "--api-address=0.0.0.0"]
    volumes: ["s3:/var/lib/scylla"]
    depends_on: [scylla1]

volumes:
  s1:
  s2:
  s3:

depends_on only controls start order, not readiness, so bring the nodes up by hand. Start the seed, wait for it to report Up/Normal, then add one joiner at a time.

docker compose -f scylla-ha/docker-compose.yml up -d scylla1

docker exec scylla-ha-scylla1-1 nodetool status

Once scylla1 shows UN, add the second, wait again, then the third:

docker compose -f scylla-ha/docker-compose.yml up -d scylla2

docker compose -f scylla-ha/docker-compose.yml up -d scylla3

Verify. nodetool status lists three nodes at UN — Up and Normal — each holding 256 tokens:

UN 172.21.0.2 ... rack1
UN 172.21.0.3 ... rack1
UN 172.21.0.4 ... rack1

A node stuck at UJ is still joining and streaming; give it time rather than restarting it. A node at DN never made it — check that its AIO problem is fixed and that the seed was UN before it started.

Step 3 — Create the keyspace and a query-driven table

RF=3 on a three-node cluster means every node holds every row, which is what makes the QUORUM behavior in the next step easy to reason about.

CREATE KEYSPACE poc WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'datacenter1': 3
};

CREATE TABLE poc.sensor_by_day (
  sensor_id text,
  day       date,
  ts        timestamp,
  value     double,
  PRIMARY KEY ((sensor_id, day), ts)
) WITH CLUSTERING ORDER BY (ts DESC);

The partition key is (sensor_id, day), not sensor_id alone. Bucketing a time series by day bounds the partition: a sensor that reports for two years does not accumulate one enormous partition that has to be read, repaired and compacted as a unit. This is the single most common modeling mistake in Cassandra-family stores, and it does not show up until the partition is already too big to fix comfortably.

Three rows are enough for the whole exercise:

docker exec scylla-ha-scylla1-1 cqlsh -e "
INSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:00:00',42.0);
INSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:01:00',43.5);
INSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:02:00',41.2);"

Verify.

cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     3

(1 rows)

Step 4 — Confirm RF=3, then lose a node

Before testing the failure, confirm the replication actually landed where you think it did. nodetool getendpoints answers “which nodes hold this partition” for a specific partition key.

docker exec scylla-ha-scylla1-1 nodetool getendpoints poc sensor_by_day "s1:2026-06-07"

Verify. Three addresses, one per node, because RF=3 on a three-node ring means every node is a replica:

172.21.0.2
172.21.0.3
172.21.0.4

Now stop one node and read at QUORUM. QUORUM of RF=3 is 2, so two survivors are exactly enough.

docker stop scylla-ha-scylla2-1

Verify.

cqlsh> CONSISTENCY QUORUM;
Consistency level set to QUORUM.
cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     3

(1 rows)

That is the whole availability claim, checked. Stop a second node and the same query starts failing with an unavailable exception, which is also correct — at that point the cluster cannot prove it is returning current data, so it refuses rather than guessing.

Bring the node back before continuing:

docker start scylla-ha-scylla2-1

Step 5 — Take a snapshot on every node

nodetool snapshot flushes memtables and then hardlinks the current SSTables into a snapshots/<tag> directory under the table. It costs almost nothing at the moment you take it, because hardlinks; it starts costing disk later as compaction replaces the live files and the snapshot keeps the old ones alive.

A snapshot is per node. There is no cluster-wide snapshot command — you run it everywhere.

for n in 1 2 3; do
  docker exec scylla-ha-scylla$n-1 nodetool snapshot -t poc_bkp poc
done

Verify. The snapshot directory exists under the table directory, and it holds at least one *-Data.db file:

$ docker exec scylla-ha-scylla1-1 sh -c 'ls -d /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp; ls /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp | grep -c Data.db'
/var/lib/scylla/data/poc/sensor_by_day-b573cef0c7c011f1b6ea6fae48e15e5e/snapshots/poc_bkp
1

The UUID in that path is the table’s identity, and it will be different on your ring. Remember that it is there; it matters in Step 7.

Check every node, not just the first. On the re-run, scylla2 had been stopped within seconds of the inserts in Step 4, came back without them, and its snapshot held no data file at all. With the default periodic commitlog sync, a node stopped right after a write can lose it locally; the other replicas still have it. A snapshot is only as good as what that node had flushed.

Step 6 — Destroy the data

The loss scenario. TRUNCATE rather than a dropped table, because truncate is the mistake people actually make — it is one word away from a SELECT in a terminal that is pointed at the wrong environment.

docker exec scylla-ha-scylla1-1 cqlsh -e "TRUNCATE poc.sensor_by_day;"

Verify.

cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     0

(1 rows)

TRUNCATE also takes a snapshot of its own before it deletes anything, because auto_snapshot is on by default. It appears next to yours, named <epoch-ms>-sensor_by_day (on the re-run, 1791976469830-sensor_by_day), and it is a complete restore source if you never took one:

docker exec scylla-ha-scylla1-1 sh -c 'ls /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/'

Step 7 — Why the obvious restore does nothing

The obvious restore is: copy the snapshot’s SSTables back next to the live ones, then tell the node to pick them up.

docker exec scylla-ha-scylla1-1 sh -c '
  cp /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp/* \
     /var/lib/scylla/data/poc/sensor_by_day-*/'
docker exec scylla-ha-scylla1-1 nodetool refresh poc sensor_by_day

That command exits 0. It prints nothing. And the count is still zero, at every consistency level and from every coordinator.

Do not leave them there, either. Copying files into a running node’s table directory is not a supported operation, and the node will read whatever is in that directory at its next start. Delete the copies, which carry the same names as the files in snapshots/poc_bkp/, before you go on.

What is not on this list is the truncation. A pre-truncate snapshot loads into the truncated table normally, and the next step does exactly that.

Step 8 — The restore that works

No drop, no recreate. First confirm which directory is the live table. The id is what the directory suffix has to match:

docker exec scylla-ha-scylla1-1 cqlsh -e "
SELECT id FROM system_schema.tables
 WHERE keyspace_name='poc' AND table_name='sensor_by_day';"

On a table that was never dropped there is exactly one sensor_by_day-* directory, and its suffix is that id without the hyphens. If there are two, someone dropped and recreated the table, and only the one that matches is live.

Copy one node’s snapshot into the live table’s upload/, as the scylla user, then refresh on that node only:

docker exec scylla-ha-scylla1-1 sh -c '
  cp /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp/* \
     /var/lib/scylla/data/poc/sensor_by_day-*/upload/'
docker exec scylla-ha-scylla1-1 nodetool refresh poc sensor_by_day --load-and-stream

One node is enough. A 2026.1 keyspace uses tablets by default, and on a tablets keyspace refresh streams each loaded SSTable to the replicas that own its data whether or not you pass --load-and-stream; I pass it so the command means the same thing on a vnode keyspace, where a plain refresh loads only into the local node. Upstream’s procedure is the same: upload each SSTable to one node, refresh with --load-and-stream. Loading all three nodes’ snapshots streams every row three times. My August run loaded all three, which is the likeliest reason it reported an SSTable count of 6.

Verify.

cqlsh> CONSISTENCY QUORUM;
Consistency level set to QUORUM.
cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     3

(1 rows)

Three rows, at QUORUM, from a snapshot taken before the truncate and loaded into the truncated table. The same count came back at ONE through each of the three coordinators and at ALL. nodetool tablestats on each node confirms the load reached every replica:

for n in 1 2 3; do
  docker exec scylla-ha-scylla$n-1 nodetool tablestats poc.sensor_by_day | grep "SSTable count"
done

		SSTable count: 1
		SSTable count: 1
		SSTable count: 1

Two things worth saying about what this restore is and is not. First, snapshots are per node. On this ring RF=3, so one healthy node’s snapshot holds every row; on a larger ring with RF smaller than the node count, each node’s snapshot is a different subset, and you load each of them, each once, with --load-and-stream. Second, refresh is a load, not a repair. Real disaster recovery on this engine also involves nodetool repair for anti-entropy and node rebuild to restream from replicas, and gc_grace_seconds governs when tombstones can actually be collected — get that wrong and you resurrect deleted rows.

For monitoring, ScyllaDB exposes Prometheus metrics on port 9180 — scylla_storage_proxy_* for coordinator-side latency and errors, and reactor utilization for the shard-per-core saturation signal that is specific to this engine.

Step 9 — Clean up

docker compose -f scylla-ha/docker-compose.yml down -v

Verify.

$ docker volume ls --filter name=scylla-ha
DRIVER    VOLUME NAME

Three nodes’ worth of SSTables and snapshots come to real disk. down without -v keeps all of it.

Troubleshooting

The next post in this series is PostgreSQL 18 won’t take a volume at /var/lib/postgresql/data. The rig itself, and the full eight-scenario results matrix, is in A failover lab: six database engines, eight scenarios, every one run.