nodetool refresh only reads upload/ — and TRUNCATE hides your SSTables

What went wrong

I had a good snapshot. I had the SSTable files on disk. I ran nodetool refresh on all three nodes, got no errors, and SELECT count(*) returned zero.

It took three separate discoveries to get the rows back, and none of them produced an error message at any point. That is what makes this expensive: every step in the obvious restore procedure succeeds, and the data still is not there.

The three behaviors, stated up front so you can stop reading if one of them is your problem:

1. nodetool refresh only loads SSTables from the table’s upload/ subdirectory. Files dropped into the table directory itself are ignored — silently, with a zero exit code.
2. TRUNCATE writes a truncation marker, and that marker hides every SSTable written before it. You can put the old files back and they stay invisible. Recovering from a TRUNCATE needs DROP TABLE and a recreate, not another restore attempt.
3. Recreating a table gives it a new UUID, and the on-disk directory is named after that UUID. The path you carefully copied files into a minute ago now belongs to a dead table.

There is a fourth thing, which happens before any of this and is the reason the cluster would not start in the first place. It gets Step 1.

The compose file is reproduced inline below. There is no public repository to clone.

Step 1 — Raise fs.aio-max-nr before the third node

The first two Scylla nodes came up fine. The third exited during startup with:

system does not satisfy minimum AIO requirements

ScyllaDB’s Seastar runtime reserves asynchronous I/O contexts per reactor when it starts. fs.aio-max-nr is a kernel-wide ceiling on those contexts, and the Linux VM behind Docker Desktop ships with a low default. Two nodes fit under it. Three did not.

The limit belongs to the VM, not to a container, so raise it from a privileged throwaway container. It applies to every container on that daemon, including ones you start later.

docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576

Verify.

$ docker run --rm --privileged alpine sysctl -w fs.aio-max-nr=1048576
fs.aio-max-nr = 1048576

Step 2 — Bring the ring up one node at a time

Three nodes, each pinned to a single shard and 1200 MB, which is what makes three of them fit on one laptop. scylla1 is the seed; the other two join it.

name: scylla-ha

services:
  scylla1:
    image: scylladb/scylla:2026.1
    command: ["--seeds=scylla1", "--smp=1", "--memory=1200M", "--overprovisioned=1", "--api-address=0.0.0.0"]
    ports: ["9042:9042"]
    volumes: ["s1:/var/lib/scylla"]

  scylla2:
    image: scylladb/scylla:2026.1
    command: ["--seeds=scylla1", "--smp=1", "--memory=1200M", "--overprovisioned=1", "--api-address=0.0.0.0"]
    volumes: ["s2:/var/lib/scylla"]
    depends_on: [scylla1]

  scylla3:
    image: scylladb/scylla:2026.1
    command: ["--seeds=scylla1", "--smp=1", "--memory=1200M", "--overprovisioned=1", "--api-address=0.0.0.0"]
    volumes: ["s3:/var/lib/scylla"]
    depends_on: [scylla1]

volumes:
  s1:
  s2:
  s3:

depends_on only controls start order, not readiness, so bring the nodes up by hand. Start the seed, wait for it to report Up/Normal, then add one joiner at a time.

docker compose -f scylla-ha/docker-compose.yml up -d scylla1

docker exec scylla-ha-scylla1-1 nodetool status

Once scylla1 shows UN, add the second, wait again, then the third:

docker compose -f scylla-ha/docker-compose.yml up -d scylla2

docker compose -f scylla-ha/docker-compose.yml up -d scylla3

Verify. nodetool status lists three nodes at UN — Up and Normal — each holding 256 tokens:

UN 172.21.0.2 ... rack1
UN 172.21.0.3 ... rack1
UN 172.21.0.4 ... rack1

A node stuck at UJ is still joining and streaming; give it time rather than restarting it. A node at DN never made it — check that its AIO problem is fixed and that the seed was UN before it started.

Step 3 — Create the keyspace and a query-driven table

RF=3 on a three-node cluster means every node holds every row, which is what makes the QUORUM behavior in the next step easy to reason about.

CREATE KEYSPACE poc WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'datacenter1': 3
};

CREATE TABLE poc.sensor_by_day (
  sensor_id text,
  day       date,
  ts        timestamp,
  value     double,
  PRIMARY KEY ((sensor_id, day), ts)
) WITH CLUSTERING ORDER BY (ts DESC);

The partition key is (sensor_id, day), not sensor_id alone. Bucketing a time series by day bounds the partition: a sensor that reports for two years does not accumulate one enormous partition that has to be read, repaired and compacted as a unit. This is the single most common modeling mistake in Cassandra-family stores, and it does not show up until the partition is already too big to fix comfortably.

Three rows are enough for the whole exercise:

docker exec scylla-ha-scylla1-1 cqlsh -e "
INSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:00:00',42.0);
INSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:01:00',43.5);
INSERT INTO poc.sensor_by_day (sensor_id, day, ts, value) VALUES ('s1','2026-06-07','2026-06-07 09:02:00',41.2);"

Verify.

cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     3

(1 rows)

Step 4 — Confirm RF=3, then lose a node

Before testing the failure, confirm the replication actually landed where you think it did. nodetool getendpoints answers “which nodes hold this partition” for a specific partition key.

docker exec scylla-ha-scylla1-1 nodetool getendpoints poc sensor_by_day "s1:2026-06-07"

Verify. Three addresses, one per node, because RF=3 on a three-node ring means every node is a replica:

172.21.0.2
172.21.0.3
172.21.0.4

Now stop one node and read at QUORUM. QUORUM of RF=3 is 2, so two survivors are exactly enough.

docker stop scylla-ha-scylla2-1

Verify.

cqlsh> CONSISTENCY QUORUM;
Consistency level set to QUORUM.
cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     3

(1 rows)

That is the whole availability claim, checked. Stop a second node and the same query starts failing with an unavailable exception, which is also correct — at that point the cluster cannot prove it is returning current data, so it refuses rather than guessing.

Bring the node back before continuing:

docker start scylla-ha-scylla2-1

Step 5 — Take a snapshot on every node

nodetool snapshot flushes memtables and then hardlinks the current SSTables into a snapshots/<tag> directory under the table. It costs almost nothing at the moment you take it, because hardlinks; it starts costing disk later as compaction replaces the live files and the snapshot keeps the old ones alive.

A snapshot is per node. There is no cluster-wide snapshot command — you run it everywhere.

for n in 1 2 3; do
  docker exec scylla-ha-scylla$n-1 nodetool snapshot -t poc_bkp poc
done

Verify. The snapshot directory exists under the table directory:

$ docker exec scylla-ha-scylla1-1 sh -c 'ls -d /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp'
/var/lib/scylla/data/poc/sensor_by_day-6f1c8ad0-6373-11f1-9c4d-3a17e2c3b901/snapshots/poc_bkp

The UUID in that path is the table’s identity, and it will be different on your ring. Remember that it is there. It becomes the third gotcha in Step 7.

Step 6 — Destroy the data

The loss scenario. TRUNCATE rather than a dropped table, because truncate is the mistake people actually make — it is one word away from a SELECT in a terminal that is pointed at the wrong environment.

docker exec scylla-ha-scylla1-1 cqlsh -e "TRUNCATE poc.sensor_by_day;"

Verify.

cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     0

(1 rows)

Step 7 — Why the obvious restore does nothing

The obvious restore is: copy the snapshot’s SSTables back next to the live ones, then tell the node to pick them up.

docker exec scylla-ha-scylla1-1 sh -c '
  cp /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp/* \
     /var/lib/scylla/data/poc/sensor_by_day-*/'
docker exec scylla-ha-scylla1-1 nodetool refresh poc sensor_by_day

That command exits 0. It prints nothing. And the count is still zero.

Move the files into upload/ and refresh again, and the count is still zero. That is the second behavior.

So: drop, recreate, copy into upload/, refresh. And it is still zero, for a third reason.

Three behaviors, three silent no-ops, one restore that has to be done in a specific order to work.

Step 8 — The restore that works

The ordering matters: stage the snapshot files out of the table directory first, because the drop in the next command takes the directory context with it.

Stage on every node:

for n in 1 2 3; do
  docker exec scylla-ha-scylla$n-1 sh -c '
    mkdir -p /tmp/stage &&
    cp /var/lib/scylla/data/poc/sensor_by_day-*/snapshots/poc_bkp/* /tmp/stage/'
done

Drop and recreate the table, which discards the truncation marker:

docker exec scylla-ha-scylla1-1 cqlsh -e "DROP TABLE poc.sensor_by_day;"
docker exec scylla-ha-scylla1-1 cqlsh -e "
CREATE TABLE poc.sensor_by_day (
  sensor_id text, day date, ts timestamp, value double,
  PRIMARY KEY ((sensor_id, day), ts)
) WITH CLUSTERING ORDER BY (ts DESC);"

Confirm which directory is now the live one. The id here is what the directory suffix has to match:

docker exec scylla-ha-scylla1-1 cqlsh -e "
SELECT id FROM system_schema.tables
 WHERE keyspace_name='poc' AND table_name='sensor_by_day';"

docker exec scylla-ha-scylla1-1 sh -c 'ls -d /var/lib/scylla/data/poc/sensor_by_day-*'

You will see at least two directories: the stale one from before the drop, and the new live one. Copy into the new one’s upload/, on every node:

for n in 1 2 3; do
  docker exec scylla-ha-scylla$n-1 sh -c '
    LIVE=$(ls -dt /var/lib/scylla/data/poc/sensor_by_day-* | head -1)
    echo "$LIVE"
    mkdir -p "$LIVE/upload"
    cp /tmp/stage/* "$LIVE/upload/"
    chown -R scylla:scylla "$LIVE/upload"'
done

Now refresh, on every node:

for n in 1 2 3; do
  docker exec scylla-ha-scylla$n-1 nodetool refresh poc sensor_by_day
done

Verify.

cqlsh> CONSISTENCY QUORUM;
Consistency level set to QUORUM.
cqlsh> SELECT count(*) FROM poc.sensor_by_day;

 count
-------
     3

(1 rows)

Three rows, read at QUORUM, restored from a snapshot taken before a truncate. nodetool tablestats will confirm the loaded files are being served — on my run it reported a read count of 40 and an SSTable count of 6 for this table after the exercise.

docker exec scylla-ha-scylla1-1 nodetool tablestats poc.sensor_by_day

Two things worth saying about what this restore is and is not. First, snapshots are per node and this ring is RF=3, so every node had a complete copy and loading all three was straightforward; on a larger ring with RF smaller than the node count, each node’s snapshot is a different subset and the staging has to keep them apart. Second, refresh is a load, not a repair. Real disaster recovery on this engine also involves nodetool repair for anti-entropy and node rebuild to restream from replicas, and gc_grace_seconds governs when tombstones can actually be collected — get that wrong and you resurrect deleted rows.

For monitoring, ScyllaDB exposes Prometheus metrics on port 9180 — scylla_storage_proxy_* for coordinator-side latency and errors, and reactor utilization for the shard-per-core saturation signal that is specific to this engine.

Step 9 — Clean up

docker compose -f scylla-ha/docker-compose.yml down -v

Verify.

$ docker volume ls --filter name=scylla-ha
DRIVER    VOLUME NAME

Three nodes’ worth of SSTables and snapshots come to real disk. down without -v keeps all of it.

Troubleshooting

The next post in this series is PostgreSQL 18 won’t take a volume at /var/lib/postgresql/data. The rig itself, and the full eight-scenario results matrix, is in A failover lab you can run on one laptop.