A Container Is Not a Recovery Plan: Three Stateful Pi Swarm Trials

A container can come back on another Raspberry Pi and still have lost everything that made it useful. Docker Swarm knows where to run a task. It does not carry a node-local database along for the ride.

I tested the less glamorous half of recovery in a four-Pi lab: stop a disposable worker’s Docker daemon, recover a known backup somewhere else, and prove that the records—not merely the web page—came back.

Three persistence recipes passed. Linkding recovered in 64.5 seconds, a PostgreSQL-only test workload in 28.9 seconds, and a Redis-only test workload in 24.4 seconds. Those are useful experimental results, not a promise of automatic high availability.

One real application, two deliberately small database tests

Linkding was the actual application: a SQLite-backed bookmark manager populated through its HTTP API. The PostgreSQL and Redis trials were smaller synthetic workloads operated through their native database clients. They were not full PostgreSQL-backed or Redis-backed applications, and I am not counting them as such.

Every image was pulled and verified as ARM64 before the relevant trial. Tests ran sequentially, with CPU and process limits plus small database-specific settings. Nothing was migrated from production. Each workload had uniquely labeled, disposable data and a separate recovery destination.

The backup recipes differed in ways that matter. SQLite used its online backup API to produce a consistent standalone database. PostgreSQL used a logical dump while the controlled synthetic workload was quiescent. Redis used a synchronous SAVE and copied the completed RDB snapshot. A manifest recorded original capture times, expected records, image identity, and a synthetic payload checksum. Restic then protected that recovery unit in an isolated repository on a surviving Pi.

The failure was real, but carefully bounded

For each measured trial I stopped both Docker’s service and its activation socket on the disposable source worker. Before doing so, I checked that the worker had no unrelated running application workload. The operating system and SSH stayed up. An independent timed restart safeguard and a script cleanup path brought Docker back.

This was therefore a Docker-outage drill, not a power-cut test. It exercised loss of the source application runtime without pretending to exercise damaged filesystems, abrupt storage loss, or a dead machine.

While source Docker was verified inactive, the surviving host successfully listed the repository and restored the exact recorded snapshot. That proves the recovery source did not depend on the failed Docker runtime. It does not prove offsite disaster recovery.

Fence first. Restore second. Start last.

The service was scaled to zero and constrained to one recovery destination before restored data was allowed to start. Original data stayed on the source, but that source was no longer an eligible writer. SQLite and Redis data were placed before application startup. PostgreSQL was imported using one isolated temporary initializer, which was removed before the Swarm task started.

When the original worker returned, I checked that it had no competing trial writer and that the destination had exactly one. All four nodes returned to Ready/Active, and their management connections were healthy. A replacement task running beside an old writer would not have counted as success.

What the stopwatch measured

• Linkding: script-local outage confirmation 2.7 seconds; Swarm reported the worker down at 15.1 seconds; restored-data preparation 1.5 seconds; application readiness and record verification 36.3 seconds; total recovery 64.5 seconds.
• PostgreSQL: outage confirmation 1.2 seconds; Swarm down at 14.6 seconds; restore and isolated import 5.4 seconds; readiness and native verification 2.6 seconds; total 28.9 seconds.
• Redis: outage confirmation 1.3 seconds; Swarm down at 14.7 seconds; restore preparation 1.5 seconds; readiness and native verification 1.9 seconds; total 24.4 seconds.

Total recovery ran from issuing the Docker-stop operation to successful restored-record verification. It includes orchestration and fencing overhead beyond the named stages. Preparation and image downloads are excluded. Each recipe had one measured failure injection; these are not averages or tail-latency estimates. All three finished below the experiment’s five-minute target.

The missing record was part of the proof

Each snapshot contained one deliberately created record. I then wrote a second record, verified that it existed at the source, and injected the outage. After recovery, the first record and its payload checksum matched; the post-snapshot record was absent.

That absence is not a defect in the restore. It is what restoring that exact recovery point should do. Original captures were only 3.7, 3.9, and 4.3 seconds old at failure injection. This demonstrates snapshot semantics with fresh tiny datasets. It says nothing about an existing nightly backup schedule or how many hours of changes that schedule might lose.

Linkding’s synthetic payload lived in bookmark notes. PostgreSQL stored bytes in a bytea field, and Redis stored a hash payload. No Linkding file attachment was created, so file-attachment recovery remains untested.

The rough edges are worth keeping in the report

The first PostgreSQL bootstrap failed: rejecting network authentication does not waive the official image’s initialization password requirement. Docker’s default convergence wait then outlived the orchestration tool’s deadline. I fenced that disposable attempt, supplied a protected password file, and switched service creation to detached mode with bounded readiness polling. The subsequent recovery trial passed. That preparation failure is separate from the measured recovery interval.

A repeated read-only SQLite verification also needed attention. A completed standalone online-backup file can retain a WAL-mode header and otherwise try to create sidecar files on a read-only mount. Immutable read-only mode solved that isolated-artifact check. It would be inappropriate for a live database or one still dependent on unmerged WAL data.

All three exact snapshots then passed a second, separately launched native restore check using new validation directories. Temporary validation containers were removed, and receipts were retained.

What this experiment does not establish

The recovery code was manually launched and scripted. No unattended failover service was installed. Automatic Swarm scheduling and scripted data recovery are different mechanisms.

Images were already available and datasets were tiny. Cold downloads, heavy application load, reverse-proxy routing, manager failure, power loss, offsite recovery, and production backup freshness were not tested. Worker memory controllers were absent, so Docker’s memory accounting and memory caps could not be treated as reliable. CPU and process bounds and application-level database settings helped keep the experiment small; they are not a capacity study.

The practical result is narrower and still encouraging: for these three disposable persistence recipes, an exact backup on another Pi could be recovered safely, with exclusive ownership and verified data, in well under five minutes. Before adding replicated storage, that is a recovery option worth understanding.