Sonarr recovery without handholding: two unattended fault windows

A restore script is not the same thing as unattended recovery. In the earlier cluster experiments, I launched the scripts and watched them bring the data back. Those runs established useful recipes. They did not establish that anything would happen if I stopped helping.

The next challenge was to remove that help. I built a disposable Sonarr instance, installed a persistent controller, stopped Docker on its worker, and left recovery to ordinary software. The first attempt failed. After fixing the defect and returning to a clean baseline, two subsequent hands-off fault windows passed, restoring onto two different workers.

A real application, without the real library

This was ARM64 Sonarr 4.0.20.3014, using a tested, digest-pinned LinuxServer image. It was not a live Sonarr migration. No production library, NAS mount, downloader, indexer credentials, or production backup repository was involved.

The synthetic state was deliberately more interesting than an empty homepage: tags, a custom quality profile, a tagged delay profile, and a synthetic root-folder record. The second run added another captured tag and changed a delay setting. After each backup, I added a separate marker that should disappear when the application returned to that recovery point.

That distinction matters. A green HTTP response could come from a freshly initialized application with all its useful settings missing. The test instead compared Sonarr’s own API records against the captured state, checked that the later marker was absent, and verified a pre-outage logging record through the application’s logging API.

The automation owns the recovery

A systemd timer runs a small Python controller on a surviving worker. It polls ownership and Docker availability, makes native Sonarr backups on a schedule, and stores qualified recovery units in an isolated Restic repository. Restic runs on the host, so stopping Docker on the source does not also remove the recovery mechanism.

Sonarr’s supported backup command supplies the main database, configuration, and backup information file. Its standard ZIP does not include the logging database, so that database receives a separate SQLite online backup. Each artifact is checked, hashed, and associated with an exact snapshot and original capture time. These are independently consistent database artifacts, not a claimed transaction spanning both databases.

The controller serializes operations, uses durable phase checkpoints, bounds readiness polling, and backs off after failures. Captures are spaced about five minutes apart; recovery refuses a source capture older than thirty minutes. Retention is restricted to the disposable repository and its owned capture artifacts.

Stopping Docker is the fault, not the fencing proof

The deliberately narrow failure mode keeps SSH available. The controller must verify that both the Docker daemon and its socket are inactive and that no relevant writer process remains. It then excludes the old worker from desired placement and retires the old data directory’s original bind path before allowing that worker to return.

Only then does it restore into a fresh generation, validate the required files and databases, apply the correct application ownership, and read back the exact restored bind mount and exclusive placement constraint before starting one replacement writer.

A timed availability safeguard was armed for each fault, but neither successful recovery needed it to fire. The controller itself returned the original worker. A separate read-only observer continued watching afterward and recorded exactly one Sonarr writer, with no stale writer resurrecting on the original host.

An HTTP timeout does not start a restore. An unreachable source does not become conveniently assumed dead. Without independently verified fencing, the controller refuses that recovery path.

The failed attempt was useful—and still failed

The first revision combined mount addition and removal for the same target in one Docker service update. Docker’s command-line processing removed the intended mount. The replacement started without the restored configuration and rejected the original API key.

That violated the test’s central rule: never start an empty replacement and hope the data appears later. I preserved the failed attempt, fixed the update sequence outside the acceptance window, and added an explicit deployment readback that prevents startup unless the restored bind really exists. An identical add/remove constraint operation also needed to become reentrant rather than removing its own placement restriction.

A separate safety review caught another ordinary scripting trap: a failed precondition followed by semicolon-separated commands can be hidden by the final command’s successful exit status. Both local and SSH command execution now fail fast, with negative tests showing that a failed gate prevents later actions. These were development corrections, not invisible nudges during the successful fault windows.

Two hands-off windows, two destinations

• First successful window: 27.9 seconds from fault injection to application-native verification. The original capture was 25.5 seconds old at injection.
• Second successful window: 31.3 seconds to verification on a different worker. A new scheduled capture was 7.3 seconds old and included the new captured marker and changed setting.

Both runs preserved the expected native configuration and lost the deliberately added post-backup marker. Exact snapshot identities, source hashes, database integrity, ownership, and writer counts were retained in receipts. Independent observer logs distinguished the fault injector from the software performing recovery.

Both fault injections used the same otherwise-idle worker so unrelated workloads would not be stopped. A planned relocation reset the baseline between runs; it is not counted as a third unattended fault recovery. During that separate relocation, I also interrupted the controller process after its durable restore checkpoint. The installed timer resumed the operation without an agent-issued repair command.

After the second recovery, the timer produced another successful capture from the new owner. That is a useful final check: the backup producer followed the application rather than continuing to protect an abandoned location.

What this proves, and what it does not

This establishes a small, repeatable unattended recovery path for a disposable stateful application when its worker’s Docker service fails but authenticated SSH remains available. During the successful windows, I injected the fault and observed. I did not issue the scale, placement, restoration, startup, or service-specific repair commands.

It does not establish production-ready high availability. The sole manager stayed alive. Loss of the controller or repository host, full power failure, partition fencing, cold image pulls, and recovery of a real media library remain outside the tested result. Images were already available on the eligible workers. The test data was small. CPU and process limits were configured, but absent memory cgroups mean Docker memory enforcement cannot be claimed.

The useful advance is narrower than an HA slogan: recovery no longer needs an assistant to decide the next command for this qualified failure. The automation stays installed and running, with the receipts explaining exactly where that confidence begins and ends.

Implementation references

• LinuxServer Sonarr image documentation
• Official Sonarr API documentation
• Sonarr backup and restore guidance