There is an old administrative proverb that remains undefeated in the world of systems engineering: An untested backup is just a wish.

You can set up elegant backup schedules, configure military-grade AES-256 encryption, and send encrypted blocks to ultra-redundant cloud storage. But if a hard drive fails, ransomware strikes, or your server dies, none of that matters unless you can answer one critical question: Can you actually restore your data from scratch?

Running a periodic "backup fire drill" is the difference between smooth recovery and a panic-filled weekend. In this deep dive, we’ll look under the hood at how Duplicati handles total local data loss, how to set up automated integrity checks, and how to perform out-of-band emergency restores when the standard interface isn't an option.

1. Proactive Defense: Automated Integrity Checks & Test Restores

The worst time to discover a corrupt storage bucket or missing chunk on your cloud backend is when you’re attempting a disaster recovery. Duplicati provides built-in mechanisms to verify that what you uploaded remains intact and readable.

Automated Remote Verification

By default, Duplicati performs a lightweight sanity check at the end of every backup run. It randomly selects a small sample of remote files, downloads them, decrypts them, and verifies that their content hashes match the local records.

You can tune this behavior using advanced options in your job configuration:

• --backup-test-samples=N: Specifies how many remote files (from each volume type) to download and verify after each backup (default is 1). Increasing this to 3 or 5 gives higher confidence on larger repositories without imposing heavy bandwidth costs.

  
  

• --backup-test-percentage=N: Alternatively, set a percentage (e.g., 0.1 or 2.5) of all remote files to test after every run.

  
  

• --full-remote-verificaiton=true: Instead of simply downloading and checking the hash matches the recorded hash, also decrypt and decompress and check the internal contents matches the local database.

  
  

Automated Test Restores

Verification testing checks volume health, but testing actual file reconstruction tests the entire pipeline. You can automate periodic test restores using the Duplicati Command Line (duplicati-cli) wrapped in a simple cron job or system task:

# Example: Restore a specific critical folder to a temporary directory
duplicati-cli restore "s3://my-backup-bucket/data" "C:\ImportantDocs\*" \
  --passphrase="YourSecretEncryptionKey" \
  --restore-path="/tmp/test-restore/" \
  --overwrite=true

# Example: Restore a specific critical folder to a temporary directory
duplicati-cli restore "s3://my-backup-bucket/data" "C:\ImportantDocs\*" \
  --passphrase="YourSecretEncryptionKey" \
  --restore-path="/tmp/test-restore/" \
  --overwrite=true

# Example: Restore a specific critical folder to a temporary directory
duplicati-cli restore "s3://my-backup-bucket/data" "C:\ImportantDocs\*" \
  --passphrase="YourSecretEncryptionKey" \
  --restore-path="/tmp/test-restore/" \
  --overwrite=true

If your test script successfully reconstructs the sample directory and checks file hashes, your DR pipeline is proven working.

2. Under the Hood: How Duplicati Rebuilds Its SQLite Database

To understand how Duplicati recovers from total hardware failure—a "bare-metal" restore, you first need to understand how local and remote state interact.

Local State vs. Remote Storage

Duplicati uses a stateless, client-side deduplication architecture:

1. Local SQLite Database: Acts as an index map. It tracks file paths, modification times, block hashes, and maps which block hashes are stored inside which remote volumes.

   
   

2. Remote Storage: Contains three distinct types of compressed, encrypted files:

   • dlist (Directory List): Snapshots containing the entire file tree, metadata, file sizes, and block hashes for a specific backup set.

   • dindex (Index Files): Small lookup maps that correlate specific block hashes to their corresponding dblockpayload files.

   • dblock (Data Blocks): The actual compressed and encrypted data payloads.

     
     

┌──────────────────────────────────────────────────────────┐
│                   Remote Storage                         │
│                                                          │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐  │
│  │  dlist File  │   │ dindex File  │   │ dblock File  │  │
│  │ (File Tree)  │   │  (Hash Map)  │   │ (Data Chunks)│  │
│  └──────┬───────┘   └──────┬───────┘   └──────────────┘  │
└─────────┼──────────────────┼─────────────────────────────┘
          │                  │
          ▼                  ▼
┌──────────────────────────────────────────────────────────┐
│                  Local Reconstruction                    │
│                                                          │
│              Rebuilt SQLite Local Database               │
└──────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────┐
│                   Remote Storage                         │
│                                                          │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐  │
│  │  dlist File  │   │ dindex File  │   │ dblock File  │  │
│  │ (File Tree)  │   │  (Hash Map)  │   │ (Data Chunks)│  │
│  └──────┬───────┘   └──────┬───────┘   └──────────────┘  │
└─────────┼──────────────────┼─────────────────────────────┘
          │                  │
          ▼                  ▼
┌──────────────────────────────────────────────────────────┐
│                  Local Reconstruction                    │
│                                                          │
│              Rebuilt SQLite Local Database               │
└──────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────┐
│                   Remote Storage                         │
│                                                          │
│  ┌──────────────┐   ┌──────────────┐   ┌──────────────┐  │
│  │  dlist File  │   │ dindex File  │   │ dblock File  │  │
│  │ (File Tree)  │   │  (Hash Map)  │   │ (Data Chunks)│  │
│  └──────┬───────┘   └──────┬───────┘   └──────────────┘  │
└─────────┼──────────────────┼─────────────────────────────┘
          │                  │
          ▼                  ▼
┌──────────────────────────────────────────────────────────┐
│                  Local Reconstruction                    │
│                                                          │
│              Rebuilt SQLite Local Database               │
└──────────────────────────────────────────────────────────┘

What Happens During a Bare-Metal Database Rebuild?

Imagine your client machine completely explodes. The local SSD dies, taking the SQLite database with it. You provision a brand-new machine, install Duplicati, point it to your cloud storage bucket, and click Rebuild Database (or run a restore directly).

Here is the sequence Duplicati executes behind the scenes:

1. Header & Metadata Scan: Duplicati fetches the list of all remote volume filenames from the remote backend.

   
   

2. dlist Processing: It downloads all small dlist files first. From these, Duplicati recreates the exact historical snapshot trees, reconstructing which files existed, their paths, metadata, and required block hash chains for every backup version.

   
   

3. dindex Processing: Duplicati downloads the lightweight dindex files. Because these index files store block-to-volume associations, Duplicati can populate its local SQLite lookup table without needing to download massive dblock payload archives.

   
   

4. Targeted dblock Inspection (Only If Needed): If dindex files are missing or incomplete, Duplicati will fall back to downloading individual dblock files to index their contents directly. Once processing is complete, the local SQLite database is fully restored.

   
   

Because the metadata (dlist) and index mappings (dindex) are kept separate from raw payload data (dblock), rebuilding a database mapping terabytes of backup data only requires downloading a small fraction of that size over the network.

If there is no local database and you start a restore process, Duplicati will figure out which part of the database needs to be rebuilt, and will only rebuild a partial database to speed up the operation. If you plan on multiple restores, you may want to recreate the database in full first, so each subsequent operation is faster.

3. Emergency Mode: Out-of-Band Restores with RecoveryTool

What if disaster strikes and you cannot access the standard Duplicati Web GUI? What if the background service won't start, or you are working on a stripped-down rescue environment where running the full Duplicati is impractical? What if the remote data is badly damaged and it is not possible to create the local database?

This is where the standalone Duplicati Recovery Tool (duplicati-recovery-tool or Duplicati.CommandLine.RecoveryTool.exe) comes in.

Why Use RecoveryTool?

The standard GUI and CLI operations rely on constructing a full local SQLite database before extracting files. While this provides fast, indexable lookups, constructing that database on a massive repository can take time.

RecoveryTool is designed as a fail-safe, linear extraction utility. It operates independently of the main server database, performing recovery in direct, sequential steps.

Step 1: Download  ──►  Step 2: Index  ──►  Step 3: Restore

Step 1: Download  ──►  Step 2: Index  ──►  Step 3: Restore

Step 1: Download  ──►  Step 2: Index  ──►  Step 3: Restore

Step-by-Step RecoveryTool Execution

Step 1: Download Remote Volumes

First, pull all encrypted volumes from your storage target to a local folder and decrypt them in transit:

duplicati-recovery-tool download \
  "s3://my-backup-bucket/data" \
  "/tmp/local_recovery_cache" \
  --passphrase="YourSecretEncryptionKey" \
  --aws-access-key-id="YOUR_KEY" \
  --aws-secret-access-key="YOUR_SECRET"

duplicati-recovery-tool download \
  "s3://my-backup-bucket/data" \
  "/tmp/local_recovery_cache" \
  --passphrase="YourSecretEncryptionKey" \
  --aws-access-key-id="YOUR_KEY" \
  --aws-secret-access-key="YOUR_SECRET"

duplicati-recovery-tool download \
  "s3://my-backup-bucket/data" \
  "/tmp/local_recovery_cache" \
  --passphrase="YourSecretEncryptionKey" \
  --aws-access-key-id="YOUR_KEY" \
  --aws-secret-access-key="YOUR_SECRET"

Step 2: Index Local Files

Next, instruct the tool to inspect the downloaded unencrypted volumes and generate a standalone text index (index.txt) of all block hashes:

duplicati-recovery-tool index \
  "/tmp/local_recovery_cache" \
  --indexfile="/tmp/local_recovery_cache/index.txt"

duplicati-recovery-tool index \
  "/tmp/local_recovery_cache" \
  --indexfile="/tmp/local_recovery_cache/index.txt"

duplicati-recovery-tool index \
  "/tmp/local_recovery_cache" \
  --indexfile="/tmp/local_recovery_cache/index.txt"

Step 3: Restore Files

Finally, reconstruct your files to your desired restore location. You can specify a version number or target specific paths:

duplicati-recovery-tool restore \
  "/tmp/local_recovery_cache" \
  0 \
  --indexfile="/tmp/local_recovery_cache/index.txt" \
  --target-path="/mnt/restored_data"

duplicati-recovery-tool restore \
  "/tmp/local_recovery_cache" \
  0 \
  --indexfile="/tmp/local_recovery_cache/index.txt" \
  --target-path="/mnt/restored_data"

duplicati-recovery-tool restore \
  "/tmp/local_recovery_cache" \
  0 \
  --indexfile="/tmp/local_recovery_cache/index.txt" \
  --target-path="/mnt/restored_data"

Because RecoveryTool works directly against local files using simple disk-backed index files, it bypasses database locks, service dependencies, and complex server configurations. It is your ultimate insurance policy.

A more lightweight RecoveryTool

Duplicati also has a fully self-contained Python recovery tool that works the same way as the bundled RecoveryTool. This tool does not support backends, so you need to somehow make the files available with mounts or similar, but otherwise makes the restore in the same linear flow.

Summary Checklist for Your DR Fire Drill

To ensure you are fully prepared for disaster, run through this checklist once every quarter:

• [ ] Verify Settings: Ensure --backup-test-samples or --backup-test-percentage is set to appropriate values on all core jobs.

  
  

• [ ] Off-Site Passphrase Storage: Confirm your encryption passphrases and bucket credentials are stored securely outside the source machine (e.g., in a secure password manager).

  
  

• [ ] Test Database Rebuild: Practice creating a temporary backup job on a test machine and running a full database rebuild from remote storage.

  
  

• [ ] Run RecoveryTool: Execute an out-of-band test restore using duplicati-recovery-tool to confirm complete independence from the GUI service.

  
  

Don't wait for the fire to start before reading the extinguisher manual. Test your restores today!