News
Under the Hood: How Duplicati's Block-Level Deduplication and Encryption Work
Peek under the hood of Duplicati to discover how our zero-trust architecture protects your data long before it reaches the cloud. Learn how client-side AES-256 encryption, block-level deduplication, and local SQLite state tracking work together to deliver maximum privacy, minimal bandwidth usage, and seamless disaster recovery.
When it comes to backups, "just trust us" isn't a strategy. Modern backup software shouldn't operate as a opaque black box. Especially when it’s responsible for protecting sensitive production data, financial records, or personal history before pushing it across the public internet.
At Duplicati, our core engineering philosophy rests on technical transparency and zero-trust security. We believe security auditors, system administrators, and developers deserve to know exactly what happens to their data before a single byte leaves the host machine.
Here is a look inside the architecture powering Duplicati's encryption, deduplication, and state reconstruction.
1. Client-Side AES-256 Encryption: True Zero-Trust (BYOK)
Duplicati is designed ground-up as a zero-trust backup engine. In a zero-trust model, remote storage targets - whether AWS S3, Backblaze B2, Google Cloud Storage, or a remote SSH server - are treated as inherently untrusted environments.
How Key Management Works
Bring-Your-Own-Key (BYOK): Duplicati relies on strong client-side encryption using standard AES-256 or GNU Privacy Guard (GPG).
Local Processing: Encryption occurs strictly in local memory before payload packaging and network transmission. Unencrypted plain-text data never hits the wire.
Zero Knowledge Architecture: Neither the cloud destination nor Duplicati maintains a centralized key server. Your passphrase acts as the master key. If a storage backend suffers a major data breach, the attacker acquires nothing more than high-entropy, unreadable binary blobs.
Key Security Takeaway: Without your local passphrase, reversing the AES-256 encrypted payload stored in the cloud is computationally infeasible. You maintain full ownership of encryption keys at all times.
2. The Data Pipeline: Chunking, Deduplication, and Compression
Sending full file copies on every backup cycle wastes storage space, consumes massive bandwidth, and stresses network interface controllers (NICs). Duplicati solves this using a multi-stage data processing pipeline:
Step 1: Fixed-Block Chunking
Instead of treating files as atomic units, Duplicati breaks incoming files into uniform byte blocks (default size: 1 MiB, fully configurable based on your dataset size).
If a 10 GB file changes by only 2 KB, traditional file-level backups re-upload the entire 10 GB file. Duplicati analyzes only the altered 1 MiB blocks containing those changed bytes.
Step 2: Content-Addressable SHA-256 Deduplication
Every individual block is run through a SHA-256 cryptographic hashing function to calculate its unique content identifier:
Fingerprint = SHA-256(Block Data)
Duplicati registers this hash in a local registry. If three different files on your system contain identical 1 MiB blocks - or if a file was renamed, moved, or duplicated - Duplicati identifies that matching SHA-256 signature and stores that block only once.
Step 3: Volume Packing & Compression
Cloud providers often impose strict API rate limits or request charges (such as S3 PUT request costs). Uploading millions of 1 MiB files individually would degrade performance and incur high API costs.
To solve this, Duplicati aggregates deduplicated blocks into larger volume archives (typically 50 MiB .zip volumes). These volumes are compressed inline (using standard Deflate, LZMA, or standard compression algorithms) and encrypted with AES-256 before initiating the multi-part upload to your storage backend.
Pipeline Stage | Action | Benefit |
1. Chunking | Splits files into uniform blocks (e.g., 1 MiB) | Isolates file changes to granular byte ranges |
2. Deduplication | Hashes blocks via SHA-256 to index unique data | Eliminates redundant storage across files and versions |
3. Compression | Packs unique blocks into larger volumes (e.g., 50 MiB) | Reduces bandwidth usage and minimizes cloud API request costs |
4. Encryption | Encrypts volume archives locally using AES-256 | Ensures complete data privacy prior to network transit |
3. State Management: The SQLite Local Map & Self-Healing Disaster Recovery
To keep backup runs fast and minimize network latency, Duplicati maintains a local SQLite database. This local database acts as a relational index mapping your host machine's file system state to the remote cloud volumes.
How the Local Index Accelerates Backups
The local SQLite database tracks three key relationships:
Files —> Blocks: Which SHA-256 blocks make up a specific version of a file.
Blocks —> Volumes: Which remote encrypted archive volume (
dblock) contains a given SHA-256 block payload.Volume Catalog: Index of remote manifest files (
dlist) and remote index files (dindex).
During an incremental backup, Duplicati queries its local SQLite database to determine which blocks already exist in the cloud. It bypasses the need to download or query remote manifests over the wire, allowing subsequent runs to complete in seconds.
Disaster Recovery: Rebuilding the Map from Scratch
A common concern with index-driven backup systems is: "What happens if my machine dies, my local drive burns, and I lose the local SQLite database?"
Duplicati is engineered for catastrophic host recovery. Every backup set pushed to remote storage includes lightweight metadata files alongside the encrypted payload volumes:
dblock.zip.aes: The compressed, encrypted payload containing actual data blocks.dindex.zip.aes: Index mapping showing which block hashes live inside correspondingdblockfiles.dlist.zip.aes: Lists file path structures, timestamps, and block references required to restore a given backup set.
In a disaster scenario on a brand-new machine, Duplicati downloads only the lightweight dlist and dindex header files from your cloud backend. By reading these index headers and applying your passphrase, Duplicati reconstructs the entire local SQLite database from scratch.
You don't need to download terabytes of raw data just to recover your backup state- Duplicati restores the index first, allowing you to selectively restore individual files or entire system trees on demand.
Built for Trust, Security, and Efficiency
Duplicati’s underlying mechanics prioritize data ownership, efficiency, and resilience:
You own your keys: Zero-trust AES-256 encryption guarantees complete privacy from third-party storage vendors.
You save resources: Block-level deduplication and volume compression minimize network overhead and storage costs.
You are never locked in: Deterministic metadata files mean your data can always be reconstructed, even if your local hardware is completely destroyed.
By decoupling data storage from state indexing, Duplicati delivers strong cloud-native backup performance without compromising security or software transparency.
Ready to start efficient encrypted backups? Download Duplicati and run backups for free.
