Blog

When profiling a backup with many small volumes, a large portion of the time spent went to generating the AES header IV. This blog post describes the problem, the solution, and the performance improvements. The solution has been merged in the sharpaescrypt project in pull request #1 and published in the nuget package SharpAESCrypt 2.0.3, which is now used in Duplicati in the merged pull request #5597.

The AES header IV generation was slow as it queried the operating system for a MAC address on every call (and it had an error, resulting in a default value being used). Fixing the error and caching the MAC address improved the performance by up to 1.85 times. This led to encryption being essentially free for this particular backup.

During the benchmarking of Duplicati under different parameter configurations, one particular step in the process took up a considerable amount of time (20 minutes out of the 65 minute total runtime). This blog post describes the identification of the problem, the solution, and the resulting impact. The solution has been merged in the pull request #5595.

We recently had to migrate part of our app hosted with DigitalOcean to make it more resilient and scalable. As a backup client, Duplicati collects backup report data, with events happening throughout the day. This means there were no particular off-peak hours we could migrate in. Since the clients do not retry submitting the reports, any downtime would mean lost reports. Here’s how we worked around it to bring downtime down from 10 minutes to just 30 seconds…

With anything related to security it is virtually impossible to anticipate every single scenario in advance, and this gives attackers an advantage. If you do not use JWT, you are essentially betting on your solution being smarter than people hired specifically to create a secure authentication and authorization system.

The article recommends “just using a ‘normal’ opaque session token and storing it in the database”. That is certainly easier to implement and understand: just create a random string, and check that the user sends the correct string. What could go wrong? …

A user recently asked for an overview of Duplicati from a security perspective. Because this information isn’t currently documented in one place, I’m sharing a version of that overview here to explain the different components that make up Duplicati, and why they were chosen to provide maximal safeguards.

But first, some context:

How traditional backups usually work

A traditional backup is typically created by making an initial full copy of the files. This first backup takes up a lot of space and is problematic for storage over the internet, with limited bandwidth.

The trouble with incremental backups

To save on storage of subsequent backups, most systems rely on incremental backups after that initial backup. Rather than …

I promised myself I would never write a file format without a version field, but here we are.

As part of upgrading Duplicati to .NET8, I discovered that the updater manifest file for Duplicati pointed to the “one golden zip file” that was used with .NET4. With .NET8 there is no golden zip file since we have different versions for each operating system, so I had to rework the contents of the file in an incompatible way. This presented a couple of problems:

• After trying to just enter the new content, I realized the key-length was too short and the file format didn’t support changing the key size.

• Worse: This was the first time I had to update the file format, and I realized there was no version field.

Duplicati is a free and open source backup client that securely stores encrypted, incremental, compressed backups on cloud storage services and remote file servers. I’ve worked on the project for 15 years, along with over 100 contributors, but for various life reasons have had less time to dedicate to the project in the last few years. PRs have been taking longer to get merged, and bigger changes (like upgrading to .NET 8) were dragging.

In March, I announced the formation of Duplicati, Inc, an open core company building on top of the open source project. This development means I can dedicate more time to the project and hire developers to work on improvements and add to the project’s velocity. One of the first major improvements I wanted to make is upgrading the project to .NET 8.