Internally, Duplicati uses SQLite to keep track of the files, their blocks, the file hashes, the block hashes, etc. The reason for using SQLite is that it is a self-contained file-based database, which makes it easy to deploy and use without needing a separate database server. Furthermore, it’s a mature and well-tested database engine, it’s lightweight, and it has a small footprint, all of which should make it a good fit for Duplicati, both in terms of performance, stability, deployability, and ease of use.

However, achieving high performance with SQLite requires some tuning and optimizations, especially when dealing with large datasets and high query rates. This became especially apparent …