gilbertchen/benchmarking

Objective

To benchmark the performance and storage efficiency of 4 backup tools, Duplicacy, restic, Attic, and duplicity, using datasets that are publicly available.

Disclaimer

As the developer of Duplicacy, I have little first-hand experience with other tools, other than setting them up and running for these experiments for the first time for this performance study. It is highly possible that configurations for other tools may not be optimal. Therefore, results presented here should not be viewed as conclusive until they are independently confirmed by other people.

Setup

All tests were performed on a Mac mini 2012 model running macOS Sierra (10.12.3), with a 2.3 GHZ Intel i7 4-core processor and 16 GB memory.

The following table lists several important configuration parameters or algorithms that may have significant impact on the overall performance.

[1] The chunk size in Duplicacy is configurable with the default being 4MB. It was set it to 1MB to match that of restic

[2] Enabled by -e repokey-blake2 which is only available in 1.1.0+

Backing up the Linux code base

The first dataset is the Linux code base mostly because it is the largest github repository that we could find and it has frequent commits (good for testing incremental backups). Its size is 1.76 GB with about 58K files, so it is a relatively small repository consisting of small files, but it represents a popular use case where a backup tool runs alongside a version control program such as git to frequently save changes made between checkins.

To test incremental backup, a random commit on July 2016 was selected, and the entire code base is rolled back to that commit. After the initial backup was finished, other commits were chosen such that they were about one month apart. The code base is then moved forward to these commits one by one to emulate incremental changes. Details can be found in linux-backup-test.sh.

Backups were all saved to a storage directory on the same hard disk as the code base, to eliminate the performance variations introduced by different implementation of networked or cloud storage backends.

Here are the elapsed real times (in seconds) as reported by the time command, with the user CPU times and system CPU times in the parentheses:

Clearly Duplicacy was the winner by a comfortable margin. It is interesting that restic, while being the second fastest, consumed far more CPU times than the elapsed real times, which is bad for the user case where users want to keep the backup tool running in the background to minimize the interference with other tasks. This could be caused by using too many threads (or more precisely goroutines, since restic is written in GO) in its local storage backend implementation. However, even if this issue is fixable, as restic currently does not support compression, the addition of compression will only further slow down its backup speeds.

Now let us look at the sizes of the backup storage after each backup:

Although duplicity was the most storage efficient, it should be noted that it uses zlib, which is known to compress better than lz4 used by Duplicacy and Attic. In addition, duplicity has a serious flaw in its incremental model -- the user has to decide whether to perform a full backup or an incremental backup on each run. That is because while an incremental backup saves a lot of storage space, it is also dependent on previous backups due to the design of duplicity, making it impossible to delete any single backup on a long chain of dependent backups. So there is always a dilemma of how often to perform a full backup for duplicity users.

We also ran linux-restore-test.sh to test restore speeds. The destination directory was emptied before each restore, so we only test full restore, not incremental restore. Again, Duplicacy was not only the fastest but also the most stable. The restore times of restic and Attic increased considerably for backups created later, with restic's performance deteriorating far more quickly. This is perhaps due to the fact that both restic and Attic group a number of chunks into a pack, so to restore a later backup one may need to unpack many packs belonging to earlier backups. In contrast, chunks in Duplicacy are independent entities and are never packed, so any backup can be quickly restored from chunks that compose that backup, without the need to retrieve data from other backups.

Backing up a VirtualBox virtual machine

The second test was targeted at the other end of the spectrum - a dataset with fewer but much larger files. Virtual machine files typically fall into this category. The particular dataset for this test is a VirtualBox virtual machine file. The base disk image is 64 bit CentOS 7, downloaded from http://www.osboxes.org/centos/. Its size is about 4 GB, still small compared to virtual machines that are actually being used everyday, but it is enough to quantify performance differences between these 4 backup tools.

The first backup was performed right after the virtual machine had been set up without installing any software. The second backup was performed after installing common developer tools using the command yum groupinstall 'Development Tools'. The third backup was performed after a power on immediately followed by a power off.

The following table lists the backup times by these 4 tools. With default settings, Duplicacy was generally slower than Attic. However, this is mainly because Attic does not compute file hashes, while Duplicacy does. For a fair comparison, an option was added to Duplicacy to disable file hash computation and that made Duplicacy slightly faster than Attic. This is not to say that Duplicacy should make this option the default. Although chunk hashes along can guarantee the integrity of backups, file hashes can be useful in many ways. For instance, file hashes enable users to quickly identify which files in existing backups are changed. They also allow third-party tools to compare files on disks to those in the backups. It is unlikely that Duplicacy will stop computing file hashes by default in favor of slight performance gains.

Surprisingly, for the third backup restic was the fastest. This can be explained partly by the lack of compression, partly by the high CPU usage.

Not surprisingly, duplicity is still the most storage efficient with restic being the worst:

A full restore was also performed for each backup. Again, not computing the file hash helped improve the performance, but at the risk of possible undetected data corruption.

Conclusion

The performances of 4 different backup tools on two publicly available datasets were compared. Duplicacy is clearly the top performer for the first dataset and as fast as Attic for the second if the file hash computation is disabled. However, it should be noted as both datasets are small and may be very different in nature from your data to be backed up. Therefore, I strongly encourage you to run your own experiments using scripts available in this github repository in order to determine which one is the best for you.