Linux Time Machine
Install | Uninstall | TL;DR | Features | How does it work | Restore | Retention | Usage | FAQ | Disclaimer | License
timemachine is a tiny and stable KISS driven and POSIX compliant script that mimics the behavior of OSX's timemachine.
It uses rsync to incrementally back up your data to a different
directory, hard disk or remote server via SSH. All operations are incremental, atomic and automatically resumable.
By default it uses the rsync options: --recursive, --perms, --owner, --group, --times and --links.
In case your target filesystem does not support any of those options or you cannot use them due
to missing permission, you can explicitly disable them via --no-perms, --no-owner, --no-group, --no-times, and --copy-links.
See FAQ for examples.
Motivation
The goal of this project is to have a cross-operating system and minimal as possible backup script that can be easily reviewed by anyone without great effort. Additionally it should provide one task only and do it well without the need of external requirements and only rely on default installed tools.
🎉 Install
sudo make install🚫 Uninstall
sudo make uninstall☕ TL;DR
Using POSIX.1-2008 argument syntax:
# Recursive, incremental and atomic backup (locally)
$ timemachine /source/dir /target/dir
# Recursive, incremental and atomic backup (via ssh)
$ timemachine /source/dir user@host:target/dir
# Recursive, incremental and atomic backup (via ssh with non-standard port)
$ timemachine --port 10000 /source/dir user@host:target/dir
# Append rsync options
$ timemachine /source/dir /target/dir -- --specials --progress
$ timemachine /source/dir /target/dir -- --specials --no-perms
$ timemachine /source/dir /target/dir -- --archive --progress
# Make the timemachine script be more verbose
$ timemachine -v /source/dir /target/dir
$ timemachine --verbose /source/dir /target/dir
# Make the timemachine script be even more verbose
$ timemachine -d /source/dir /target/dir
$ timemachine --debug /source/dir /target/dir
# Make the timemachine script and rsync more verbose
$ timemachine -v /source/dir /target/dir -- --verbose
$ timemachine --verbose /source/dir /target/dir -- --verbose⭐ Features
| Feature | Description |
|---|---|
| SSH or local | Local backups as well as backups via SSH are supported. |
| Incremental | Backups are always done incrementally using rsync's ability to hardlink to previous backup directories. You can nevertheless always see the full backup on the file system of any incrementally made backup without having to generate it. This will also be true when deleting any of the previously created backup directories. See the Backups section for how this is achieved via rsync. Incremental Backups also mean that only the changes on your source, compared to what is already on the target, have to be backed up. This will save you time as well as disk space on the target disk. |
| Partial | When backing up, files are transmitted partially, so in case a 2GB movie file backup is interrupted the next run will pick up exactly where it left off at that file and will not start to copy it from scratch. |
| Resumable | Not only is this script keeping partial files, but also the whole backup run is also resumable. Whenever there is an unfinished backup and you start timemachine again, it will automatically resume it. It will resume any previously failed backup as long as it finally succeeds. |
| Atomic [1] | The whole backup procedure is atomic. Only if and when the backup procedure succeeds, it is then properly named and symlinked. Any non-successful backup directory is either waiting to be resumed or to be deleted. |
- [1] The backup process is atomic, but not the backup itself.
rsynccopies files as it finds them and in the meantime there could already be changes on the source. To achieve an atomic backup, either back up from a read-only volume or from a snapshot.
ℹ️ How does it work?
Directory structure
The following directory structure will be created:
$ tree -L 1 /my/backup/folder
.
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/current will always link to the latest created backup.
All backups are incremental except the first created one.
You can nevertheless safely remove all previous folders and the remaining folders will still have all of their content.
Backup strategy
Except for the first one, backups are always and automatically done incrementally,
so the least amount of space is consumed.
Due to rsync's ability, every directory will still contain all files, even though they are just
incremental backups. This is archived via hardlinks.
$ du -hd1 .
497M ./2018-01-06__18-43-30
24K ./2018-01-06__18-44-23
24K ./2018-01-06__18-50-44
24K ./2018-01-06__18-50-52
497M .You can also safely delete the initial full backup directory without having to worry about losing any of your full backup data:
$ rm -rf ./2018-01-06__18-43-30
$ du -hd1 .
497M ./2018-01-06__18-44-23
24K ./2018-01-06__18-50-44
24K ./2018-01-06__18-50-52
497M .rsync and hardlinks are magic :-)
Failure handling and resume
In case the timemachine script aborts (self-triggered, disk unavailable or for any other reason)
you can simply run it again to automatically resume the last failed run.
This is due to the fact that the backup process is atomic. During a non-complete run,
all data will be stored in a directory named .inprogress/. This will hold all already
transferred data and will be picked up during the next run.
Once the backup is complete, it will be renamed and symlinked to current.
$ tree -a -L 1 /my/backup/folder
.
├── .inprogress/
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/♻️ Restore
No special software is required to restore your data. Backed up files can be easily browsed and thus copied back to where you need them. Recall the backup directory structure:
$ tree -L 1 /my/backup/folder
.
├── 2018-01-06__18-43-30/
├── 2018-01-06__18-44-23/
├── 2018-01-06__18-50-44/
├── 2018-01-06__18-50-52/
└── current -> 2018-01-06__18-50-52/Chose a backup directory and simply copy them to where you need it:
# Test it out in dry run mode before applying
$ rsync --archive --progress --dry-run /my/backup/folder/2018-01-06__18-50-52/ /src/
# Apply restoration
$ rsync --archive --progress /my/backup/folder/2018-01-06__18-50-52/ /src/🔁 Retention
As decribed above this project is KISS driven and only tries to do one job: back up your data.
Retention is a delicate topic as you want to be sure that data is removed as intended. For this there are already well-established tools that do an excellent job and have proven themselves over time: tmpreaper and tmpwatch.
🔒 Reliability
The script is written and maintained with maximum care. In order to retain a reliable and stable backup solution, a lot of effort goes into a vast amount of integration and regression tests. These tests not only give you measurable confidence, but also help new contributors to not accidentally introduce new or old bugs.
💻 Usage
Available options
$ timemachine -h
Usage: timemachine [-vd] <source> <dest> -- [rsync opts]
timemachine [-vdpi] <source> <host>:<dest> -- [rsync opts]
timemachine [-vdpi] <source> <user>@<host>:<dest> -- [rsync opts]
timemachine [-vdpi] <source> <ssh-alias>:<dest> -- [rsync opts]
timemachine [-vdpi] <host>:<source> <dest> -- [rsync opts]
timemachine [-vdpi] <user>@<host>:<source> <dest> -- [rsync opts]
timemachine [-vdpi] <ssh-alias>:<source> <dest> -- [rsync opts]
timemachine -V, --version
timemachine -h, --help
This shell script mimics the behavior of OSX's timemachine.
It uses rsync to incrementally back up your data to a different directory or remote server via SSH.
All operations are incremental, atomic and automatically resumable.
By default it uses --recursive --perms --owner --group --times --links.
In case your target filesystem does not support any of those options, you can explicitly
disable those options via --no-perms --no-owner --no-group --no-times and --copy-links.
Required arguments:
<source> Local source directory
<dest> Local destination directory.
<host>:<dest> SSH host and source/destination directory on server
<user>@<host>:<dest> SSH user, SSH host and source/destination directory on server
<ssh-alias>:<dest> SSH alias (defined in ~/.ssh/config) and source/destination directory on server
Options:
-p, --port Specify alternative SSH port for remote backups if it is not 22.
-i, --identity Specify path to SSH key.
-v, --verbose Be verbose.
-d, --debug Be even more verbose.
Misc Options:
-V, --version Print version information and exit
-h, --help Show this help screen
Examples:
Simply back up one directory recursively
timemachine /home/user /data
Do the same, but be verbose
timemachine -v /home/user /data
Append rsync options and be very verbose
timemachine -d /home/user /data -- --progress --verbose
Log to file
timemachine -v /home/user /data > /var/log/timemachine.log 2> /var/log/timemachine.err
Documentation:
View more examples at: https://github.com/cytopia/linux-timemachine
Use with cron
The following can be used as an example crontab entry. It assumes you have an external disk (NFS, USB, etc..) that mounts at /backup. Before adding the crontab entry, ensure the filesystem in /backup is mounted and use:
$ touch /backup/mountedThis guards against accidentally backing up to an unmounted directory
Next, add the following to crontab using crontab -e as whichever user you intend to run the backup script as. You may need to place this in the root crontab if you are backing up sensitive files that only root can read
0 2 * * * if [[ -e /backup/mounted ]]; then /usr/local/bin/timemachine /home/someuser /backup; fiThis will cause timemachine to run at 2AM once per day. Since timemachine keeps track of backups with granularity up to the hour, minute and second, you could have it run more than once per day if you want backups to run more often.
💡 FAQ
Should I add trailing directory slashes (/)?
Trailing directory slashes only matter for the source directory and will not make a difference if added to the destination directory.
# The following backs up the contents of the src directory
$ timemachine src/ dst/
$ tree -L 2 /dst
.
├── 2018-01-06__18-43-30/
│ └── file.txt
└── current -> 2018-01-06__18-43-30/# The following backs up the the src directory itself
$ timemachine src dst/
$ tree -L 3 /dst
.
├── 2018-01-06__18-43-30/
│ └── src
│ └── file.txt
└── current -> 2018-01-06__18-43-30/How to dry-run the backup?
$ timemachine src/ dst/ -- --dry-runHow to use a non-standard SSH port?
$ timemachine --port 1337 src/ user@host:dst/How to use SSH aliases from ~/.ssh/config?
$ cat ~/.ssh/config
Host my-ssh-alias
HostName 192.168.0.1
Port 1234
User john
IdentityFile ~/.ssh/id_rsa__alternative_key$ timemachine src/ my-ssh-alias:dst/How to speed up remote backups?
# With this option, rsync compresses the file data as it is sent to the des‐
# tination machine, which reduces the amount of data being transmitted
$ timemachine src/ user@host:dst/ --compressHow to preserve ACLs?
$ timemachine src/ dst/ -- --aclsHow to preserve extended attributes?
$ timemachine src/ dst/ -- --xattrsHow to disable preserving file and directory permissions?
$ timemachine src/ dst/ -- --no-owner --no-permsHow to disable preserving modification time?
$ timemachine src/ dst/ -- --no-timesHow to copy the content instead of a symlink?
# This is useful in case your file system does not support symlinks.
# It is recommended to read rsync man page about symlinks to be sure
# about what you are doing
$ timemachine src/ dst/ -- --copy-links --safe-links
$ timemachine src/ dst/ -- --copy-links --safe-links --keep-dirlinksHow to ensure all files in the back up have the ownership of current user?
# Regardless of who owns the files, ensure the backup has uid and gid of current user
# This will only work if you have read-permission on all files.
$ timemachine src/ dst/ -- --no-owner --no-group
# If you do not have permission to read all files, you require sudo or root permission.
# The following will instruct rsync to ensure the backed up data has the uid and gid
# of the desired user.
$ sudo timemachine src/ dst/ -- --chown=<user>:<group>❗ Disclaimer
Backups are one of the most important things. We all care about our data and want it to be safe, so do not blindly trust scripts when it comes to backups! Do review the code, it is not too complex and kept as short as possible. Have a look at how much effort goes into the integration tests to provide measurable stability.
Learn about rsync it is a very powerful tool and you might even be able to just use this for backups.
There are many other backup solutions out there that might be a better fit for your needs. Do your own research, look at GitHub issues, source code, integration tests and try them out as well.
Contributing
See Contributing guidelines to help to improve this project.
📄 License
Copyright (c) 2017 cytopia