bitpocket is a small but smart script that does 2-way directory synchronization. It uses rsync to do efficient data transfer and tracks local file creation/removal to avoid known rsync problem when doing 2-way syncing with deletion.
bitpocket can use any server which you have ssh access to for its central storage. If you have gigabytes of free disk space on your hosting server you can finally make use of it.
Clone repository and symlink bitpocket bin to sth in your $PATH:
$ git clone git://github.com/ku1ik/bitpocket.git
$ ln -s `pwd`/bitpocket/bin/bitpocket ~/bin/bitpocket
Or download script and place it in a directory in your $PATH:
$ curl -sL https://raw.github.com/ku1ik/bitpocket/master/bin/bitpocket > ~/bin/bitpocket
$ chmod +x ~/bin/bitpocket
Create an empty directory on some host that will be the master copy of your files:
$ ssh [email protected]
$ mkdir ~/BitPocketMaster
On each machine you want to synchronize initialize an empty directory as your bitpocket:
$ mkdir ~/BitPocket
$ cd ~/BitPocket
$ bitpocket init [email protected] ~/BitPocketMaster
After installation, you use the bitpocket command for synchronization and other tasks.
Running bitpocket help will display the following message.
usage: bitpocket { init [<REMOTE_HOST>] <REMOTE_PATH>
| sync | help | pack | log | cron | list }
Available commands:
sync Run the sync process. If no command is specified, sync is run by
default.
init Initialize a new bitpocket folder. Requires path and optional
remote host params. Remote path must already exist.
pack Pack any existing (automatic) backups into a git repository.
cron Run sync optimized for cron, logging output to file instead of
stdout.
log Display the log generated by the cron command
list List all files in the sync set (honoring include/exclude/filter
config).
help Show this message.
Options:
-f, --force Clean up stale lock files automatically
-p, --pretend Don't really perform the sync or update the current
state. Instead, show what would be synchronized.
Note: All commands (apart from help), must be run in the root of a
new or existing bitpocket directory structure.
To synchronize your local slave with master just run bitpocket sync inside your bitpocket directory:
$ cd ~/BitPocket
$ bitpocket sync
Ensure that you run bitpocket at least once immediately after creating a new slave and before adding new files to the slave directory. If there are files in the master they will be pulled into the slave. You may then move files into your slave directory and they will be detected as added.
bitpocket does not include a full-fledged versioning system at the moment, but it does automatically create a local backup of any files before overwriting or deleting with changes from the BitPocketMaster. These backups are placed into a timestamped directory (one directory per sync). The path to this directory is:
.bitpocket/backups/YYYY-MM-DD.hhmmss
As these files accumulate, you may want to remove them, but you can also run
bitpocket pack to combine all the backup files into a git repository contained
within the .bitpocket directory:
$ cd ~/BitPocket
$ bitpocket pack
This requires an installation of git, but allows all the space saving advantages of git when making repeated changes to the same files.
There is a discussion about potential directions for versioning direction here: github.com/ku1ik/bitpocket/issues/15
If bitpocket is run with the cron parameter ( bitpocket cron ), it will perform a sync, but instead of showing the progress on stdout, it will redirect all output to a log file:
$ cd ~/BitPocket
$ bitpocket cron
As the name of this parameter implies, this is mainly useful when running bitpocket through the cron command. (See "Automatic sync with cron" for more information about how to configure this).
When running bitpocket in cron with bitpocket cron it will append its output
to .bitpocket/log file. You can review the tail end of an existing log file,
or watch live log as it is generated, with following command:
$ cd ~/BitPocket
$ bitpocket log
You may want to know which files will be synchronized before actually performing
the syncronization. You can verify which files are in the synchronization set
by running bitpocket list:
$ cd ~/BitPocket
$ bitpocket list
Note that this does not list changed files, it only lists all the local files that bitpocket will look at in determining which files to sync. Also, note that if there are new files in the master that will be added on a sync, they will not be included here. This command is only intended to verify which files are in the synchronization set. (See "Configuring file exclusion and inclusion" for information about how to control which files are in the synchronization set).
Add following line to your crontab to synchronize every 5 minutes:
*/5 * * * * cd ~/BitPocket && nice ~/bin/bitpocket cron
Note that cron usually has very limited environment and your ssh keys with passphrases won't work in cron jobs as ssh-agents/keyrings don't work there. Thus it's preferable to generate passphrase-less ssh key for bitpocket authentication:
$ cd ~/BitPocket
$ ssh-keygen -t rsa -C bitpocket-`hostname` -N '' -f .bitpocket/id_rsa
$ ssh-copy-id -i .bitpocket/id_rsa [email protected]
and uncomment line with RSYNC_SSH in .bitpocket/config file.
If you want some files to be ignored by bitpocket you can create .bitpocket/exclude file and list the paths there:
*.avi
jola
/misio.txt
*.avi and jola will be matched anywhere in path, misio.txt will be matched at bitpocket root dir ( ~/BitPocket/misio.txt ).
This exclude file is passed to rsync as --exclude-from argument, check man rsync for INCLUDE/EXCLUDE PATTERN RULES.
You can set up even more advanced exclusion/inclusion rules. In all, there there are three files that you can create to change this configuration:
.bitpocket/exclude
.bitpocket/include
.bitpocket/filter
Be aware that all the quirks from rsync exclusion/inclusion rules carry over
into bitpocket. If you decide that you need such advanced configuration, make
sure that you understand those rules very well, and consider double checking
them before syncing by running bitpocket list.
Both local and remote backups can be enabled or disabled in the configuration. By default, local backups are enabled and remote backups are not. Based on how intend to use bitpocket, you may wish to change this behavior. Add or change these lines in your .bitpocket/config file:
# BACKUPS=true
# REMOTE_BACKUPS=false
You can pass additional switches to rsync by setting RSYNC_OPTS in
.bitpocket/config file. Generated config file includes (commented out)
example setting for dereferencing symlinks:
# RSYNC_OPTS="-L"
Just uncomment it and change at will.
When syncing takes more than 10 seconds (SLOW_SYNC_TIME setting) bitpocket can fire off user provided command in background. This can be usefull to notify user about long sync happening, preventing him from turning off the machine during sync etc.
There are 3 settings that can be enabled in .bitpocket/config file:
# SLOW_SYNC_TIME=10
# SLOW_SYNC_START_CMD="notify-send 'BitPocket sync in progress...'"
# SLOW_SYNC_STOP_CMD="notify-send 'BitPocket sync finished'"
Just uncomment them and change at will.
You can show tray icon during long sync with traytor and following settings:
SLOW_SYNC_START_CMD='~/bin/traytor -t "BitPocket syncing..." -c "xdg-open ." .bitpocket/icons & echo $! >.bitpocket/traytor.pid'
SLOW_SYNC_STOP_CMD='kill `cat .bitpocket/traytor.pid`'
You can add a remote mount point check to ensure a remote path is available prior to the sync running. If the folder is not mounted on the remote server, then the sync will be aborted. The path must be an absolute path which is expected to be a mountpoint on the remote server. Add or change this lines in your .bitpocket/config file:
# REMOTE_MOUNTPOINT=/
- Marcin Kulik | @ku1ik | https://github.com/ku1ik | http://ku1ik.com/
- torfason | https://github.com/torfason