arqinator

Cross-platform restoration of Arq backups. Download arqinator binaries from the releases page.

Please explicitly test listing directories and recovering files before assuming it will work for you.

Features

Cross-platform support for Windows, Mac OS X, and Linux.
- Tested on Windows 7 32-bit, Mac OS X Yosemite 10.10.5 64-bit, and Ubuntu 14.04 LTS 64-bit
Deployable as a single executable file, no external dependencies required.
List all backup sets in a particular backup type using list-backup-sets
List the contents of directories or information about a file using list-directory-contents
Recover single files, sub-folders and their contents, or entire backup sets, using recover

Limitations

Currently only supports the following backup types:
- S3
- Google Cloud Storage
- SFTP (only unencrypted SSH private keys)
arqinator has been tested on backups created by Arq 4.14.5 only. I do not know if arqinator works on previous versions of Arq. I'm doubtful that arqinator will work on previous major versions of Arq (i.e. 3 or 2).
Files are downloaded in serial, might take a while to recover a lot of data.

TODO

soft links aren't supported
do you want to 'chown' files and folders to the UID/GID backed up?
- maybe offer as an option?
support multiple encryption passwords for multiple accounts
- maybe have a text-file based configuration?
support all backup types possible with Arq, start with Dropbox.
explicitly check SHA1 hashes of blobs to confirm no corruption.
Files are downloaded in serial, should do it in parallel.

Testing done so far

I've successfully listed backup sets, listed directory contents, and downloaded specific files and subfolders on:

Backups created by Mac OS X 10.10.5 onto S3, Google Cloud Storage, and SFTP, and then retrieved back onto the same Mac OS X host.
Backups created by Windows 7 onto S3, and then retrieved back onto the same Windows 7 host and a different Ubuntu 14.04 host.

Getting started

0. Prerequisites

How to set environment variables on:

Download arqinator binaries from the releases page.

1. Configure Credentials

For all backup methods, set the following environment variable to the password you're using for encrypting your Arq backups:

ARQ_ENCRYPTION_PASSWORD=mysecretpassword

Only Arq backup sets encrypted by this password will be visible to you when you run list-backup-sets. If the password is incorrect they will appear to be invisible. Run with --verbose if you want more information.

S3

Set the following environment variables depending on which backup method you are using:

AWS_ACCESS_KEY_ID=AKID1234567890
AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY

Google Cloud Storage

Go to the Google Developers Console: https://console.developers.google.com//project/_/apiui/credential
Download a JSON private key file using: https://goo.gl/SK5Rb7
To determine your project ID, click on the gear in the top right, and click "Project information"
To determine your bucket name, on the left navigation pane go to Storage - Cloud Storage - Browser, then find your bucket.

SFTP

The preferred way of using SFTP is to use password-less SSH login by putting your SSH public key into the SFTP server's authorized_keys. When you do so currently arqinator only supports unencrypted SSH private keys. However if you want to log into the SFTP server using a plaintext password you can set the following environment variable:

ARQ_SFTP_PASSWORD=my-sftp-password

2. List backup sets

Note that there will be a difference between how paths appear on Windows and Linux/Mac:

S3, Mac

$ arqinator \
    --backup-type s3 \
    --s3-region us-west-2 \
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 \
    list-backup-sets

ArqBackupSet
    UUID 98DB38F8-B9C6-4296-9385-3C1BF858ED5D
    ComputerName Mill
    UserName ai
    Folders
        LocalPath /Users/ai
        UUID 8D4FAD2A-9E08-46F7-829D-E9601A65455D

S3, Windows

arqinator ^
    --backup-type s3 ^
    --s3-region us-west-2 ^
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 ^
    list-backup-sets

ArqBackupSet
    UUID E7CFDEED-AB08-4970-A377-78F8313AC39C
    ComputerName THE_RAIN
    UserName SYSTEM
    Folders
        LocalPath /C/Users/username/Downloads/apsw-3.7.15.1-r1
        UUID FE8BE3EE-B63B-4D1F-A7E9-6707297823B5

Google Cloud Storage, Mac

$ arqinator \
    --backup-type googlecloudstorage \
    --gcs-json-private-key-filepath /Users/ai/keys/gcs.json \
    --gcs-project-id midyear-courage-109219 \
    --gcs-bucket-name arq-560729839528 \
    list-backup-sets

ArqBackupSet
    UUID 7FE8D069-B218-4E17-8E58-0C7FAF8CFAFC
    ComputerName Mill
    UserName ai
    Folders
        LocalPath /Users/ai/temp/apsw-3.7.15.1-r1
        UUID E6F4BC5E-B21F-4828-ADCC-8521F9DBC4C9

SFTP, Mac, verbose mode

$ arqinator \
      --backup-type sftp \
      --sftp-host asims-mac-mini.local \
      --sftp-port 22 \
      --sftp-remote-path /Users/aihsan/arq_backup \
      --sftp-username aihsan \
      --sftp-private-key-filepath /Users/ai/.ssh/id_rsa \
      --verbose \
      list-backup-sets

ArqBackupSet
    UUID 76A4E004-FCB9-47D7-B080-16A236439F5C
    ComputerName Mill
    UserName ai
    Folders
        LocalPath /Users/ai/temp/apsw-3.7.15.1-r1
        UUID 1BFC0BD6-9877-4562-9692-05EB3A5EF20C

3. List directory contents of backups

Again note that paths for Windows will look a little unusual.

S3, Mac

$ arqinator \
    --backup-type s3 \
    --s3-region us-west-2 \
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 \
    list-directory-contents \
    --backup-set-uuid 98DB38F8-B9C6-4296-9385-3C1BF858ED5D \
    --folder-uuid 8D4FAD2A-9E08-46F7-829D-E9601A65455D \
    --path /Users/ai/.ssh

-rwx------	2013-06-29 05:43:06 -0700 PDT	1.7kB	ai_keypair_3.pem
-rw-r--r--	2013-06-28 13:44:47 -0700 PDT	0B	    config
-rw-------	2014-12-31 19:41:47 -0800 PST	1.7kB	digitalocean
-rw-r--r--	2014-12-31 19:41:47 -0800 PST	402B	digitalocean.pub
-rw-------	2012-04-11 12:49:10 -0700 PDT	1.7kB	id_rsa
-rw-r--r--	2012-04-14 13:16:50 -0700 PDT	396B	id_rsa.pub
-rw-------	2014-05-17 05:04:13 -0700 PDT	1.7kB	interview-ec2.pem
-rw-r--r--	2015-09-08 14:09:39 -0700 PDT	18kB	known_hosts

S3, Windows

arqinator ^
    --backup-type s3 ^
    --s3-region us-west-2 ^
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 ^
    list-directory-contents ^
    --backup-set-uuid E7CFDEED-AB08-4970-A377-78F8313AC39C ^
    --folder-uuid FE8BE3EE-B63B-4D1F-A7E9-6707297823B5 ^
    --path /C/Users/username/Downloads/apsw-3.7.15.1-r1

----------      2015-10-08 20:36:21 +0100 BST   6.1kB   .DS_Store
d---------      1970-01-01 00:00:00 +0000 GMT   9.3MB   build
----------      2012-12-22 09:05:48 +0000 GMT   7.1kB   checksums
d---------      1970-01-01 00:00:00 +0000 GMT   1.6MB   doc
----------      2010-01-05 22:53:28 +0000 GMT   699B    MANIFEST.in
----------      2009-09-13 05:50:06 +0100 BST   4.3kB   mingwsetup.bat
----------      2012-12-22 10:57:02 +0000 GMT   1.0kB   PKG-INFO
----------      2012-12-26 18:01:38 +0000 GMT   33kB    setup.py
d---------      1970-01-01 00:00:00 +0000 GMT   7.6MB   sqlite3
d---------      1970-01-01 00:00:00 +0000 GMT   554kB   src
----------      2012-12-26 18:05:00 +0000 GMT   29kB    testdbx
----------      2012-12-26 18:05:00 +0000 GMT   12kB    testdbx-journal
----------      2012-12-22 09:36:24 +0000 GMT   340kB   tests.py
----------      2012-12-26 18:03:31 +0000 GMT   285kB   tests.pyc
d---------      1970-01-01 00:00:00 +0000 GMT   168kB   tools

Google Cloud Storage, Mac

$ arqinator \
    --backup-type googlecloudstorage \
    --gcs-json-private-key-filepath /Users/ai/keys/gcs.json \
    --gcs-project-id midyear-courage-109219 \
    --gcs-bucket-name arq-560729839528 \
    list-directory-contents \
    --backup-set-uuid 7FE8D069-B218-4E17-8E58-0C7FAF8CFAFC \
    --folder-uuid E6F4BC5E-B21F-4828-ADCC-8521F9DBC4C9 \
    --path /Users/ai/temp/apsw-3.7.15.1-r1

-rw-r--r--	2015-10-08 12:36:21 -0700 PDT	6.1kB	.DS_Store
-rw-r--r--	2010-01-05 14:53:28 -0800 PST	699B	MANIFEST.in
-rw-rw-r--	2012-12-22 02:57:02 -0800 PST	1.0kB	PKG-INFO
drwxr-xr-x	2015-10-08 12:36:21 -0700 PDT	9.3MB	build
-rw-r--r--	2012-12-22 01:05:48 -0800 PST	7.1kB	checksums
drwxr-xr-x	2012-12-26 09:56:54 -0800 PST	1.6MB	doc
-rw-r--r--	2009-09-12 21:50:06 -0700 PDT	4.3kB	mingwsetup.bat
-rw-rw-r--	2012-12-26 10:01:38 -0800 PST	33kB	setup.py
drwxr-xr-x	2012-12-26 10:02:47 -0800 PST	7.6MB	sqlite3
drwxr-xr-x	2012-12-26 10:02:47 -0800 PST	554kB	src
-rw-r--r--	2012-12-26 10:05:00 -0800 PST	29kB	testdbx
-rw-r--r--	2012-12-26 10:05:00 -0800 PST	12kB	testdbx-journal
-rw-r--r--	2012-12-22 01:36:24 -0800 PST	340kB	tests.py
-rw-r--r--	2012-12-26 10:03:31 -0800 PST	285kB	tests.pyc
drwxr-xr-x	2012-12-26 09:56:54 -0800 PST	168kB	tools

SFTP, Mac

arqinator \
    --backup-type sftp \
    --sftp-host asims-mac-mini.local \
    --sftp-port 22 \
    --sftp-remote-path /Users/aihsan/arq_backup \
    --sftp-username aihsan \
    --sftp-private-key-filepath /Users/ai/.ssh/id_rsa \
    --verbose
    list-directory-contents \
    --backup-set-uuid 76A4E004-FCB9-47D7-B080-16A236439F5C \
    --folder-uuid 1BFC0BD6-9877-4562-9692-05EB3A5EF20C \
    --path /Users/ai/temp/apsw-3.7.15.1-r1/build

-rw-r--r--	2015-10-09 09:50:34 -0700 PDT	10kB	.DS_Store
drwxr-xr-x	2012-12-26 10:03:31 -0800 PST	1.5MB	lib.macosx-10.4-x86_64-2.7
drwxr-xr-x	2015-10-09 09:50:34 -0700 PDT	7.8MB	temp.macosx-10.4-x86_64-2.7

4. Restore

You can restore either individual files or entire folders. Note that you need to use a Linux-like directory path for Windows backups:

S3, Mac, recovering a single file

$ arqinator \
    --backup-type s3 \
    --s3-region us-west-2 \
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 \
    recover \
    --backup-set-uuid 98DB38F8-B9C6-4296-9385-3C1BF858ED5D \
    --folder-uuid 8D4FAD2A-9E08-46F7-829D-E9601A65455D \
    --source-path /Users/ai/output.txt \
    --destination-path /Users/ai/temp/output.txt

S3, Windows, recovering an entire folder in verbose mode

arqinator ^
    --backup-type s3 ^
    --s3-region us-west-2 ^
    --s3-bucket-name arq-akiajmthnhpkz2ixzrxq-us-west-2 ^
    --verbose ^
    recover ^
    --backup-set-uuid E7CFDEED-AB08-4970-A377-78F8313AC39C ^
    --folder-uuid FE8BE3EE-B63B-4D1F-A7E9-6707297823B5 ^
    --source-path /C/Users/username/Downloads/apsw-3.7.15.1-r1/tools ^
    --destination-path /C/temp/tools

Google Cloud Storage, Mac, recovering a single file

$ arqinator \
    --backup-type googlecloudstorage \
    --gcs-json-private-key-filepath /Users/ai/keys/gcs.json \
    --gcs-project-id midyear-courage-109219 \
    --gcs-bucket-name arq-560729839528 \
    recover \
    --backup-set-uuid 7FE8D069-B218-4E17-8E58-0C7FAF8CFAFC \
    --folder-uuid E6F4BC5E-B21F-4828-ADCC-8521F9DBC4C9 \
    --source-path /Users/ai/temp/apsw-3.7.15.1-r1/PKG-INFO \
    --destination-path /Users/ai/temp/PKG-INFO

SFTP, Mac, recovering a folder

$ arqinator \
    --backup-type sftp \
    --sftp-host asims-mac-mini.local \
    --sftp-port 22 \
    --sftp-remote-path /Users/aihsan/arq_backup \
    --sftp-username aihsan \
    --sftp-private-key-filepath /Users/ai/.ssh/id_rsa \
    recover \
    --backup-set-uuid 76A4E004-FCB9-47D7-B080-16A236439F5C \
    --folder-uuid 1BFC0BD6-9877-4562-9692-05EB3A5EF20C \
    --source-path /Users/ai/temp/apsw-3.7.15.1-r1/build \
    --destination-path /Users/ai/temp/foobar

asimihsan/arqinator

asimihsan

Reviews

Repository Details

arqinator

Features

Limitations

TODO

Testing done so far

Getting started

0. Prerequisites

1. Configure Credentials

S3

Google Cloud Storage

SFTP

2. List backup sets

S3, Mac

S3, Windows

Google Cloud Storage, Mac

SFTP, Mac, verbose mode

3. List directory contents of backups

S3, Mac

S3, Windows

Google Cloud Storage, Mac

SFTP, Mac

4. Restore

S3, Mac, recovering a single file

S3, Windows, recovering an entire folder in verbose mode

Google Cloud Storage, Mac, recovering a single file

SFTP, Mac, recovering a folder

More Repositories