fast-archiver

fast-archiver is a command-line tool for archiving directories, and restoring those archives written in [Go](http://golang.org).

fast-archiver uses a few techniques to try to be more efficient than traditional tools:

1. It reads a number of files concurrently and then serializes the output. Most other tools use sequential file processing, where operations like open(), lstat(), and close() can cause a lot of overhead when reading huge numbers of small files. Making these operations concurrent means that the tool is more often reading and writing data than you would be otherwise.
2. It begins archiving files before it has completed reading the directory entries that it is archiving, allowing for a fast startup time compared to tools that first create an inventory of files to transfer.

How Fast?

On a test workload of 2,089,214 files representing a total of 90 GiB of data, fast-archiver was compared with tar and rsync for reading data files and transfering them over a network. The test scenario was a PostgreSQL database, with many of the files being small, 8-24kiB in size.

Compared with tar, fast-archiver took 33% of the execution time (27m 38s vs. 1h 23m 23s) to read the test workload and output the archive to /dev/null. The tar output had to be redirected through cat to create a comparable scenario, because tar recognized /dev/null and shortcuts the actual data file reading and writing. Here's the raw timing output for some hard data:

$ time fast-archiver -c -o /dev/null /db/data
skipping symbolic link /db/data/pg_xlog
1008.92user 663.00system 27:38.27elapsed 100%CPU (0avgtext+0avgdata 24352maxresident)k
0inputs+0outputs (0major+1732minor)pagefaults 0swaps

$ time tar -cf - /db/data | cat > /dev/null
tar: Removing leading `/' from member names
tar: /db/data/base/16408/12445.2: file changed as we read it
tar: /db/data/base/16408/12464: file changed as we read it
32.68user 375.19system 1:23:23elapsed 8%CPU (0avgtext+0avgdata 81744maxresident)k
0inputs+0outputs (0major+5163minor)pagefaults 0swaps

Compared with rsync, fast-archiver piped over ssh can transfer the database from one machine to another in 1h 30m, vs. rsync in 3h.

These huge reductions in time may not be typical, but they happen to be the workload that fast-archiver was designed for.

Examples

Creates an archive (-c) reading the directory target1, and redirects the archive to the file named target1.fast-archive:

fast-archiver -c target1 > target1.fast-archive
fast-archiver -c -o target1.fast-archive target1

Extracts the archive target1.fast-archive into the current directory:

fast-archiver -x < target1.fast-archive
fast-archiver -x -i target1.fast-archive

Creates a fast-archive remotely, and restores it locally, piping the data through ssh:

ssh [email protected] "cd /db; fast-archive -c data --exclude=data/\*.pid" | fast-archiver -x

Installation

The fast-archiver repository contains both a command-line tool (at the root) and a package called falib which contains the archive reading and writing code. To make the build work correctly with both the library and the command-line tool, it's necessary to setup the correct GOPATH and directory references.

Here's a quick set of steps to setup the build:

• Install [Go](http://golang.org).
• Setup $GOPATH, for example: export GOPATH=$HOME/go-projects. Probably better to set it up in your .bash_aliases.
• go get -u github.com/replicon/fast-archiver

_or_

• go get -d github.com/replicon/fast-archiver && $GOPATH/src/github.com/replicon/fast-archiver/build.sh

Command-line arguments

Create-mode only

Extract-mode only