• Stars
    star
    306
  • Rank 136,456 (Top 3 %)
  • Language
    Python
  • License
    ISC License
  • Created almost 9 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Optional tools to help manage data in a mergerfs pool

mergerfs-tools

Optional tools to help manage data in a mergerfs pool.

INSTALL

All of these suplimental tools are self contained Python3 apps. Make sure you have Python 3 installed and either run make install or copy the file to /usr/local/bin or wherever you keep your binarys and make it executable (chmod +x).

TOOLS

mergerfs.ctl

A wrapper around the mergerfs xattr interface.

Download latest

$ mergerfs.ctl -h
usage: mergerfs.ctl [-h] [-m MOUNT] {add,remove,list,get,set,info} ...

positional arguments:
  {add,remove,list,get,set,info}

optional arguments:
  -h, --help            show this help message and exit
    -m MOUNT, --mount MOUNT
                            mergerfs mount to act on
$ mergerfs.ctl info
- mount: /storage
  version: 2.14.0
  pid: 1234
  srcmounts:
    - /mnt/drive0
    - /mnt/drive1
$ mergerfs.ctl -m /storage add path /mnt/drive2
$ mergerfs.ctl info
- mount: /storage
  version: 2.14.0
  pid: 1234
  srcmounts:
    - /mnt/drive0
    - /mnt/drive1
    - /mnt/drive2

mergerfs.fsck

Audits permissions and ownership of files and directories in a mergerfs mount and allows for manual and automatic fixing of them.

It's possible that files or directories can be duplicated across multiple drives and that their metadata become out of sync. Permissions, ownership, etc. This can cause some strange behavior depending on the mergerfs policies used. This tool helps find and fix those inconsistancies.

Download latest

$ mergerfs.fsck -h
usage: mergerfs.fsck [-h] [-v] [-s] [-f {manual,newest,nonroot}] dir

audit a mergerfs mount for inconsistencies

positional arguments:
  dir                   starting directory

  optional arguments:
    -h, --help            show this help message and exit
    -v, --verbose         print details of audit item
    -s, --size            only consider if the size is the same
    -f {manual,newest,nonroot}, --fix {manual,newest,nonroot}
                          fix policy
$ mergerfs.fsck -v -f manual /path/to/dir

mergerfs.dup

Duplicates files & directories across branches in a pool. The file selected for duplication is picked by the dup option. Files will be copied to drives with the most free space. Deleted from others if prune is enabled.

See usage for more. Run as root. Requires rsync to be installed.

Download latest

usage: mergerfs.dup [<options>] <dir>

Duplicate files & directories across multiple drives in a pool.
Will print out commands for inspection and out of band use.

positional arguments:
  dir                    starting directory

optional arguments:
  -c, --count=           Number of copies to create. (default: 2)
  -d, --dup=             Which file (if more than one exists) to choose to
                         duplicate. Each one falls back to `mergerfs` if
                         all files have the same value. (default: newest)
                         * newest   : file with largest mtime
                         * oldest   : file with smallest mtime
                         * smallest : file with smallest size
                         * largest  : file with largest size
                         * mergerfs : file chosen by mergerfs' getattr
  -p, --prune            Remove files above `count`. Without this enabled
                         it will update all existing files.
  -e, --execute          Execute `rsync` and `rm` commands. Not just
                         print them.
  -I, --include=         fnmatch compatible filter to include files.
                         Can be used multiple times.
  -E, --exclude=         fnmatch compatible filter to exclude files.
                         Can be used multiple times.

mergerfs.dedup

Finds and removes duplicate files across mergerfs pool's branches. Use the ignore, dedup, and strict options to target specific use cases.

Download latest

usage: mergerfs.dedup [<options>] <dir>

Remove duplicate files across branches of a mergerfs pool. Provides
multiple algos for determining which file to keep and what to skip.

positional arguments:
  dir                    Starting directory

optional arguments:
  -v, --verbose          Once to print `rm` commands
                         Twice for status info
                         Three for file info
  -i, --ignore=          Ignore files if... (default: none)
                         * same-size      : have the same size
                         * different-size : have different sizes
                         * same-time      : have the same mtime
                         * different-time : have different mtimes
                         * same-hash      : have the same md5sum
                         * different-hash : have different md5sums
  -d, --dedup=           What file to *keep* (default: newest)
                         * manual        : ask user
                         * oldest        : file with smallest mtime
                         * newest        : file with largest mtime
                         * largest       : file with largest size
                         * smallest      : file with smallest size
                         * mostfreespace : file on drive with most free space
  -s, --strict           Skip dedup if all files have same value.
                         Only applies to oldest, newest, largest, smallest.
  -e, --execute          Will not perform file removal without this.
  -I, --include=         fnmatch compatible filter to include files.
                         Can be used multiple times.
  -E, --exclude=         fnmatch compatible filter to exclude files.
                         Can be used multiple times.

# mergerfs.dedup /path/to/dir
# Total savings: 10.0GB

# mergerfs.dedup -e -d newest /path/to/dir
mergerfs.dedup -v -d newest /media/tmp/test
rm -vf /mnt/drive0/test/foo
rm -vf /mnt/drive1/test/foo
rm -vf /mnt/drive2/test/foo
rm -vf /mnt/drive3/test/foo
# Total savings: 10.0B

mergerfs.balance

Will move files from the most filled drive (percentage wise) to the least filled drive. Will do so till the most and least filled drives come within a user defined percentage range (defaults to 2%).

Run as root. Requires rsync to be installed.

Download latest

usage: mergerfs.balance [-h] [-p PERCENTAGE] [-i INCLUDE] [-e EXCLUDE]
                        [-I INCLUDEPATH] [-E EXCLUDEPATH] [-s EXCLUDELT]
                        [-S EXCLUDEGT]
                        dir

balance files on a mergerfs mount based on percentage drive filled

positional arguments:
  dir                   starting directory

optional arguments:
  -h, --help            show this help message and exit
  -p PERCENTAGE         percentage range of freespace (default 2.0)
  -i INCLUDE, --include INCLUDE
                        fnmatch compatible file filter (can use multiple
                        times)
  -e EXCLUDE, --exclude EXCLUDE
                        fnmatch compatible file filter (can use multiple
                        times)
  -I INCLUDEPATH, --include-path INCLUDEPATH
                        fnmatch compatible path filter (can use multiple
                        times)
  -E EXCLUDEPATH, --exclude-path EXCLUDEPATH
                        fnmatch compatible path filter (can use multiple
                        times)
  -s EXCLUDELT          exclude files smaller than <int>[KMGT] bytes
  -S EXCLUDEGT          exclude files larger than <int>[KMGT] bytes

# mergerfs.balance /media
from: /mnt/drive1/foo/bar
to:   /mnt/drive2/foo/bar
rsync ...

mergerfs.consolidate

Consolidate files in a single mergerfs directory onto a single drive, recursively. This does NOT move all files at and below that directory to 1 drive. If you want to move data between drives simply use normal rsync or similar. This tool is only useful in niche usecases where the person wants to colocate files of their TV, music, etc. files onto a single drive after the fact. If you really wanted that you should probably use path preservation. For most people there is only downsides to using path preservation or colocating files.

Run as root. Requires rsync to be installed.

Download latest

usage: mergerfs.consolidate [<options>] <dir>

positional arguments:
  dir                    starting directory

optional arguments:
  -m, --max-files=       Skip directories with more than N files.
                         (default: 256)
  -M, --max-size=        Skip directories with files adding up to more
                         than N. (default: 16G)
  -I, --include-path=    fnmatch compatible path include filter.
                         Can be used multiple times.
  -E, --exclude-path=    fnmatch compatible path exclude filter.
                         Can be used multiple times.
  -e, --execute          Execute `rsync` commands as well as print them.
  -h, --help             Print this help.

SUPPORT

Contact / Issue submission

Support development

This software is free to use and released under a very liberal license. That said if you like this software and would like to support its development donations are welcome.

  • PayPal: https://paypal.me/trapexit
  • GitHub Sponsors: https://github.com/sponsors/trapexit
  • Patreon: https://www.patreon.com/trapexit
  • SubscribeStar: https://www.subscribestar.com/trapexit
  • Ko-Fi: https://ko-fi.com/trapexit
  • Open Collective: https://opencollective.com/trapexit
  • Bitcoin (BTC): bc1qjwlywkqxgrxql3m7a7fvcsf3z3t98jvtekqp2j
  • Bitcoin Cash (BCH): qrvymmkvuk7703m7cx0pqxc3mz4mmsn6ngn9xw52kc
  • Bitcoin SV (BSV): 1FkFuxRtt3f8LbkpeUKRZq7gKJFzGSGgZV
  • Bitcoin Gold (BTG): Gfk8QbMJFgpMTcY7uB63axy6HU7uTPPWNj
  • Basic Attention Token (BAT): 0x6241857fa5fb7667FB7a792b13E83fDEabe96f7F
  • Chainlink (LINK): 0x6241857fa5fb7667FB7a792b13E83fDEabe96f7F
  • Dash (DASH): Xu2U3Nd3G4hM5TRQUBcP4DHJFzXH93xB84
  • Dogecoin (DOGE): DGFBPsRBYL8wHbgnvKbYkVn5FvAe854p1c
  • Ethereum (ETH): 0x6241857fa5fb7667FB7a792b13E83fDEabe96f7F
  • Filecoin (FIL): f1wpypkjcluufzo74yha7p67nbxepzizlroockgcy
  • LBRY Credits (LBC): bFusyoZPkSuzM2Pr8mcthgvkymaosJZt5r
  • Litecoin (LTC): LfL7jLNYuVpy7v5TyRyc3yRZ2uhqc4UoR3
  • Monero (XMR): 45BBZMrJwPSaFwSoqLVNEggWR2BJJsXxz7bNz8FXnnFo3GyhVJFSCrCFSS7zYwDa9r1TmFmGMxQ2HTntuc11yZ9q1LeCE8f
  • Tezos (XTZ): tz1ZxerkbbALsuU9XGV9K9fFpuLWnKAGfc1C
  • Zcash (ZEC): t1bjbVBK7tx9EGBrnD2wDfjGV9yZrcyfMmr
  • Other crypto currencies: contact me for address

LINKS

More Repositories

1

mergerfs

a featureful union filesystem
C++
3,401
star
2

backup-and-recovery-howtos

Guides to setting up a media storage system, backing it up, and recovering from failures
311
star
3

scorch

Silent CORruption CHecker and filesystem audit tool
Python
168
star
4

bbf

bad block finder and repair tool
C++
96
star
5

chip-8_documentation

A collection of documentation on the CHIP-8 and related
57
star
6

3do-bios

Collection of 3DO BIOSes and font roms
41
star
7

portfolio_os

3DO M1 Opera Portfolio Operating System source code
C
34
star
8

3do-devkit

A new development environment for the 3DO Opera game console
C
30
star
9

portfolio_os_m2

3DO M2 Portfolio OS v3.0 and tooling source dump
C
23
star
10

chipce8

CHIP-8 emulator for the PC Engine / TurboGrafx-16
Assembly
19
star
11

wiiqt

collection of wii-related Qt/C++ stuff
C++
15
star
12

3DO-information

Collection of information on the 3DO
HTML
15
star
13

icebreaker2

Source to the 3DO game Icebreaker 2
C++
9
star
14

icebreaker

Source to the 3DO game Icebreaker
C++
7
star
15

dmcache

a tool to help setup dm-cache for drives not controlled by LVM2
Python
7
star
16

mhddfs

FUSE filesystem which merges paths together
5
star
17

crc32b

A simple C crc32b implementation
C
5
star
18

3it

An all in one 3DO image conversion tool
C++
4
star
19

3do-star-control-2

Source code to the original 3DO version of Star Control 2
C
3
star
20

support

Ways to support my work
3
star
21

modbin

A replacement modbin tool for 3DO software development
C
3
star
22

3dt

An all in one 3DO disc tool
C++
3
star
23

crc32c

A simple C crc32c implementation
C
2
star
24

4DO

A copy of the 4DO subversion repo for reference
C#
2
star
25

rawdrv

C
2
star
26

wandb

Write Almost Never (Erlang) DataBase
Erlang
2
star
27

hge

HNC Game Engine
C
2
star
28

3do-cpplib

C++
1
star
29

bfif

brute force image finder
C++
1
star
30

tag

Python
1
star
31

jdt

json object data type for erlang
Erlang
1
star
32

dockerfile-debuild

docker container which helps create debootstrap'ed containers
Shell
1
star
33

freedo

A copy of the freedo subversion repo for reference
C++
1
star
34

3at

3DO Audio Tool
C++
1
star
35

libportfolio

A library which mimics the 3DO Company's Portfolio OS APIs to allow cross platform development
1
star
36

3doplay

A copy of the 3doplay subversion repo for reference
C++
1
star