Hardware Target Game Database
The Hardware Target Game Database is an archival research initiative with the goal of allowing users to build real-hardware optimized ROM packs based on suggested file/folder layouts.
Because most flash-carts and optical drive emulators require specific ROMs and fixes, it is a monumental task to compile 100% complete/working setups, and is often beyond the capabilities of any one person. Thousands of hours have been invested in the SourceMaterial DataBases (or SMDBs) of this project with the goal of 100% complete, 100% working real-hardware compatible arrangements of the highest quality ROM dumps. File hierarchies are shared via SMDB text files which contain all of the information needed to identify, sort and rename files.
What's in a SMDB file? A SMDB file generated by the parse_pack
script is a simple archival text record describing exact files (using
hash values) and the location of these files withing a folder
hierarchy. One record per line, six tab-separated columns per record:
- SHA256 value,
- folder and file name (Unix/Linux format),
- SHA1,
- MD5,
- CRC32,
- file size (in bytes, new feature not yet available in all SMDBs)
SMDBs are provided for a range of flash-carts. These SMDBs allow
users to dump all of their legally acquired ROMs into a single folder
(zip files accepted). When the build_pack script is run on that
directory, the ROMs will be analyzed (via hash comparisons), renamed
and sorted into complete, flash-cart friendly Packs, as described in
an SMDB. This allows creators to share file and folder setups without
having to share the ROMs themselves.
Tools Included
The build_pack, parse_pack, verify_pack and base_sorter scripts are written
in python3. To launch a script, install
python if need be, and then open a Windows
console or a Linux/Unix terminal.
(note to Windows users: for arguments expecting a folder name, please
use "C:\My Folder" or "C:\My Folder\\", but not "C:\My Folder\". The trailing backslash \ escapes the closing quote ",
which breaks the argument list)
parse_pack.py For making SMDBs (example command):
"C:\XXX\parse_pack.py" -f "C:\XXX\Folder to be parsed" -o "C:\XXX\SMDB.txt"
-f (or --folder) indicates the target ROM pack
-o (or --output) is the text file that will contain the hash
values, filenames, and folder structure
build_pack.py For building a pack based on a pre-made SMDB (example command):
"C:\XXX\build_pack.py" -i "C:\XXX\Folder with unorganized ROMs" -d "C:\XXX\SMDB.txt" -o "C:\XXX\Output folder for rebuilt pack" -m "C:\XXX\Missing.txt"
-i (or --input_folder) is the folder containing the unorganized
ROMs
-d (or --database) is the SMDB file describing the way your ROMs
are organized
-o (or --output_folder) is the folder in which to build the ROM
pack
-m (or --missing) is the text file that will list the ROMs missing
in order to reach the 100% mark
Options for advanced users:
--file_strategy {copy,hardlink,smart} changes the way files are
copied to the destination folder. The default is to physically
duplicate the files (copy). The options hardlink and smart avoid
file duplication and saves storage space for pack builders, only on
filesystems supporting this feature. Use hardlink when both source
and destination files are on the same filesystem, and smart when
destination files are on another filesystem. Please note that when
copying to a FAT32 or exFAT SD card, hardlinks are automatically
converted into normal files.
-s (or --skip_existing) avoids overwriting files that already
exist in the destination folder.
-x (or --drop_initial_directory) skips the first directory level
of the SMDB pack, so you can rename it to your convenience. For
instance, if your output folder is ./NES/, the pack will be built
under ./NES/EverDrive N8/.... With the option
--drop_initial_directory, the pack will be built directly under
./NES/....
verify_pack.py For verifying pack folders match up correctly with SMDBs (example command):
"C:\XXX\verify_pack.py" -f "C:\XXX\Folder to be parsed" -d "C:\XXX\SMDB.txt" -m "C:\XXX\Mismatch.txt"
-f (or --folder) indicates the target ROM pack
-d (or --database) is the SMDB file describing the way your ROMs
are organized
-m (or --mismatch) is the text file that will list the ROMs in incorrect locations,
extra files, and ROMs missing when compared to the SMDB.
-x (or --drop_initial_directory) skips the first directory level
of the SMDB pack, so you can rename it to your convenience. For
instance, if your pack folder is ./NES/, the pack will be verified
using ./NES/EverDrive N8/.... With the option
--drop_initial_directory, the pack will be verified using ./NES/....
base_sorter.py For automatically sorting an unsorted ROM pack with no available SMDB. Useful for starting a new SMDB:
"C:\XXX\base_sorter.py" -i "C:\XXX\Folder to be sorted"
-i (or --input_folder) is the folder containing the unorganized ROMs
-g (or --alphabetical_group_min_count) sets the minimun file count
for the alphabetical group dirs (defaults to 149)
-t (or --file_type) limits the sorting to one specific file type or extension.
eg: -t 7z will only sort .7z files
-d (or --discs) puts discs into a folder (single discs will be put into a folder also)
--debug turns debug messages on
Depending on your python installation, you may need to begin your
command with the location of python.exe (for example,
C:\Users\XXX\AppData\Local\Programs\Python\Python36-32\python.exe). More
information for pack builders are available in the
wiki.
GUI
A graphical user interface is available in
@Aleyr's
repository for
both the build_pack.py and parse_pack.py scripts. If you are
having difficulty with the command line options, please consider
trying the GUI version.
Requirements
python 3.5 or newer
Linux, MacOS, or Windows
Coding
Scripts and code by @frederic-mahe and @steve1515, with awesome patches by @eatnumber1, @coughlanio and @Slashbunny.
EverDrive Pack SMDB layouts by @SmokeMonsterPacks.
Similar tools
- GammaCopy (Windows only, direct support of SMDB files, fast execution)
- clrmamepro,
- romcenter,
- romvault,
- SabreTools