thombashi/allpairspy thombashi/allpairspy

  • Stars
    star
    258
  • Rank 158,189 (Top 4 %)
  • Language
    Python
  • License
    MIT License
  • Created almost 8 years ago
  • Updated about 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
thombashi/allpairspy
github

thombashi

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A python library for test combinations generator. The generator allows one to create a set of tests using "pairwise combinations" method, reducing a number of combinations of variables into a lesser set that covers most situations.

allpairspy forked from bayandin/allpairs

PyPI package version Supported Python versions Linux/macOS/Windows CI status Test coverage

AllPairs test combinations generator

AllPairs is an open source test combinations generator written in Python, developed and maintained by MetaCommunications Engineering. The generator allows one to create a set of tests using "pairwise combinations" method, reducing a number of combinations of variables into a lesser set that covers most situations.

For more info on pairwise testing see http://www.pairwise.org.

Features

  • Produces good enough dataset.
  • Pythonic, iterator-style enumeration interface.
  • Allows to filter out "invalid" combinations during search for the next combination.
  • Goes beyond pairs! If/when required can generate n-wise combinations.

Get Started

Basic Usage

Sample Code:
from allpairspy import AllPairs

parameters = [
    ["Brand X", "Brand Y"],
    ["98", "NT", "2000", "XP"],
    ["Internal", "Modem"],
    ["Salaried", "Hourly", "Part-Time", "Contr."],
    [6, 10, 15, 30, 60],
]

print("PAIRWISE:")
for i, pairs in enumerate(AllPairs(parameters)):
    print("{:2d}: {}".format(i, pairs))
Output:
PAIRWISE:
 0: ['Brand X', '98', 'Internal', 'Salaried', 6]
 1: ['Brand Y', 'NT', 'Modem', 'Hourly', 6]
 2: ['Brand Y', '2000', 'Internal', 'Part-Time', 10]
 3: ['Brand X', 'XP', 'Modem', 'Contr.', 10]
 4: ['Brand X', '2000', 'Modem', 'Part-Time', 15]
 5: ['Brand Y', 'XP', 'Internal', 'Hourly', 15]
 6: ['Brand Y', '98', 'Modem', 'Salaried', 30]
 7: ['Brand X', 'NT', 'Internal', 'Contr.', 30]
 8: ['Brand X', '98', 'Internal', 'Hourly', 60]
 9: ['Brand Y', '2000', 'Modem', 'Contr.', 60]
10: ['Brand Y', 'NT', 'Modem', 'Salaried', 60]
11: ['Brand Y', 'XP', 'Modem', 'Part-Time', 60]
12: ['Brand Y', '2000', 'Modem', 'Hourly', 30]
13: ['Brand Y', '98', 'Modem', 'Contr.', 15]
14: ['Brand Y', 'XP', 'Modem', 'Salaried', 15]
15: ['Brand Y', 'NT', 'Modem', 'Part-Time', 15]
16: ['Brand Y', 'XP', 'Modem', 'Part-Time', 30]
17: ['Brand Y', '98', 'Modem', 'Part-Time', 6]
18: ['Brand Y', '2000', 'Modem', 'Salaried', 6]
19: ['Brand Y', '98', 'Modem', 'Salaried', 10]
20: ['Brand Y', 'XP', 'Modem', 'Contr.', 6]
21: ['Brand Y', 'NT', 'Modem', 'Hourly', 10]

Filtering

You can restrict pairs by setting filtering function to filter_func at AllPairs constructor.

Sample Code:
from allpairspy import AllPairs

def is_valid_combination(row):
    """
    This is a filtering function. Filtering functions should return True
    if combination is valid and False otherwise.

    Test row that is passed here can be incomplete.
    To prevent search for unnecessary items filtering function
    is executed with found subset of data to validate it.
    """

    n = len(row)

    if n > 1:
        # Brand Y does not support Windows 98
        if "98" == row[1] and "Brand Y" == row[0]:
            return False

        # Brand X does not work with XP
        if "XP" == row[1] and "Brand X" == row[0]:
            return False

    if n > 4:
        # Contractors are billed in 30 min increments
        if "Contr." == row[3] and row[4] < 30:
            return False

    return True

parameters = [
    ["Brand X", "Brand Y"],
    ["98", "NT", "2000", "XP"],
    ["Internal", "Modem"],
    ["Salaried", "Hourly", "Part-Time", "Contr."],
    [6, 10, 15, 30, 60]
]

print("PAIRWISE:")
for i, pairs in enumerate(AllPairs(parameters, filter_func=is_valid_combination)):
    print("{:2d}: {}".format(i, pairs))
Output:
PAIRWISE:
 0: ['Brand X', '98', 'Internal', 'Salaried', 6]
 1: ['Brand Y', 'NT', 'Modem', 'Hourly', 6]
 2: ['Brand Y', '2000', 'Internal', 'Part-Time', 10]
 3: ['Brand X', '2000', 'Modem', 'Contr.', 30]
 4: ['Brand X', 'NT', 'Internal', 'Contr.', 60]
 5: ['Brand Y', 'XP', 'Modem', 'Salaried', 60]
 6: ['Brand X', '98', 'Modem', 'Part-Time', 15]
 7: ['Brand Y', 'XP', 'Internal', 'Hourly', 15]
 8: ['Brand Y', 'NT', 'Internal', 'Part-Time', 30]
 9: ['Brand X', '2000', 'Modem', 'Hourly', 10]
10: ['Brand Y', 'XP', 'Modem', 'Contr.', 30]
11: ['Brand Y', '2000', 'Modem', 'Salaried', 15]
12: ['Brand Y', 'NT', 'Modem', 'Salaried', 10]
13: ['Brand Y', 'XP', 'Modem', 'Part-Time', 6]
14: ['Brand Y', '2000', 'Modem', 'Contr.', 60]

Data Source: OrderedDict

You can use collections.OrderedDict instance as an argument for AllPairs constructor. Pairs will be returned as collections.namedtuple instances.

Sample Code:
from collections import OrderedDict
from allpairspy import AllPairs

parameters = OrderedDict({
    "brand": ["Brand X", "Brand Y"],
    "os": ["98", "NT", "2000", "XP"],
    "minute": [15, 30, 60],
})

print("PAIRWISE:")
for i, pairs in enumerate(AllPairs(parameters)):
    print("{:2d}: {}".format(i, pairs))
Sample Code:
PAIRWISE:
 0: Pairs(brand='Brand X', os='98', minute=15)
 1: Pairs(brand='Brand Y', os='NT', minute=15)
 2: Pairs(brand='Brand Y', os='2000', minute=30)
 3: Pairs(brand='Brand X', os='XP', minute=30)
 4: Pairs(brand='Brand X', os='2000', minute=60)
 5: Pairs(brand='Brand Y', os='XP', minute=60)
 6: Pairs(brand='Brand Y', os='98', minute=60)
 7: Pairs(brand='Brand X', os='NT', minute=60)
 8: Pairs(brand='Brand X', os='NT', minute=30)
 9: Pairs(brand='Brand X', os='98', minute=30)
10: Pairs(brand='Brand X', os='XP', minute=15)
11: Pairs(brand='Brand X', os='2000', minute=15)

Parameterized testing with pairwise by using pytest

Parameterized testing: valee matrix

Sample Code:
import pytest
from allpairspy import AllPairs

def function_to_be_tested(brand, operating_system, minute) -> bool:
    # do something
    return True

class TestParameterized(object):
    @pytest.mark.parametrize(["brand", "operating_system", "minute"], [
        values for values in AllPairs([
            ["Brand X", "Brand Y"],
            ["98", "NT", "2000", "XP"],
            [10, 15, 30, 60]
        ])
    ])
    def test(self, brand, operating_system, minute):
        assert function_to_be_tested(brand, operating_system, minute)
Output:
$ py.test test_parameterize.py -v
============================= test session starts ==============================
...
collected 16 items

test_parameterize.py::TestParameterized::test[Brand X-98-10] PASSED      [  6%]
test_parameterize.py::TestParameterized::test[Brand Y-NT-10] PASSED      [ 12%]
test_parameterize.py::TestParameterized::test[Brand Y-2000-15] PASSED    [ 18%]
test_parameterize.py::TestParameterized::test[Brand X-XP-15] PASSED      [ 25%]
test_parameterize.py::TestParameterized::test[Brand X-2000-30] PASSED    [ 31%]
test_parameterize.py::TestParameterized::test[Brand Y-XP-30] PASSED      [ 37%]
test_parameterize.py::TestParameterized::test[Brand Y-98-60] PASSED      [ 43%]
test_parameterize.py::TestParameterized::test[Brand X-NT-60] PASSED      [ 50%]
test_parameterize.py::TestParameterized::test[Brand X-NT-30] PASSED      [ 56%]
test_parameterize.py::TestParameterized::test[Brand X-98-30] PASSED      [ 62%]
test_parameterize.py::TestParameterized::test[Brand X-XP-60] PASSED      [ 68%]
test_parameterize.py::TestParameterized::test[Brand X-2000-60] PASSED    [ 75%]
test_parameterize.py::TestParameterized::test[Brand X-2000-10] PASSED    [ 81%]
test_parameterize.py::TestParameterized::test[Brand X-XP-10] PASSED      [ 87%]
test_parameterize.py::TestParameterized::test[Brand X-98-15] PASSED      [ 93%]
test_parameterize.py::TestParameterized::test[Brand X-NT-15] PASSED      [100%]

Parameterized testing: OrderedDict

Sample Code:
import pytest
from allpairspy import AllPairs

def function_to_be_tested(brand, operating_system, minute) -> bool:
    # do something
    return True

class TestParameterized(object):
    @pytest.mark.parametrize(
        ["pair"],
        [
            [pair]
            for pair in AllPairs(
                OrderedDict(
                    {
                        "brand": ["Brand X", "Brand Y"],
                        "operating_system": ["98", "NT", "2000", "XP"],
                        "minute": [10, 15, 30, 60],
                    }
                )
            )
        ],
    )
    def test(self, pair):
        assert function_to_be_tested(pair.brand, pair.operating_system, pair.minute)

Other Examples

Other examples could be found in examples directory.

Installation

Installation: pip

pip install allpairspy

Installation: apt

You can install the package by apt via a Personal Package Archive (PPA):

sudo add-apt-repository ppa:thombashi/ppa
sudo apt update
sudo apt install python3-allpairspy

Known issues

  • Not optimal - there are tools that can create smaller set covering all the pairs. However, they are missing some other important features and/or do not integrate well with Python.
  • Lousy written filtering function may lead to full permutation of parameters.
  • Version 2.0 has become slower (a side-effect of introducing ability to produce n-wise combinations).

Dependencies

Python 3.5+ no external dependencies.

Sponsors

Dmitry Belyaev (b4tman) Charles Becker (chasbecker) Arturi0

Become a sponsor

More Repositories

1

sqlitebiter

A CLI tool to convert CSV / Excel / HTML / JSON / Jupyter Notebook / LDJSON / LTSV / Markdown / SQLite / SSV / TSV / Google-Sheets to a SQLite database file.
Python
841
star
2

tcconfig

A tc command wrapper. Make it easy to set up traffic control of network bandwidth/latency/packet-loss/packet-corruption/etc. to a network-interface/Docker-container(veth).
Python
787
star
3

pytablewriter

pytablewriter is a Python library to write a table in various formats: AsciiDoc / CSV / Elasticsearch / HTML / JavaScript / JSON / LaTeX / LDJSON / LTSV / Markdown / MediaWiki / NumPy / Excel / Pandas / Python / reStructuredText / SQLite / TOML / TSV.
Python
602
star
4

pathvalidate

A Python library to sanitize/validate a string such as filenames/file-paths/etc.
Python
215
star
5

SimpleSQLite

SimpleSQLite is a Python library to simplify SQLite database operations: table creation, data insertion and get data as other data formats. Simple ORM functionality for SQLite.
Python
128
star
6

pytablereader

A Python library to load structured table data from files/strings/URL with various data format: CSV / Excel / Google-Sheets / HTML / JSON / LDJSON / LTSV / Markdown / SQLite / TSV.
Python
105
star
7

DateTimeRange

DateTimeRange is a Python library to handle a time range. e.g. check whether a time is within the time range, get the intersection of time ranges, truncate a time range, iterate through a time range, and so forth.
Python
102
star
8

pingparsing

pingparsing is a CLI-tool/Python-library parser and transmitter for ping command β†ͺ️
Python
77
star
9

pytest-md-report

A pytest plugin to generate test outcomes reports with markdown table format.
Python
32
star
10

subprocrunner

A Python wrapper library for subprocess module.
Python
21
star
11

typepy

A Python library for variable type checker/validator/converter at a run time.
Python
17
star
12

humanreadable

humanreadable is a Python library to convert human-readable values to other units.
Python
16
star
13

DataProperty

A Python library for extract property from data.
Python
15
star
14

cleanpy

cleanpy is a CLI tool to remove caches and temporary files related to Python.
Python
13
star
15

sqliteschema

sqliteschema is a Python library to dump table schema of a SQLite database file.
Python
11
star
16

elasticsearch-faker

elasticsearch-faker is a CLI tool to generate fake data for Elasticsearch.
Python
11
star
17

tcolorpy

tcolopy is a Python library to apply true color for terminal text.
Python
11
star
18

ghscard

A JavaScript widget to generate interactive GitHub user/repository/organization cards for static web pages (like GitHub pages).
JavaScript
10
star
19

excelrd

excelrd is a modified version of xlrd to work for the latest Python versions.
Python
9
star
20

cmakew

cmakew is a CMake wrapper CLI tool.
Python
9
star
21

thank-you-stars

thank-you-stars is a CLI tool to stars to a PyPI package and its dependencies hosted on GitHub ⭐
Python
8
star
22

tabledata

tabledata is a Python library to represent tabular data.
Python
7
star
23

mbstrdecoder

Python library for multi-byte character string decoder.
Python
6
star
24

pytest-discord

A pytest plugin to notify test results to a Discord channel.
Python
5
star
25

tblfaker

tblfaker is a Python library to generate fake tabular data.
Python
5
star
26

CriterionSample

πŸ“” Examples of Criterion (https://github.com/Snaipe/Criterion)
C
5
star
27

python-lib-project-template

A project template for a Python library with CI configurations
Python
5
star
28

envinfopy

envinfopy is a Python Library to get execution environment information.
Python
4
star
29

NFStest

NFStest mirror with command help wiki.
Python
4
star
30

readmemaker

A Python utility library to help make a README file from document files.
Python
4
star
31

retryrequests

A Python library that make HTTP requests with exponential back-off retry by using requests package.
Python
4
star
32

gtest-project-template

C++ test project template using Google Test.
C++
4
star
33

releasecmd

releasecmd is a release subcommand for setup.py (setuptools.setup). The subcommand create a git tag and push, and upload packages to PyPI.
Python
3
star
34

appconfigpy

A Python library to create/load an application configuration file.
Python
3
star
35

msgfy

msgfy is a Python library for convert Exception instance to a human-readable error message.
Python
3
star
36

python-cli-project-template

A project template for a Python CLI tool with CI configurations
Python
2
star
37

homebrew-sqlitebiter

Homebrew Formula for sqlitebiter
Ruby
2
star
38

docker-alias

Docker aliases and functions 🐳
Shell
2
star
39

universal-ctags-installer

Build and installation script for Universal Ctags
Shell
2
star
40

gh-upgrade

gh-upgrade is a gh extension that updates the gh and its extensions to the latest version.
Shell
2
star
41

docker-opengrok

Docker image for OpenGrok
Dockerfile
2
star
42

vscode-snippets

User defined snippets for Visual Studio Code
2
star
43

pytablewriter-altrow-theme

pytablewriter-altrow-theme is a pytablewriter plugin to provide a theme that colored rows alternatively.
Python
1
star
44

thutils

Personal python utility library (may delete in the future)
Python
1
star
45

install-gh

Simple one-liner installer of gh (cli/cli) from the release assets.
Shell
1
star
46

Connectathon_README

Connectathon test suite README
HTML
1
star
47

pathvalidate-cli

pathvalidate-cli is a command line interface for pathvalidate library. The tool can do sanitize/validate a string such as file-names/file-paths.
Python
1
star
48

PythonExamples

πŸ“” Learning Python libraries by examples.
Jupyter Notebook
1
star
49

dotfiles

♻️ dotfiles
Python
1
star
50

df-diskcache

df-diskcache is a Python library for caching pandas.DataFrame objects to local disk.
Python
1
star
51

gh-git-describe

gh extension to execute `git describe` on a remote GitHub repository.
Go
1
star
52

list-sponsors

A CLI tool to list GitHub sponsors of a user/organization with specified format (markdown/html/rst).
Python
1
star
53

pre-commit-bin

Single executable binary packages of pre-commit/pre-commit
Shell
1
star
54

gh-update

gh-update is a gh extension that updates the gh to the latest version.
Shell
1
star