swagger-coverage
Swagger-coverage gives a full picture about coverage of API tests (regression) based on OAS (Swagger). By saying coverage we mean not a broad theme functionality, but presence (or absence) of calls defined by API methods, parameters, return codes or other conditions which corresponds specification of API.
How it works
Producing coverage report consists of two parts. Firstly, during test execution, filter/interceptor/proxy save information of calls in swagger format in specific folder on executing tests. The next stage is to compare saved result with generated conditions from current API specification and builds report.
How to use and examples
You can use swagger-coverage with any language and framework. You need to have proxy/filter/interceptor that accumulates data in swagger format. Swagger-coverage have rest-assured integration from the box.
There is also a Karate integration, which has its own manual.
Add filter dependency:
<dependency>
<groupId>com.github.viclovsky</groupId>
<artifactId>swagger-coverage-rest-assured</artifactId>
<version>${latest-swagger-coverage-version}</version>
</dependency>or if use gradle, it can be added like
compile "com.github.viclovsky:swagger-coverage-rest-assured:$latest-swagger-coverage-version"
Just add filter into test client SwaggerCoverageRestAssured (SwaggerCoverageV3RestAssured for v3). For instance, as presented below:
RestAssured.given().filter(new SwaggerCoverageRestAssured())- Download and run command line. Download zip archive and unpack it. Don't forget to replace {latest-swagger-coverage-version} to latest version.
wget https://github.com/viclovsky/swagger-coverage/releases/download/{latest-swagger-coverage-version}/swagger-coverage-{latest-swagger-coverage-version}.zip
unzip swagger-coverage-commandline-{latest-swagger-coverage-version}.zip
Here is help of unzip swagger-commandline
./swagger-coverage-commandline --help
Options:
* -s, --spec
Path to local or URL to remote swagger specification.
* -i, --input
Path to folder with generated files with coverage.
-c, --configuration
Path to file with report configuration.
--help
Print commandline help.
-q, --quiet
Switch on the quiet mode.
Default: false
-v, --verbose
Switch on the verbose mode.
Default: false
To compare result of API tests with current API specification and build report call command line tool after running tests like that:
./swagger-coverage-commandline -s swagger.json -i swagger-coverage-output
Output of the command:
19:21:21 INFO OperationSwaggerCoverageCalculator - Empty coverage:
...
19:21:21 INFO OperationSwaggerCoverageCalculator - Partial coverage:
...
19:21:21 INFO OperationSwaggerCoverageCalculator - Full coverage:
...
19:21:21 INFO OperationSwaggerCoverageCalculator - Conditions: 874/2520
19:21:21 INFO OperationSwaggerCoverageCalculator - Empty coverage 49.284 %
19:21:21 INFO OperationSwaggerCoverageCalculator - Partial coverage 12.034 %
19:21:21 INFO OperationSwaggerCoverageCalculator - Full coverage 38.682 %
19:21:21 INFO FileSystemResultsWriter - Write html report in file '.../swagger-coverage-report.html'
Results (swagger-coverage-report.html/swagger-coverage-results.json) will be created after running of swagger-coverage.
Configuration options
Swagger-coverage report can be configured by json-file. You can control list of coverage, which be generated and checked for results.
Rules configuration options
Options for different rules are placed in "rules" section. You can disable some rules or change their behavior.
Checking response http-status
This rule create condition for every status from responses-section of swagger specification. Condition mark covered when report generator find specific status in results files. Options for this rules are placed in status subsection in rules sections.
You can setup next options:
enable - true/false. You can disable this rule. Default value is true.
filter - [val1,val2]. Rule will ignore all status, which not in filter list.
ignore - [val1,val2]. Rule will ignore all status, which in ignore list.
{
"rules" : {
"status": {
"enable": true,
"ignore": ["400","500"],
"filter": ["200"]
},
....
},
....
}
Checking the list of declared and received statuses
This rule create condition for comparing declared and received status. Condition marked as covered when result not contains any of undeclared status. Uncovered state of this condition indicates missed status in original swagger-documentation or server errors. Options for this rules are placed in only-declared-status subsection in rules sections.
You can setup next options:
enable - true/false. You can disable this rule. Default value is true.
{
"rules" : {
....
"only-declared-status" : {
"enable" : true
}
},
....
}
Excluding deprecated operations from the coverage report statistic
This rule is created for cases when you don't want to measure coverage of deprecated operations, but only for actual ones.
If an operation is deprecated then it will be excluded from Full, Partial, and Empty categories and won't affect the "Operations coverage summary"
Options for this rule are placed in "exclude-deprecated" subsection in rules sections.
You can set up next options:
enable - true/false.
By default, this rule is not enabled. Add it to the config file with true value to enable this rule, like in the example below:
{
"rules" : {
....
"exclude-deprecated" : {
"enable" : true
}
},
....
}
If you need you can add your rules for generation of conditions. So, please, send your PRs.
Result writer configuration
Options for report generation setting are placed in writers sections.
HTML report writer
Options for html-report placed in subsection html of writers sections.
You can setup next options:
locale - two latter language code. Now supported only en/ru.
filename - filename for html report.
numberFormat - Extended Java decimal format to control how numbers are displayed in the report.
{
....
"writers": {
"html": {
"locale": "ru",
"filename":"report.html",
"numberFormat": "0.##"
}
}
}
Report customization
To customize your http report with your own template set full path to the template like below:
{
....
"writers": {
"html": {
....
"customTemplatePath": "/full/path/to/report_custom.ftl"
}
}
}
Look here to see all available options.
Demo
I have prepared several tests. Thus you are able to have a look and touch swagger-coverage. Just run run.sh script.
Important remark
Swagger-coverage works fine with clients which were generated from swagger (for example: https://github.com/OpenAPITools/openapi-generator). Because all methods/parameters which will be saved are 100% compatible with current API specification.
Requirements
For a moment swagger-coverage is compatible only with OpenApi specifications v2 & v3. It is possible that swagger-coverage will support other versions.
Pull Requests
My project is open for any enhancement. So, your help is much appreciated. Please, feel free to open your pull request or issue and I will consider it in several days.
Created & Maintained By
Contributing to swagger-coverage
Thanks to all people who contributed. Especially
who have contributed significant improvements to swagger-coverage.
License
Swagger coverage is released under version 2.0 of the Apache License