• Stars
    star
    219
  • Rank 181,133 (Top 4 %)
  • Language
    Groovy
  • License
    Apache License 2.0
  • Created almost 5 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Gradle plugin to retry tests that have failed to mitigate test flakiness.

Test Retry Gradle plugin

A Gradle plugin that augments Gradleโ€™s built-in test task with the ability to retry tests that have failed.

Version GitHub license

What it does

The plugin causes failed tests to be retried within the same task. After executing all tests, any failed tests are retried. The process repeats with tests that continue to fail until the maximum specified number of retries has been attempted, or there are no more failing tests.

By default, all failed tests passing on retry prevents the test task from failing. This mode prevents flaky tests from causing build failure. This setting can be changed so that flaky tests cause build failure, which can be used to identify flaky tests.

When something goes badly wrong and all tests start failing, it can be preferable to not keep retrying tests. This can happen for example if a disk fills up or a required database is not available. To avoid this, the plugin can be configured to stop retrying after a certain number of total test failures.

NOTE: Retrying tests alone is not a viable flaky test mitigation strategy. This plugin should only be used alongside processes for tracking and fixing discovered flaky tests.

Usage

Apply the plugin using one of the two methods described on the Gradle Plugin Portal, where the plugin is listed as org.gradle.test-retry. It is compatible with Gradle 5.0 and later versions.

By default, retrying is not enabled.

Retrying is configured per test task via the retry extension added to each task by the plugin.

build.gradle:
test {
    retry {
        maxRetries = 2
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}
build.gradle.kts:
test {
    retry {
        maxRetries.set(2)
        maxFailures.set(20)
        failOnPassedAfterRetry.set(true)
    }
}

Limiting retry to CI builds

You may find that local developer builds do not benefit much from retry behaviour, particularly when those tests are invoked via your IDE. In that case we recommend enabling retry only for CI builds.

build.gradle:
boolean isCiServer = System.getenv().containsKey("CI")
test {
    retry {
        if (isCiServer) {
            maxRetries = 2
            maxFailures = 20
        }
        failOnPassedAfterRetry = true
    }
}

The retry extension

The retry extension is of the following type:

package org.gradle.testretry;

import org.gradle.api.Action;
import org.gradle.api.provider.Property;
import org.gradle.api.provider.SetProperty;
import org.gradle.api.tasks.testing.Test;

/**
 * Allows configuring test retry mechanics.
 * <p>
 * This extension is added with the name 'retry' to all {@link Test} tasks.
 */
public interface TestRetryTaskExtension {

    /**
     * The name of the extension added to each test task.
     */
    String NAME = "retry";

    /**
     * Whether tests that initially fail and then pass on retry should fail the task.
     * <p>
     * This setting defaults to {@code false},
     * which results in the task not failing if all tests pass on retry.
     * <p>
     * This setting has no effect if {@link Test#getIgnoreFailures()} is set to true.
     *
     * @return whether tests that initially fails and then pass on retry should fail the task
     */
    Property<Boolean> getFailOnPassedAfterRetry();

    /**
     * The maximum number of times to retry an individual test.
     * <p>
     * This setting defaults to {@code 0}, which results in no retries.
     * Any value less than 1 disables retrying.
     *
     * @return the maximum number of times to retry an individual test
     */
    Property<Integer> getMaxRetries();

    /**
     * The maximum number of test failures that are allowed before retrying is disabled.
     * <p>
     * The count applies to each round of test execution.
     * For example, if maxFailures is 5 and 4 tests initially fail and then 3 again on retry,
     * this will not be considered too many failures and retrying will continue (if maxRetries {@literal >} 1).
     * If 5 or more tests were to fail initially then no retry would be attempted.
     * <p>
     * This setting defaults to {@code 0}, which results in no limit.
     * Any value less than 1 results in no limit.
     *
     * @return the maximum number of test failures that are allowed before retrying is disabled
     */
    Property<Integer> getMaxFailures();

    /**
     * The filter for specifying which tests may be retried.
     */
    Filter getFilter();

    /**
     * The filter for specifying which tests may be retried.
     */
    void filter(Action<? super Filter> action);

    /**
     * A filter for specifying which tests may be retried.
     *
     * By default, all tests are eligible for retrying.
     */
    interface Filter {

        /**
         * The patterns used to include tests based on their class name.
         *
         * The pattern string matches against qualified class names.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class name only has to match one pattern to be included.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeClasses();

        /**
         * The patterns used to include tests based on their class level annotations.
         *
         * The pattern string matches against the qualified class names of a test class's annotations.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class need only have one annotation matching any of the patterns to be included.
         *
         * Annotations present on super classes that are {@code @Inherited} are considered when inspecting subclasses.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeAnnotationClasses();

        /**
         * The patterns used to exclude tests based on their class name.
         *
         * The pattern string matches against qualified class names.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class name only has to match one pattern to be excluded.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getExcludeClasses();

        /**
         * The patterns used to exclude tests based on their class level annotations.
         *
         * The pattern string matches against the qualified class names of a test class's annotations.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class need only have one annotation matching any of the patterns to be excluded.
         *
         * Annotations present on super classes that are {@code @Inherited} are considered when inspecting subclasses.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getExcludeAnnotationClasses();

    }


    /**
     * The set of criteria specifying which test classes must be retried as a whole unit
     * if retries are enabled and the test class passes the configured filter.
     */
    ClassRetryCriteria getClassRetry();

    /**
     * The set of criteria specifying which test classes must be retried as a whole unit
     * if retries are enabled and the test class passes the configured filter.
     */
    void classRetry(Action<? super ClassRetryCriteria> action);

    /**
     * The set of criteria specifying which test classes must be retried as a whole unit
     * if retries are enabled and the test class passes the configured filter.
     */
    interface ClassRetryCriteria {

        /**
         * The patterns used to include tests based on their class name.
         *
         * The pattern string matches against qualified class names.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class name only has to match one pattern to be included.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeClasses();

        /**
         * The patterns used to include tests based on their class level annotations.
         *
         * The pattern string matches against the qualified class names of a test class's annotations.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class need only have one annotation matching any of the patterns to be included.
         *
         * Annotations present on super classes that are {@code @Inherited} are considered when inspecting subclasses.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeAnnotationClasses();

    }

}

Supported test frameworks

Other versions are likely to work as well, but are not tested.

Framework Version Tested

JUnit4

4.13.2

JUnit5

5.9.2

Spock

2.3-groovy-3.0

TestNG

7.5

Parameterized tests

In a few cases, test selection for testing frameworks limits the granularity at which tests can be retried. In each case, this plugin retries at worst at method level. For JUnit5 @ParameterizedTest, TestNG @Test(dataProvider = "โ€ฆโ€‹"), and Spock @Unroll tests the plugin will retry the entire method with all parameters including those that initially passed.

Test dependencies

The plugin supports retrying Spock @Stepwise tests and TestNG @Test(dependsOn = { โ€ฆ }) tests.

  • Upstream tests (those that the failed test depends on) are run because a flaky test may depend on state from the prior execution of an upstream test.

  • Downstream tests are run because a flaky test causes any downstream tests to be skipped in the initial test run.

Custom test frameworks

Some projects may use test tasks with a custom TestFramework to execute tests. If this is the case, the plugin disables retries and emits the following warning:

> Task :unsupportedTestTaskUnitTest
Test retry requested for task :unsupportedTestTaskUnitTest with unsupported test framework CustomTestFramework - failing tests will not be retried

To avoid this warning, we can disable retries for the unsupported test task with:

build.gradle:
test.named('unsupportedTestTaskUnitTest') {
    retry {
        maxRetries = 0
    }
}
build.gradle.kts:
tasks.named<Test>("unsupportedTestTaskUnitTest") {
    retry {
        maxRetries.set(0)
    }
}

Filtering

By default, all tests are eligible for retrying. The filter component of the test retry extension can be used to control which tests should be retried and which should not.

The decision to retry a test or not is based on the tests reported class name, regardless of the name of the test case or method. The annotations present or not on this class can also be used as the criteria.

build.gradle:
test {
    retry {
        maxRetries = 3
        filter {
            // filter by qualified class name (* matches zero or more of any character)
            includeClasses.add("*IntegrationTest")
            excludeClasses.add("*DatabaseTest")

            // filter by class level annotations
            // Note: @Inherited annotations are respected
            includeAnnotationClasses.add("*Retryable")
            excludeAnnotationClasses.add("*NonRetryable")
        }
    }
}

Retry on class-level

By default, individual tests are retried. The classRetry component of the test retry extension can be used to control which test classes must be retried as a whole unit. Test classes still have to pass the configured filter.

build.gradle:
test {
    retry {
        maxRetries = 3
        classRetry {
            // configure by qualified class name (* matches zero or more of any character)
            includeClasses.add("*StepWiseIntegrationTest")

            // configure by class level annotations
            // Note: @Inherited annotations are respected
            includeAnnotationClasses("Stepwise")
        }
    }
}

Reporting

Gradle

Each execution of a test is discretely reported in Gradle-generated XML and HTML reports.

Gradle test reporting

Gradle flaky test reporting

Similar to the XML and HTML reports, the console log will also report each individual test execution. Before retrying a failed test, Gradle will execute the whole test suite of the test task. This means that all executions of the same test may not be grouped in the console log.

Gradle console reporting

Gradle Enterprise

Gradle build scans (--scan option) report discrete test executions as "Execution [N of total]" and will mark a test with both a failed and then a passed outcome as flaky.

Gradle build scan reporting

Flaky tests can also be visualized across many builds using the Gradle Enterprise Tests Dashboard.

Gradle Enterprise top tests report

IDEs

The plugin has been tested with IDEA, Eclipse IDE and Netbeans.

IDEA

When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.

IDEA test reporting

Eclipse

When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.

Eclipse test reporting

Netbeans

Netbeans only shows the last execution of a test.

Netbeans test reporting

CI tools

The plugin has been tested with the reporting of TeamCity and Jenkins.

TeamCity

Flaky tests (tests being executed multiple times but with different results) are detected by TeamCity and marked as flaky. TeamCity lists each test that was executed and how often it was run in the build.

By default, TeamCity will fail your build if at least one test fails. When using failOnPassedAfterRetry = false (ie. the default for this plugin), this failure condition should be disabled.

Teamcity test reporting

Jenkins

Jenkins reports each test execution discretely.

Jenkins test reporting

More Repositories

1

gradle

Adaptable, fast automation for all
Groovy
16,800
star
2

kotlin-dsl-samples

Samples builds using the Gradle Kotlin DSL
Kotlin
3,706
star
3

gradle-profiler

A tool for gathering profiling and benchmarking information for Gradle builds
Java
1,279
star
4

gradle-completion

Gradle tab completion for bash and zsh
Shell
966
star
5

gradle-build-action

Execute your Gradle build and trigger dependency submission
672
star
6

android-cache-fix-gradle-plugin

Gradle plugin that fixes Android build caching problems.
Groovy
440
star
7

wrapper-validation-action

Gradle Wrapper Validation Action
256
star
8

oreilly-gradle-book-examples

Java
249
star
9

native-samples

Samples of modern build automation for native languages with Gradle
Java
153
star
10

actions

A collection of GitHub Actions to accelerate your Gradle Builds on GitHub
TypeScript
147
star
11

gradle-build-scan-quickstart

An example project to experience the Build Scanยฎ service of Gradle Enterprise with Gradle builds.
Kotlin
132
star
12

foojay-toolchains

Java Toolchain Resolve Plugin based on the foojay DiscoAPI
Kotlin
120
star
13

native-platform

Java bindings for various native APIs
Java
116
star
14

declarative-gradle

Declarative Gradle is a project targeting better isolation of concern and expressing any build in a clear and understandable way
Java
110
star
15

gradle-native

The home of Gradle's support for natively compiled languages
92
star
16

build-tool-training-exercises

Exercises for live gradle.com/training sessions
Java
90
star
17

gradle-enterprise-build-validation-scripts

Executable scripts to assist in validating that your Gradle and Maven builds are in an optimal state in terms of maximizing work avoidance when using Develocity.
Shell
86
star
18

github-dependency-graph-gradle-plugin

Gradle Plugin for Extracting Dependency Information to send to GitHub
Groovy
81
star
19

develocity-build-config-samples

Code samples that demonstrate how to customize your Develocity build configuration using Gradle, Maven, Bazel or sbt
Java
68
star
20

playframework

Gradle Play Support
Java
47
star
21

guides

The Gradle Guides at https://guides.gradle.org.
Java
47
star
22

wrapper-upgrade-gradle-plugin

Gradle plugin that detects and updates Gradle and Maven wrappers to the latest Gradle and Maven version.
Java
43
star
23

gradle-site-plugin

Kotlin
43
star
24

build-tool-roadmap

Gradle Build Tool roadmap
41
star
25

common-custom-user-data-gradle-plugin

Gradle plugin that enhances published build scans by adding a set of tags, links and custom values that have proven to be useful for many projects building with Develocity.
Java
41
star
26

kotlin-dsl-docs

Generates Kotlin DSL API reference
Kotlin
38
star
27

gradle-talks

A javascript based custom slide and build framework for presentations. Many of the Gradle engineers have been using this for their presentations. Those presentations are part of this repo and can be found in the talks directory.
Ruby
35
star
28

exemplar

Discover and verify code samples and services
Java
34
star
29

gradle2kts

Gradle Groovy to Gradle Kotlin conversion tool - discontinued spike
Kotlin
34
star
30

gradle-checksum

A Gradle plugin for creating checksums for files in your build.
Groovy
31
star
31

kotlin-dsl-conventions

Gradle Kotlin DSL conventional plugins
Kotlin
26
star
32

maven-build-scan-quickstart

An example project to experience the Build Scanยฎ service of Develocity with Maven builds.
Java
20
star
33

bazel-comparison

20
star
34

develocity-oss-projects

19
star
35

gradle-distributions

Repository for Gradle Build Tool distributions downloads
17
star
36

performance-comparisons

A set of synthetic projects used to benchmark Gradle against other build tools
16
star
37

community

Gradle Community content and Gradle Cookbook. Open for Contributions
HTML
16
star
38

gradle-java-modules

A (former) place for experimenting with Java 9's module system. Gradle officially supports Java Modules since version 6.4
Java
16
star
39

gcc2speedscope

Space usage analysis for the Gradle configuration cache via speedscope
Kotlin
15
star
40

cc-hackathon-2022

Configuration Cache Hackathon 2022
15
star
41

.github

Maintains all of the default policies for the Gradle organization
14
star
42

perf-enterprise-large

A large Java based build to use when benchmarking and profiling Gradle
Groovy
14
star
43

gradle-enterprise-export-api-samples

A repository of samples that demonstrate how to use the Gradle Enterprise Export API.
14
star
44

gradle-jdocbook

A Gradle plugin for jdocbook
Groovy
13
star
45

cucumber-companion

Maven & Gradle plugins providing convenient support for running Cucumber test directly from Maven/Gradle
Groovy
13
star
46

multi-project-composite-gradle-plugins-builds

Multi-project build using composite Gradle plugins
Kotlin
12
star
47

develocity-api-samples

A repository of samples that demonstrate how to use the Develocity API.
Java
12
star
48

webinar-getting-started-with-the-gradle-kotlin-dsl

Webinar - Getting Started with the Kotlin DSL
Kotlin
12
star
49

plugin-portal-requests

Gradle Plugin Portal issues and requests.
12
star
50

speed-challenge

Instructions and overview for the Gradle and Maven Speed Challenge event
11
star
51

imaginate

Using the Kotlin language for production, test and build makes it easier for everyone to work with your software code base. Letโ€™s go 100% Kotlin!
Kotlin
11
star
52

continuous-delivery-jump-start

Sample application used for training "Continuous Delivery Jump Start"
Java
11
star
53

santa-tracker-performance

Performance tests for Santa Tracker Android project
Shell
10
star
54

perf-android-large-2

Another large Android build for performance experiments
10
star
55

build-analysis-demo

Build data analysis applications
Kotlin
10
star
56

gradle-issue-reproducer

Template repository for providing Gradle issue reproducers
10
star
57

gradle-benchmark-base

Base scenarios for Gradle Profiler to benchmark Gradle builds
9
star
58

gradle-org-conventions-plugin

Java
9
star
59

gradle-hello-world-plugin

Groovy
9
star
60

gradle-profiler-plugin

Java
9
star
61

develocity-testing-annotations

Common annotations for Develocity and Test Retry
Java
9
star
62

tooling-commons

A small layer on top of the Gradle Tooling API that provides the Tooling Client and other convenience useful for IDE integration.
Java
9
star
63

gradle-client

Desktop application acting as a Gradle Tooling API client
Kotlin
9
star
64

configuration-cache-report

Kotlin
8
star
65

jfr-polyfill

A polyfill for JDK Flight Recorder (JFR) to avoid errors on JDKs that don't support JFR yet
Java
8
star
66

common-custom-user-data-maven-extension

Maven extension that enhances published build scans by adding a set of tags, links and custom values that have proven to be useful for many projects building with Develocity.
Java
8
star
67

develocity-gitlab-templates

GitLab CI/CD templates to automatically connect Gradle/Maven builds to Develocity
Shell
7
star
68

bt-dev-prod-data-collector

Data collector for Gradle Build Tool Developer productivity metrics
Kotlin
7
star
69

gradle-hazelcast-plugin

Groovy
6
star
70

source-resolution-demo

Java
6
star
71

gradle-enterprise-build-optimization-experiments

Self-guiding experiments to optimize the performance of your Gradle and Maven builds with Gradle Enterprise.
6
star
72

gradle-plugin-analyzer

Analyzes the bytecode of Gradle plugins
Java
6
star
73

integrations

A repository for Gradle Integrations and the related knowledge base
6
star
74

gradle-rules-configuration-workshop

6
star
75

gradle-project-templates

Learning day experiment: project init templates for Gradle
Java
5
star
76

greeting-plugin-example

Java
5
star
77

gradle-dependency-constrain

Java
5
star
78

maven-build-cache-unstable-inputs-samples

An example project containing a build with unstable inputs
Java
5
star
79

dpeuni-gradle-intro-devs-init

Hands-on exercise for DPE University
5
star
80

develocity-actions

Shared Github Actions
TypeScript
4
star
81

develocity-bamboo-plugin

Develocity plugin for Bamboo
Java
4
star
82

stable-plugins-dsl

Getting the plugins {} DSL block out of incubation
Java
4
star
83

perf-native-large

A Place to Profile Particularly Prickly Projects
C
4
star
84

backend-code-project

A scaffold for the Develocity Backend Engineer code project assignment.
Java
3
star
85

ide-smoke-tests

Java
3
star
86

gradle-all

A composite build that includes all the pieces of Gradle
Kotlin
3
star
87

ge-export

Java
3
star
88

webinar-dep-mgmt-part-1

Kotlin
3
star
89

gradle-to-trace-converter

Kotlin
3
star
90

provider-api-migration-testbed

A testbed to try out mitigation strategies for the provider API migration
Kotlin
3
star
91

gradle-model-vis

JavaScript
3
star
92

gradle-performance

Gradle performance benchmarks using the Gradle build tool itself
Shell
3
star
93

sbt-build-scan-quickstart

An example project showcasing how to experience the Build Scanยฎ service of Develocity with sbt
Scala
3
star
94

dpeuni-gradle-intro-devs-tasks

Hands-on exercise for DPE University
Java
3
star
95

declarative-samples-android-app

A sample Android application written in the Declarative Gradle DSL, using the prototype Declarative Android `androidApplication` Software Type defined in the `org.gradle.experimental.android-ecosystem` ecosystem plugin.
Kotlin
3
star
96

gradle-groovy-all

Replacement for groovy-all.jar discontinued in Groovy 2.5. This is intended to be used with Gradle only.
Kotlin
3
star
97

declarative-samples-java-app

A sample Java application written in the Declarative Gradle DSL, using the prototype Declarative Java java-application ecosystem plugin.
Java
2
star
98

apachecon2021

Instructions and code repository for the ApacheCon 2021 Gradle Virtual Booth Event
2
star
99

kotlin-relocation-test

Cache relocatability test for Kotlin using Spek
Groovy
2
star
100

github-dependency-submission-demo

Java
2
star