• Stars
    star
    199
  • Rank 187,799 (Top 4 %)
  • Language
    Groovy
  • License
    Apache License 2.0
  • Created over 4 years ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Gradle plugin to retry tests that have failed to mitigate test flakiness.

Test Retry Gradle plugin

A Gradle plugin that augments Gradle’s built-in test task with the ability to retry tests that have failed.

Version GitHub license

What it does

The plugin causes failed tests to be retried within the same task. After executing all tests, any failed tests are retried. The process repeats with tests that continue to fail until the maximum specified number of retries has been attempted, or there are no more failing tests.

By default, all failed tests passing on retry prevents the test task from failing. This mode prevents flaky tests from causing build failure. This setting can be changed so that flaky tests cause build failure, which can be used to identify flaky tests.

When something goes badly wrong and all tests start failing, it can be preferable to not keep retrying tests. This can happen for example if a disk fills up or a required database is not available. To avoid this, the plugin can be configured to stop retrying after a certain number of total test failures.

NOTE: Retrying tests alone is not a viable flaky test mitigation strategy. This plugin should only be used alongside processes for tracking and fixing discovered flaky tests.

Usage

Apply the plugin using one of the two methods described on the Gradle Plugin Portal, where the plugin is listed as org.gradle.test-retry. It is compatible with Gradle 5.0 and later versions.

By default, retrying is not enabled.

Retrying is configured per test task via the retry extension added to each task by the plugin.

build.gradle:
test {
    retry {
        maxRetries = 2
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}
build.gradle.kts:
test {
    retry {
        maxRetries.set(2)
        maxFailures.set(20)
        failOnPassedAfterRetry.set(true)
    }
}

Limiting retry to CI builds

You may find that local developer builds do not benefit much from retry behaviour, particularly when those tests are invoked via your IDE. In that case we recommend enabling retry only for CI builds.

build.gradle:
boolean isCiServer = System.getenv().containsKey("CI")
test {
    retry {
        if (isCiServer) {
            maxRetries = 2
            maxFailures = 20
        }
        failOnPassedAfterRetry = true
    }
}

The retry extension

The retry extension is of the following type:

package org.gradle.testretry;

import org.gradle.api.Action;
import org.gradle.api.provider.Property;
import org.gradle.api.provider.SetProperty;
import org.gradle.api.tasks.testing.Test;

/**
 * Allows configuring test retry mechanics.
 * <p>
 * This extension is added with the name 'retry' to all {@link Test} tasks.
 */
public interface TestRetryTaskExtension {

    /**
     * The name of the extension added to each test task.
     */
    String NAME = "retry";

    /**
     * Whether tests that initially fail and then pass on retry should fail the task.
     * <p>
     * This setting defaults to {@code false},
     * which results in the task not failing if all tests pass on retry.
     * <p>
     * This setting has no effect if {@link Test#getIgnoreFailures()} is set to true.
     *
     * @return whether tests that initially fails and then pass on retry should fail the task
     */
    Property<Boolean> getFailOnPassedAfterRetry();

    /**
     * The maximum number of times to retry an individual test.
     * <p>
     * This setting defaults to {@code 0}, which results in no retries.
     * Any value less than 1 disables retrying.
     *
     * @return the maximum number of times to retry an individual test
     */
    Property<Integer> getMaxRetries();

    /**
     * The maximum number of test failures that are allowed before retrying is disabled.
     * <p>
     * The count applies to each round of test execution.
     * For example, if maxFailures is 5 and 4 tests initially fail and then 3 again on retry,
     * this will not be considered too many failures and retrying will continue (if maxRetries {@literal >} 1).
     * If 5 or more tests were to fail initially then no retry would be attempted.
     * <p>
     * This setting defaults to {@code 0}, which results in no limit.
     * Any value less than 1 results in no limit.
     *
     * @return the maximum number of test failures that are allowed before retrying is disabled
     */
    Property<Integer> getMaxFailures();

    /**
     * The filter for specifying which tests may be retried.
     */
    Filter getFilter();

    /**
     * The filter for specifying which tests may be retried.
     */
    void filter(Action<? super Filter> action);

    /**
     * A filter for specifying which tests may be retried.
     *
     * By default, all tests are eligible for retrying.
     */
    interface Filter {

        /**
         * The patterns used to include tests based on their class name.
         *
         * The pattern string matches against qualified class names.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class name only has to match one pattern to be included.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeClasses();

        /**
         * The patterns used to include tests based on their class level annotations.
         *
         * The pattern string matches against the qualified class names of a test class's annotations.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class need only have one annotation matching any of the patterns to be included.
         *
         * Annotations present on super classes that are {@code @Inherited} are considered when inspecting subclasses.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeAnnotationClasses();

        /**
         * The patterns used to exclude tests based on their class name.
         *
         * The pattern string matches against qualified class names.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class name only has to match one pattern to be excluded.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getExcludeClasses();

        /**
         * The patterns used to exclude tests based on their class level annotations.
         *
         * The pattern string matches against the qualified class names of a test class's annotations.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class need only have one annotation matching any of the patterns to be excluded.
         *
         * Annotations present on super classes that are {@code @Inherited} are considered when inspecting subclasses.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getExcludeAnnotationClasses();

    }


    /**
     * The set of criteria specifying which test classes must be retried as a whole unit
     * if retries are enabled and the test class passes the configured filter.
     */
    ClassRetryCriteria getClassRetry();

    /**
     * The set of criteria specifying which test classes must be retried as a whole unit
     * if retries are enabled and the test class passes the configured filter.
     */
    void classRetry(Action<? super ClassRetryCriteria> action);

    /**
     * The set of criteria specifying which test classes must be retried as a whole unit
     * if retries are enabled and the test class passes the configured filter.
     */
    interface ClassRetryCriteria {

        /**
         * The patterns used to include tests based on their class name.
         *
         * The pattern string matches against qualified class names.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class name only has to match one pattern to be included.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeClasses();

        /**
         * The patterns used to include tests based on their class level annotations.
         *
         * The pattern string matches against the qualified class names of a test class's annotations.
         * It may contain '*' characters, which match zero or more of any character.
         *
         * A class need only have one annotation matching any of the patterns to be included.
         *
         * Annotations present on super classes that are {@code @Inherited} are considered when inspecting subclasses.
         *
         * If no patterns are specified, all classes (that also meet other configured filters) will be included.
         */
        SetProperty<String> getIncludeAnnotationClasses();

    }

}

Supported test frameworks

Other versions are likely to work as well, but are not tested.

Framework Version Tested

JUnit4

4.13.2

JUnit5

5.9.2

Spock

2.3-groovy-3.0

TestNG

7.5

Parameterized tests

In a few cases, test selection for testing frameworks limits the granularity at which tests can be retried. In each case, this plugin retries at worst at method level. For JUnit5 @ParameterizedTest, TestNG @Test(dataProvider = "…​"), and Spock @Unroll tests the plugin will retry the entire method with all parameters including those that initially passed.

Test dependencies

The plugin supports retrying Spock @Stepwise tests and TestNG @Test(dependsOn = { … }) tests.

  • Upstream tests (those that the failed test depends on) are run because a flaky test may depend on state from the prior execution of an upstream test.

  • Downstream tests are run because a flaky test causes any downstream tests to be skipped in the initial test run.

Custom test frameworks

Some projects may use test tasks with a custom TestFramework to execute tests. If this is the case, the plugin disables retries and emits the following warning:

> Task :unsupportedTestTaskUnitTest
Test retry requested for task :unsupportedTestTaskUnitTest with unsupported test framework CustomTestFramework - failing tests will not be retried

To avoid this warning, we can disable retries for the unsupported test task with:

build.gradle:
test.named('unsupportedTestTaskUnitTest') {
    retry {
        maxRetries = 0
    }
}
build.gradle.kts:
tasks.named<Test>("unsupportedTestTaskUnitTest") {
    retry {
        maxRetries.set(0)
    }
}

Filtering

By default, all tests are eligible for retrying. The filter component of the test retry extension can be used to control which tests should be retried and which should not.

The decision to retry a test or not is based on the tests reported class name, regardless of the name of the test case or method. The annotations present or not on this class can also be used as the criteria.

build.gradle:
test {
    retry {
        maxRetries = 3
        filter {
            // filter by qualified class name (* matches zero or more of any character)
            includeClasses.add("*IntegrationTest")
            excludeClasses.add("*DatabaseTest")

            // filter by class level annotations
            // Note: @Inherited annotations are respected
            includeAnnotationClasses.add("*Retryable")
            excludeAnnotationClasses.add("*NonRetryable")
        }
    }
}

Retry on class-level

By default, individual tests are retried. The classRetry component of the test retry extension can be used to control which test classes must be retried as a whole unit. Test classes still have to pass the configured filter.

build.gradle:
test {
    retry {
        maxRetries = 3
        classRetry {
            // configure by qualified class name (* matches zero or more of any character)
            includeClasses.add("*StepWiseIntegrationTest")

            // configure by class level annotations
            // Note: @Inherited annotations are respected
            includeAnnotationClasses("Stepwise")
        }
    }
}

Reporting

Gradle

Each execution of a test is discretely reported in Gradle-generated XML and HTML reports.

Gradle test reporting

Gradle flaky test reporting

Similar to the XML and HTML reports, the console log will also report each individual test execution. Before retrying a failed test, Gradle will execute the whole test suite of the test task. This means that all executions of the same test may not be grouped in the console log.

Gradle console reporting

Gradle Enterprise

Gradle build scans (--scan option) report discrete test executions as "Execution [N of total]" and will mark a test with both a failed and then a passed outcome as flaky.

Gradle build scan reporting

Flaky tests can also be visualized across many builds using the Gradle Enterprise Tests Dashboard.

Gradle Enterprise top tests report

IDEs

The plugin has been tested with IDEA, Eclipse IDE and Netbeans.

IDEA

When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.

IDEA test reporting

Eclipse

When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.

Eclipse test reporting

Netbeans

Netbeans only shows the last execution of a test.

Netbeans test reporting

CI tools

The plugin has been tested with the reporting of TeamCity and Jenkins.

TeamCity

Flaky tests (tests being executed multiple times but with different results) are detected by TeamCity and marked as flaky. TeamCity lists each test that was executed and how often it was run in the build.

By default, TeamCity will fail your build if at least one test fails. When using failOnPassedAfterRetry = false (ie. the default for this plugin), this failure condition should be disabled.

Teamcity test reporting

Jenkins

Jenkins reports each test execution discretely.

Jenkins test reporting

More Repositories

1

gradle

Adaptable, fast automation for all
Groovy
15,973
star
2

kotlin-dsl-samples

Samples builds using the Gradle Kotlin DSL
Kotlin
3,701
star
3

gradle-profiler

A tool for gathering profiling and benchmarking information for Gradle builds
Java
1,279
star
4

gradle-completion

Gradle tab completion for bash and zsh
Shell
966
star
5

gradle-build-action

Execute your Gradle build and trigger dependency submission
639
star
6

android-cache-fix-gradle-plugin

Gradle plugin that fixes Android build caching problems.
Groovy
440
star
7

wrapper-validation-action

Gradle Wrapper Validation Action
TypeScript
249
star
8

oreilly-gradle-book-examples

Java
249
star
9

native-samples

Samples of modern build automation for native languages with Gradle
Java
148
star
10

gradle-build-scan-quickstart

An example project to experience the Build Scan® service of Gradle Enterprise with Gradle builds.
Kotlin
132
star
11

native-platform

Java bindings for various native APIs
Java
111
star
12

gradle-native

The home of Gradle's support for natively compiled languages
91
star
13

foojay-toolchains

Java Toolchain Resolve Plugin based on the foojay DiscoAPI
Kotlin
88
star
14

build-tool-training-exercises

Exercises for live gradle.com/training sessions
Java
88
star
15

gradle-enterprise-build-validation-scripts

Executable scripts to assist in validating that your Gradle and Maven builds are in an optimal state in terms of maximizing work avoidance when using Develocity.
Shell
70
star
16

github-dependency-graph-gradle-plugin

Gradle Plugin for Extracting Dependency Information to send to GitHub
Groovy
65
star
17

develocity-build-config-samples

Code samples that demonstrate how to customize your Develocity build configuration using Gradle, Maven, Bazel or sbt
Java
62
star
18

actions

A collection of GitHub Actions to accelerate your Gradle Builds on GitHub
TypeScript
58
star
19

playframework

Gradle Play Support
Java
47
star
20

guides

The Gradle Guides at https://guides.gradle.org.
Java
47
star
21

gradle-site-plugin

Kotlin
42
star
22

wrapper-upgrade-gradle-plugin

Gradle plugin that detects and updates Gradle and Maven wrappers to the latest Gradle and Maven version.
Java
39
star
23

kotlin-dsl-docs

Generates Kotlin DSL API reference
Kotlin
37
star
24

build-tool-roadmap

Gradle Build Tool roadmap
35
star
25

common-custom-user-data-gradle-plugin

Gradle plugin that enhances published build scans by adding a set of tags, links and custom values that have proven to be useful for many projects building with Develocity.
Java
35
star
26

gradle-talks

A javascript based custom slide and build framework for presentations. Many of the Gradle engineers have been using this for their presentations. Those presentations are part of this repo and can be found in the talks directory.
Ruby
35
star
27

exemplar

Discover and verify code samples and services
Java
34
star
28

gradle2kts

Gradle Groovy to Gradle Kotlin conversion tool - discontinued spike
Kotlin
34
star
29

gradle-checksum

A Gradle plugin for creating checksums for files in your build.
Groovy
31
star
30

kotlin-dsl-conventions

Gradle Kotlin DSL conventional plugins
Kotlin
21
star
31

maven-build-scan-quickstart

An example project to experience the Build Scan® service of Develocity with Maven builds.
Java
20
star
32

bazel-comparison

20
star
33

develocity-oss-projects

18
star
34

performance-comparisons

A set of synthetic projects used to benchmark Gradle against other build tools
16
star
35

gradle-java-modules

A (former) place for experimenting with Java 9's module system. Gradle officially supports Java Modules since version 6.4
Java
16
star
36

cc-hackathon-2022

Configuration Cache Hackathon 2022
15
star
37

perf-enterprise-large

A large Java based build to use when benchmarking and profiling Gradle
Groovy
15
star
38

.github

Maintains all of the default policies for the Gradle organization
14
star
39

gradle-enterprise-export-api-samples

A repository of samples that demonstrate how to use the Gradle Enterprise Export API.
14
star
40

gradle-jdocbook

A Gradle plugin for jdocbook
Groovy
13
star
41

multi-project-composite-gradle-plugins-builds

Multi-project build using composite Gradle plugins
Kotlin
12
star
42

webinar-getting-started-with-the-gradle-kotlin-dsl

Webinar - Getting Started with the Kotlin DSL
Kotlin
12
star
43

gradle-distributions

Repository for Gradle Build Tool distributions downloads
12
star
44

gcc2speedscope

Space usage analysis for the Gradle configuration cache via speedscope
Kotlin
12
star
45

develocity-api-samples

A repository of samples that demonstrate how to use the Develocity API.
Java
12
star
46

speed-challenge

Instructions and overview for the Gradle and Maven Speed Challenge event
11
star
47

cucumber-companion

Maven & Gradle plugins providing convenient support for running Cucumber test directly from Maven/Gradle
Groovy
11
star
48

imaginate

Using the Kotlin language for production, test and build makes it easier for everyone to work with your software code base. Let’s go 100% Kotlin!
Kotlin
11
star
49

plugin-portal-requests

Gradle Plugin Portal issues and requests.
11
star
50

continuous-delivery-jump-start

Sample application used for training "Continuous Delivery Jump Start"
Java
11
star
51

declarative-gradle

Java
10
star
52

build-analysis-demo

Build data analysis applications
Kotlin
10
star
53

santa-tracker-performance

Performance tests for Santa Tracker Android project
Shell
9
star
54

gradle-org-conventions-plugin

Java
9
star
55

perf-android-large-2

Another large Android build for performance experiments
9
star
56

gradle-hello-world-plugin

Groovy
9
star
57

gradle-issue-reproducer

Template repository for providing Gradle issue reproducers
9
star
58

tooling-commons

A small layer on top of the Gradle Tooling API that provides the Tooling Client and other convenience useful for IDE integration.
Java
9
star
59

develocity-testing-annotations

Common annotations for Develocity and Test Retry
Java
8
star
60

gradle-profiler-plugin

Java
8
star
61

jfr-polyfill

A polyfill for JDK Flight Recorder (JFR) to avoid errors on JDKs that don't support JFR yet
Java
8
star
62

configuration-cache-report

Kotlin
7
star
63

gradle-benchmark-base

Base scenarios for Gradle Profiler to benchmark Gradle builds
7
star
64

common-custom-user-data-maven-extension

Maven extension that enhances published build scans by adding a set of tags, links and custom values that have proven to be useful for many projects building with Develocity.
Java
7
star
65

bt-dev-prod-data-collector

Data collector for Gradle Build Tool Developer productivity metrics
Kotlin
7
star
66

gradle-hazelcast-plugin

Groovy
6
star
67

gradle-enterprise-build-optimization-experiments

Self-guiding experiments to optimize the performance of your Gradle and Maven builds with Gradle Enterprise.
6
star
68

gradle-rules-configuration-workshop

6
star
69

maven-build-cache-unstable-inputs-samples

An example project containing a build with unstable inputs
Java
6
star
70

gradle-project-templates

Learning day experiment: project init templates for Gradle
Java
5
star
71

source-resolution-demo

Java
5
star
72

greeting-plugin-example

Java
5
star
73

gradle-dependency-constrain

Java
5
star
74

develocity-bamboo-plugin

Develocity plugin for Bamboo
Java
4
star
75

community

Gradle Community content
HTML
4
star
76

stable-plugins-dsl

Getting the plugins {} DSL block out of incubation
Java
4
star
77

perf-native-large

A Place to Profile Particularly Prickly Projects
C
4
star
78

github-actions

Shared Github Actions
TypeScript
3
star
79

ide-smoke-tests

Java
3
star
80

gradle-all

A composite build that includes all the pieces of Gradle
Kotlin
3
star
81

develocity-gitlab-templates

GitLab CI/CD templates to automatically connect Gradle/Maven builds to Develocity
3
star
82

ge-export

Java
3
star
83

webinar-dep-mgmt-part-1

Kotlin
3
star
84

integrations

A repository for Gradle Integrations and the related knowledge base
3
star
85

gradle-performance

Gradle performance benchmarks using the Gradle build tool itself
Shell
3
star
86

provider-api-migration-testbed

A testbed to try out mitigation strategies for the provider API migration
Kotlin
3
star
87

gradle-model-vis

JavaScript
3
star
88

gradle-groovy-all

Replacement for groovy-all.jar discontinued in Groovy 2.5. This is intended to be used with Gradle only.
Kotlin
3
star
89

apachecon2021

Instructions and code repository for the ApacheCon 2021 Gradle Virtual Booth Event
2
star
90

kotlin-relocation-test

Cache relocatability test for Kotlin using Spek
Groovy
2
star
91

api-evolution-demo

Demo of API evolution tricks
Java
2
star
92

Compilation-Avoidance-Experiment

A sample project for running comparative performance tests on build tools
Java
2
star
93

webinar-gradle-6

Webcast: What's New In Gradle 6.0?
Kotlin
2
star
94

ge-customer-support-zendesk-theme

Our custom zendesk theme used for support.gradle.com
Handlebars
2
star
95

build-tool-issue-automation-actions

Actions for Github issue automation
JavaScript
2
star
96

gradle-20388-multiple-toml

Example for solving gradle issue #20388
Kotlin
2
star
97

native-samples-cpp-library

A C++ library referenced by the Gradle C++ source dependencies samples
C++
2
star
98

impact_analysis

2
star
99

model-app

Java
2
star
100

native-samples-swift-library

A Swift library referenced by the Gradle Swift source dependencies samples
Swift
2
star