在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称(OpenSource Name):gradle/test-retry-gradle-plugin开源软件地址(OpenSource Url):https://github.com/gradle/test-retry-gradle-plugin开源编程语言(OpenSource Language):Groovy 52.8%开源软件介绍(OpenSource Introduction):Test Retry Gradle pluginA Gradle plugin that augments Gradle’s built-in test task with the ability to retry tests that have failed. What it doesThe plugin causes failed tests to be retried within the same task. After executing all tests, any failed tests are retried. The process repeats with tests that continue to fail until the maximum specified number of retries has been attempted, or there are no more failing tests. By default, all failed tests passing on retry prevents the test task from failing. This mode prevents flaky tests from causing build failure. This setting can be changed so that flaky tests cause build failure, which can be used to identify flaky tests. When something goes badly wrong and all tests start failing, it can be preferable to not keep retrying tests. This can happen for example if a disk fills up or a required database is not available. To avoid this, the plugin can be configured to stop retrying after a certain number of total test failures. NOTE: Retrying tests alone is not a viable flaky test mitigation strategy. This plugin should only be used alongside processes for tracking and fixing discovered flaky tests. UsageApply the plugin using one of the two methods described on the Gradle Plugin Portal, where the plugin is listed as By default, retrying is not enabled. Retrying is configured per test task via the build.gradle:
test {
retry {
maxRetries = 2
maxFailures = 20
failOnPassedAfterRetry = true
}
} build.gradle.kts:
test {
retry {
maxRetries.set(2)
maxFailures.set(20)
failOnPassedAfterRetry.set(true)
}
} Limiting retry to CI buildsYou may find that local developer builds do not benefit much from retry behaviour, particularly when those tests are invoked via your IDE. In that case we recommend enabling retry only for CI builds. build.gradle:
boolean isCiServer = System.getenv().containsKey("CI")
test {
retry {
if (isCiServer) {
maxRetries = 2
maxFailures = 20
}
failOnPassedAfterRetry = true
}
}
The |
Framework | Version Tested |
---|---|
JUnit4 |
4.13.2 |
JUnit5 |
5.8.2 |
Spock |
2.0-groovy-3.0 |
TestNG |
7.4.0 |
In a few cases, test selection for testing frameworks limits the granularity at which tests can be retried.
In each case, this plugin retries at worst at method level.
For JUnit5 @ParameterizedTest
, TestNG @Test(dataProvider = "…")
,
and Spock @Unroll
tests the plugin will retry the entire method with all parameters including those that initially passed.
The plugin supports retrying Spock @Stepwise
tests and TestNG @Test(dependsOn = { … })
tests.
Upstream tests (those that the failed test depends on) are run because a flaky test may depend on state from the prior execution of an upstream test.
Downstream tests are run because a flaky test causes any downstream tests to be skipped in the initial test run.
By default, all tests are eligible for retrying.
The filter
component of the test retry extension can be used to control which tests should be retried and which should not.
The decision to retry a test or not is based on the tests reported class name, regardless of the name of the test case or method. The annotations present or not on this class can also be used as the criteria.
test {
retry {
maxRetries = 3
filter {
// filter by qualified class name (* matches zero or more of any character)
includeClasses.add("*IntegrationTest")
excludeClasses.add("*DatabaseTest")
// filter by class level annotations
// Note: @Inherited annotations are respected
includeAnnotationClasses.add("*Retryable")
excludeAnnotationClasses.add("*NonRetryable")
}
}
}
Each execution of a test is discretely reported in Gradle-generated XML and HTML reports.
Similar to the XML and HTML reports, the console log will also report each individual test execution. Before retrying a failed test, Gradle will execute the whole test suite of the test task. This means that all executions of the same test may not be grouped in the console log.
Gradle build scans (--scan
option) report discrete test executions as "Execution [N of total]" and will mark a test with both a failed and then a passed outcome as flaky.
Flaky tests can also be visualized across many builds using the Gradle Enterprise Tests Dashboard.
The plugin has been tested with IDEA, Eclipse IDE and Netbeans.
When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.
When delegating test execution to Gradle, each execution is reported discretely as for the test reports. Running tests without Gradle delegation causes tests to not be retried.
Flaky tests (tests being executed multiple times but with different results) are detected by TeamCity and marked as flaky. TeamCity lists each test that was executed and how often it was run in the build.
By default, TeamCity will fail your build if at least one test fails.
When using failOnPassedAfterRetry = false
(ie. the default for this plugin), this failure condition should be disabled.
2023-10-27
2022-08-15
2022-08-17
2022-09-23
2022-08-13
请发表评论