Repository navigation
[SUREFIRE-2148] Build fails if retried test classes failed during setup #2749
Description
Pavlo Shevchenko opened SUREFIRE-2148 and commented
Summary
If a JUnit5 test class fails during setup and succeeds on retry, then the surefire test goal and entire build will fail. On the other hand, if the failure occurs in a test method which succeeds on retry, then the goal and the build will succeed.
Reproducer
Test class with flaky setup:
package example;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
public class FlakyClassTest {
@BeforeAll
public static void setup() throws IOException {
String testSetupMarkerFile = "testSetupMarker.txt";
if (!new File(testSetupMarkerFile).exists()) {
System.out.println("I'm failing!");
Files.write(Paths.get(testSetupMarkerFile), "Hello".getBytes());
throw new RuntimeException("I'm failing!");
} else {
System.out.println("I'm passing!");
}
}
@Test
public void test() {
}
}
Test class with flaky method:
package example;
import org.junit.jupiter.api.Test;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
public class FlakyMethodTest {
@Test
public void test() throws IOException {
String testMethodMarkerFile = "testMethodMarker.txt";
if (!new File(testMethodMarkerFile).exists()) {
System.out.println("I'm failing!");
Files.write(Paths.get(testMethodMarkerFile), "Hello".getBytes());
throw new RuntimeException("I'm failing!");
} else {
System.out.println("I'm passing!");
}
}
}Surefire config:
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M8</version>
<configuration>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
</configuration>
</plugin>
Actual behavior
- This execution succeds:
mvn test -Dtest=*FlakyMethod*- This execution fails:
mvn test -Dtest=*FlakyClass*
Expected behavior
- Both executions succeed
Affects: 3.0.0-M8
1 votes, 3 watchers
Reacted by Mauro Molinari
Activity
jira-importer commentedon Oct 10, 2024
Pavlo Shevchenko commented
Just stumbled over it again. It is a regression between 3.0.0-M4 and 3.0.0-M5. As of 3.5.1, it is still not working
mauromol commentedon Oct 9, 2025
I confirm this problem with Surefire Plugin 3.5.1. Is there any plan to address it?
[SUREFIRE-2148] Verify recovered BeforeAll does not fail build (apach…