Skip to content

Duplicate test method name in suite causes invalid test result #3264

New issue
New issue
Closed
Closed
Duplicate test method name in suite causes invalid test result#3264
Labels
bugSomething isn't working

Description

@Joris-Lambrechts-CCVLab
Joris-Lambrechts-CCVLab
opened on Feb 16, 2026

Affected version

3.5.4

Bug description

Hello,

We had some issues with tests not causing builds to fail. After debugging the plugin, I found that the reported test is a combination of the test suite and test method name. The test class name is not used. As a result, if we have multiple test classes with test methods that have identical names, the reported result of one test affects the other.

I've attached a little sample that reproduces the behaviour. I expect the build to fail, but it does not.

[INFO] Scanning for projects...
[INFO] 
[INFO] -------------< org.apache.maven:maven-surefire-plugin-bug >-------------
[INFO] Building Archetype - maven-surefire-plugin-bug 1.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ maven-surefire-plugin-bug ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 4 resources from src/main/resources to target/classes
[INFO] 
[INFO] --- compiler:3.13.0:compile (default-compile) @ maven-surefire-plugin-bug ---
[INFO] No sources to compile
[INFO] 
[INFO] --- resources:3.3.1:testResources (default-testResources) @ maven-surefire-plugin-bug ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/jorislambrechts/Desktop/maven-surefire-plugin-bug/src/test/resources
[INFO] 
[INFO] --- compiler:3.13.0:testCompile (default-testCompile) @ maven-surefire-plugin-bug ---
[INFO] Recompiling the module because of changed source code.
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 3 source files with javac [debug target 1.8] to target/test-classes
[WARNING] bootstrap class path is not set in conjunction with -source 8
  not setting the bootstrap class path may lead to class files that cannot run on JDK 8
    --release 8 is recommended instead of -source 8 -target 1.8 because it sets the bootstrap class path automatically
[WARNING] source value 8 is obsolete and will be removed in a future release
[WARNING] target value 8 is obsolete and will be removed in a future release
[WARNING] To suppress warnings about obsolete options, use -Xlint:-options.
[INFO] 
[INFO] --- surefire:3.5.4:test (default-test) @ maven-surefire-plugin-bug ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[INFO] Running org.apache.maven.surefire.SuccessTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.016 s <<< FAILURE! -- in org.apache.maven.surefire.SuccessTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.011 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s -- in org.apache.maven.surefire.FailTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.028 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.003 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.003 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s -- in org.apache.maven.surefire.RunSuite
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Flakes: 
[WARNING] org.apache.maven.surefire.RunSuite.test
[INFO]   Run 1: PASS
[ERROR]   Run 2: RunSuite.test
[ERROR]   Run 3: RunSuite.test
[ERROR]   Run 4: RunSuite.test
[ERROR]   Run 5: RunSuite.test
[ERROR]   Run 6: RunSuite.test
[ERROR]   Run 7: RunSuite.test
[INFO] 
[INFO] 
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] 
[INFO] 
[INFO] --- surefire:3.5.4:test (integration-suite) @ maven-surefire-plugin-bug ---
[INFO] Skipping execution of surefire because it has already been run for this configuration
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.512 s
[INFO] Finished at: 2026-02-16T16:36:47+01:00
[INFO] ------------------------------------------------------------------------

I did not create a PR because I'm not sure whether this is as intended or if I'm configuring the plugin incorrectly.

Thanks,
Joris

maven-surefire-plugin-bug.zip

Activity

  1. Joris-Lambrechts-CCVLab
    added
    bugSomething isn't working
    on Feb 16, 2026

  2. Joris-Lambrechts-CCVLab commented on Feb 17, 2026

    @Joris-Lambrechts-CCVLab
    Joris-Lambrechts-CCVLab
    on Feb 17, 2026
    Author

    We notice that this is not the case when there is a @ParameterizedTest that has the same method name as a normal @Test.

    class SuccessTest {

    @ValueSource(strings = {"foo"})
    @ParameterizedTest
    void test(String test) {
        Assertions.assertNotNull(test);
    }

}

    [INFO] --- surefire:3.5.4:test (default-test) @ maven-surefire-plugin-bug ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[INFO] Running org.apache.maven.surefire.SuccessTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.018 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.012 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 s -- in org.apache.maven.surefire.SuccessTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.034 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.002 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s -- in org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.RunSuite
[INFO] Running org.apache.maven.surefire.FailTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! -- in org.apache.maven.surefire.FailTest
[ERROR] org.apache.maven.surefire.RunSuite.test -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:37)
	at org.junit.jupiter.api.Assertions.fail(Assertions.java:122)
	at org.apache.maven.surefire.FailTest.test(FailTest.java:11)

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s -- in org.apache.maven.surefire.RunSuite
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR] org.apache.maven.surefire.RunSuite.test
[ERROR]   Run 1: RunSuite.test
[ERROR]   Run 2: RunSuite.test
[ERROR]   Run 3: RunSuite.test
[ERROR]   Run 4: RunSuite.test
[ERROR]   Run 5: RunSuite.test
[ERROR]   Run 6: RunSuite.test
[INFO] 
[INFO] 
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.207 s
[INFO] Finished at: 2026-02-17T13:28:44+01:00
[INFO] ------------------------------------------------------------------------
  3. mirkoalicastro
    added 2 commits that reference this issue on Apr 8, 2026

    Fixes apache#3264 : tests inside @suite incorrectly classified as flakes

    fabffc2

    Fixes apache#3264 : tests inside @suite incorrectly classified as flakes

    878db1d
  4. olamy
    closed this as completedin ce23cecon Aug 6, 2026
  5. pull
    added a commit that references this issue on Aug 6, 2026

    Fixes apache#3264 : tests inside @suite incorrectly classified as fla…

    ce23cec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions