Cucumber scenarios still reported as Tests run: 0 when junit-platform < 1.8 (#828 fix is version-gated) #3428

Affected version

3.5.3 through 3.5.6, and current master

Bug description

Cucumber scenarios are reported as Tests run: 0 when the JUnit Platform on the test classpath is older than 1.8, even though the scenarios execute and print normally. The XML report has tests="0" and no <testcase> elements.

This is a follow-up to #834 / #840 / #853 / #857 / #3166. Those were fixed by #828 in 3.5.4, but the fix is silently version-gated: it depends on TestIdentifier.getParentIdObject(), which only exists from JUnit Platform 1.8. On anything older it no-ops and the original 3.5.3 behaviour comes back.

Reproduction

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0">
  <modelVersion>4.0.0</modelVersion>
  <groupId>demo</groupId><artifactId>repro</artifactId><version>1.0</version>
  <properties>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <sf.version>3.5.6</sf.version>
    <junit5.version>5.7.2</junit5.version>
    <platform.version>1.7.2</platform.version>
  </properties>
  <dependencies>
    <dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.13.2</version><scope>test</scope></dependency>
    <dependency><groupId>io.cucumber</groupId><artifactId>cucumber-java</artifactId><version>7.18.1</version><scope>test</scope></dependency>
    <dependency><groupId>io.cucumber</groupId><artifactId>cucumber-junit</artifactId><version>7.18.1</version><scope>test</scope></dependency>
    <dependency><groupId>org.junit.vintage</groupId><artifactId>junit-vintage-engine</artifactId><version>${junit5.version}</version><scope>test</scope></dependency>
    <dependency><groupId>org.junit.platform</groupId><artifactId>junit-platform-launcher</artifactId><version>${platform.version}</version><scope>test</scope></dependency>
  </dependencies>
  <build><plugins>
    <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>${sf.version}</version></plugin>
    <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-failsafe-plugin</artifactId><version>${sf.version}</version>
      <executions><execution><goals><goal>integration-test</goal><goal>verify</goal></goals></execution></executions></plugin>
  </plugins></build>
</project>

src/test/java/com/repro/CucumberIT.java:

package com.repro;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(features = "classpath:com/repro", glue = "com.repro", plugin = {"pretty"})
public class CucumberIT {}

src/test/java/com/repro/StepDefinitions.java:

package com.repro;

import io.cucumber.java.en.Given;

public class StepDefinitions {
    @Given("a step that always passes")
    public void a_step_that_always_passes() {}
}

src/test/resources/com/repro/test.feature:

Feature: dummy

  Scenario: Run a dummy cucumber test
    Given a step that always passes

Then mvn verify.

Expected: Tests run: 1. Actual: Tests run: 0, and target/failsafe-reports/TEST-com.repro.CucumberIT.xml contains tests="0" with no <testcase>.

Version matrix

All rows on Surefire/Failsafe 3.5.6, same project, only the JUnit versions changing:

Mixed versions fail too: engine 1.7.2 with junit-platform-launcher declared at 1.10.3, and engine 1.10.3 with launcher at 1.7.2, both report 0. JUnitPlatformProviderInfo.alignProviderVersions re-resolves the launcher to match the detected junit-platform-commons, so bumping the launcher alone doesn't change what runs:

[DEBUG] Resolving artifact org.junit.platform:junit-platform-launcher:1.7.2

The same project on 3.5.2 with platform 1.7.2 reports Tests run: 1, so this is a regression rather than a new baseline requirement.

Root cause

3.5.3 (#815, SUREFIRE-1643) made TestSetRunListener bucket stats per ReportEntry.getSourceName():

private TestSetStats getTestSetStats(ReportEntry report) {
    if (statPerSourceName) {
        return detailsPerSource.computeIfAbsent(
                report.getSourceName(), s -> new TestSetStats(trimStackTrace, isPlainFormat));
    }
    return currentTestSetStats;
}

Cucumber's scenario descriptors carry no ClassSource, so their source name is derived from the feature's display name, while testSetCompleted arrives under the runner class name. Two different keys, and the flush reads an empty bucket.

#828 fixed that by resolving the class-level name from the top-level parent. But the walk gives up when the 1.8 API is absent (RunListenerAdapter):

private TestIdentifier findTopParent(TestIdentifier testIdentifier) {
    if (!hasParentId(testIdentifier)) {
        return testIdentifier;
    }
    ...
}

private boolean hasParentId(TestIdentifier testIdentifier) {
    Method getParentIdObjectMethod = ReflectionUtils.tryGetMethod(testIdentifier.getClass(), "getParentIdObject");
    if (getParentIdObjectMethod == null) {
        return false;
    }
    ...
}

On a pre-1.8 launcher, hasParentId returns false, findTopParent hands back the scenario descriptor itself, that has no ClassSource, classLevelName stays empty, and toClassMethodName falls through to the feature display name. Same mismatch as 3.5.3.

Suggested fix

TestPlan.getParent(TestIdentifier) has been in the API since 1.0 and does the same job as the reflective getParentIdObject path. Having findTopParent fall back to it instead of returning the input would make the #828 fix work on every supported platform version.