ConcurrentEventListener handler for TestStepStarted event is accessing steps after scenario failure

[Venomicon]

Hi,
we're using ConcurrentEventListener to access Step text via TestStepStarted event listener. These steps can later be logged to a file and attached to the failed test in the Allure report. Unfortunately, it does not work as expected.

👓 What did you see?

All steps for given scenario are being logged, even the ones that occur after the scenario failure.

##[SECTION]	 Starting scenario d950c44e-1029-40ae-8896-91c1c72414ec
##[STEP]	 02/29/2024 09:53:11 --- action ---
##[STEP]	 02/29/2024 09:53:11 --- check result ---
##[ERROR]	 02/29/2024 09:53:11 --- Step above is failing! ---
##[STEP]	 02/29/2024 09:53:11 --- rest ---
##[STEP]	 02/29/2024 09:53:11 --- of ---
##[STEP]	 02/29/2024 09:53:11 --- the ---
##[STEP]	 02/29/2024 09:53:11 --- scenario ---
##[SECTION]	 Scenario ended with status: FAILED

java.lang.AssertionError: 
Expecting value to be true but was false

	at org.steps.CommonSteps.checkResult(CommonSteps.java:163)
	at ✽.check result(file:///C:/Projects/ConcurrentEventListenerBug/src/test/java/org/features/ConcurrentEventListener.feature:440)


Failed scenarios:
file:///C:/Projects/ConcurrentEventListenerBug/src/test/java/org/features/ConcurrentEventListener.feature:438 # ConcurrentEventListener bug

✅ What did you expect to see?

I expect, that steps after the failed step are not logged, because as name suggests for "TestStepStarted" event - these steps have not yet started because of the failure.

📦 Which tool/library version are you using?

<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-java</artifactId>
    <version>7.15.0</version>
</dependency>
<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-testng</artifactId>
    <version>7.15.0</version>
</dependency>
<dependency>
    <groupId>io.cucumber</groupId>
    <artifactId>cucumber-picocontainer</artifactId>
    <version>7.15.0</version>
</dependency>

🔬 How could we reproduce it?

Given scenario:

Scenario: ConcurrentEventListener bug
    When action
    Then check result <---- this step is failing
    Given rest
    When of
    And the
    Then scenario

ConcurrentEventListener implementation:

public class StepLogger implements ConcurrentEventListener {

    @Override
    public void setEventPublisher(EventPublisher publisher) {
        publisher.registerHandlerFor(TestStepStarted.class, this::handleTestStepStarted);
    }

    private void handleTestStepStarted(TestStepStarted testStepStartedEvent) {
        if (testStepStartedEvent.getTestStep() instanceof PickleStepTestStep testStep) {
            Step currentStep = testStep.getStep();
            Log.step(currentStep.getText());
        }
    }
}

cucumber.properties:

cucumber.publish.quiet=true
cucumber.snippet-type=camelcase
cucumber.plugin=summary, html:target/reports/cucumber-pretty.html, json:target/reports/cucumber/cucumber.json, \
  io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm, org.utils.StepLogger

[mpkorstanje]

As part of the execution model, every step is always executed. But every step after a failing step is executed in dry-run mode and will not invoke the glue code. I.e. it is skipped.

cucumber-jvm/cucumber-core/src/main/java/io/cucumber/core/runner/TestCase.java

Lines 82 to 86 in 9fc3242

	for (PickleStepTestStep step : testSteps) {
	nextExecutionMode = step
	.run(this, bus, state, nextExecutionMode)
	.next(nextExecutionMode);
	}

The reasons for this are pretty much historic now. They have their origins in the original Ruby implementation. The test step event docs could probably be updated to include that information.

For now you may want to use the TestStepFinished event instead and filter on it's result status.