Support parameter resolution in in Scenario Outline

[Desislav-Petrov]

🤔 What's the problem you're trying to solve?

Taking the example below from the codebase:

When the feature file is executed, the output in the below looks like this:

The naming convention following the pattern of #Example1.1, 1.2, etc leads to a lack context and the issue is even more problematic in bigger codebases, with lots of tests that utilise examples heavily.

✨ What's your proposed solution?

Ideally, I'd like to be able to modify the scenario outline to look like this:
Scenario Outline: Many additions - first digit <a> second digit <b> expected result <c> so that the test executions in intellij come out with a meaningful description.

⛏ Have you considered any alternatives or workarounds?

Based on this issue for cucumber js, it's been suggested it should already possible but I couldn't get it working with the examples in this codebase as illustrated by the screenshots.

📚 Any additional context?

No response

[mpkorstanje]

Would this work?

Feature: Examples Tables

  Scenario Outline: Eating <eat> cucumbers
    Given there are <start> cucumbers
    When I eat <eat> cucumbers
    Then I should have <left> cucumbers

    Examples: These are passing
      | start | eat | left |
      |    12 |   5 |    7 |

Examples Tables
└── Eating <eat> cucumbers
    └──These are passing
        └── #1.1: Eating 5 cucumbers

And when not parameterized

Examples Tables
└── Eating cucumbers
    └──These are passing
        └── #1.1

Btw, if you right click on a node in your IDEA, there should be a "Jump to Source" option.

[Desislav-Petrov]

Hi @mpkorstanje - thx for the suggestion.

I've tried the example that you have provided and it seems there is a different behaviour when executed from within the
calculator-java-junit4 vs calculator-java-junt5.

When running as a junit4 example, it works as you have predicted (IntelliJ screen):

However, when running as a junit5 example, the behaviour seems to be different:

In both cases, the scenario file looks like the below:

Scenario Outline: Eating <eat> cucumbers
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then I should have <left> cucumbers

Examples: These are passing
  | start | eat | left |
  |    12 |   5 |    7 |
  |    13 |   5 |    7 | 

Am I doing something obviously wrong? Btw - if you believe it's a code issue I'm happy to take a look.

[mpkorstanje]

No. I think you're doing everything correctly. The names are inconsistent because we're integrating with different systems. Each with its own limitations, and Cucumber needs some internal refactoring too so that doesn't make things easier.

For JUnit 5, you can work around it by either one of these properties:

cucumber.junit-platform.naming-strategy.short.example-name=pickle
cucumber.junit-platform.naming-strategy.long.example-name=pickle

For IDEA I'll have to take a closer look.

[Desislav-Petrov]

Thx @mpkorstanje - let me do some digging and I'll report back, I have a suspicion where the problem might be.

[mpkorstanje]

Cheers! I appreciate that you'll want to help out. But I know where the problem is. Have a look at #2890 if you're interested. I'm just not happy with the solution. And doing it properly will take a bit more work.

[Desislav-Petrov]

Ah I see.. One more idea to run past you: in the package io.cucumber.junit.platform.engine.DefaultNamingStrategyProvider class, if you propagate a Pickle to the private static String longStrategy(Node node, Pickle pickle, String currentNodeName), you will have access to the parsed string that you can use as a part of forming the long name. It does require a bit of magic around the ordering due to the parent<>child relationship and I'm not convinced it will work in all cases.

[mpkorstanje]

That would be a quick way to do it. And I can't think of a reason it shouldn't work. Cucumbers AST is simple enough that for an example in a scenario outline, the scenario outline is always the 3rd node up.

But this naming scheme shows up in a few places. With the JUnit4 and TestNG runners being just about the only places where we can't use this pattern. So I'm looking at extracting this to some common code. As it so happens to be, I was already working on that from a few different angles. So I'll just have to pull it all together.

[mpkorstanje]

JUnit 5 is resolved by #3004

[mpkorstanje]

And the Teamcity plugin can be addressed as part of #3001