Mismatched Step Titles and Missing Tags in Reports After Retrying Scenarios

[Owlwasrowk]

Summary

Two issues have been observed after retrying failed scenarios in a test suite with a Scenario Outline that includes multiple Examples tables:

1. The tags from the Examples tables are lost in the final report.
2. The titles in the report are mismatched. See the screenshot in the attachment section for an illustrative example.

Steps to Reproduce

1. Clone the Repository with the example which will produce the bug https://github.com/Owlwasrowk/karate-template
2. Run the test runner using Maven with the command mvn clean test. There is used a random flakyness to illustrate the behaviour.
3. Observe the generated report for missing tags and mismatched titles.

Expected Behavior

After retrying failed scenarios and updating the report:

• The tags from the Examples tables should remain associated with the corresponding scenarios.
• The titles in the report should correctly match the scenarios that were retried.

Actual Behavior

After retrying failed scenarios and updating the results:

• The tags from the Examples tables are missing in the final report.
• There is a mismatch in the report titles, where a failed scenario from one Examples table incorrectly shows as another scenario from a different table.

Environment

• Karate Version: 1.4.1
• Java Version: openjdk 11.0.14.1 2022-02-08 LTS

Additional Information

• The mismatch in titles and missing tags only occur after the retry logic is executed.
• The initial report generated before the retry does not have these issues.

Relevance of the Bug

The tags in our Karate test scenarios are not merely for organizational purposes; they serve a critical role in integrating our test automation framework with our test management system. Each tag corresponds to a unique identifier in Xray, which allows us to map automated test results directly to the relevant test cases in our management system. The loss of tags from the Examples tables after retrying scenarios disrupts this integration.

Solution Hints

The problem with the titles appears to be related to how the updateResults method in the Suite class compares existing scenarios to determine if they are equal. The comparison logic, which relies only on the section index and the example index, can erroneously consider different scenarios as equal under certain conditions.

Mismatched Titles

The updateResults method (here) uses the equals method of the Scenario class to match scenarios. The isEqualTo method (here) checks if the section index and the example index are the same. This can lead to a situation where different scenarios from separate Example tables are incorrectly identified as the same because they share the same index positions within their respective tables.

A potential solution would be to refine the comparison logic to include a more distinctive identifier that takes into account the unique context of each scenario, ensuring that scenarios from different Example tables are not conflated.

Missing Tags

The issue with the missing tags after retrying scenarios is still under investigation.

Attachments

• karate-reports.zip: The karate-reports folder containing the generated report after the retry, showcasing the reported bug.
• 
  Screenshot from the Report where we see, that the Example Table Tags are missing, Example 3 is missing and Example 6 is duplicated.

[ptrthomas]

@Owlwasrowk thanks for the details. Karate doesn't yet support retries and this issue is a great example of what all edge cases come in the way. the best option here is for someone to go through the details (I think you have done some good analysis already) and contribute a PR

[Owlwasrowk]

@ptrthomas Thank you for your prompt response. I understand that Karate does not currently support retries out-of-the-box, and the complexities that come with implementing such a feature.

I appreciate your suggestion to contribute a PR to address this issue. The missing tags may be related to the Suite object not retaining information about the scenario's originating example table during retries.

If you have any starting points or insights that could guide the development of a solution, I would be grateful for your advice.

[ptrthomas]

@Owlwasrowk there is a developer guide that may help: https://github.com/karatelabs/karate/wiki/Developer-Guide

but anyone with Java experience should be able to dive in. I think you are spot on with the equals implementation of scenario. we are missing the concept of what I propose to call "examplesParentIndex", ideally I would rename "examplesIndex" to "examplesRowIndex" and use "examplesIndex" for what's missing

here is where example index is inited: https://github.com/karatelabs/karate/blob/v1.4.1/karate-core/src/main/java/com/intuit/karate/core/ScenarioOutline.java#L109

and here is where examples are created when parsing a feature: https://github.com/karatelabs/karate/blob/v1.4.1/karate-core/src/main/java/com/intuit/karate/core/FeatureParser.java#L282

and now if you add one more component to the isEquals check, this should work