Need to clarify semantic of Optional GHA Runners

[tarilabs]

Some of the Optional GHA TCK Runners, ie the Vendor specific ones, are completing Successfully (✅ ) even when the test case is clearly wrong.

This could be confusing for:

• Users looking at the GitHub and seeing GHA reports
• Users running the project locally (and not checking the generated tck_results.csv content in the results folder)

as they would see the green ✅ and potentially assume all the tests are passing.

This can be demonstrated with:

• DO NOT MERGE demo wrong/confusing JUnit reports of Optional GHA Runners #469

While the PR passes vendor-neutral Validation check (both .dmn and .xml files are semantically valid), the test should Fail as it's clearly a wrong assertion; none of the optional runners reports in tck_results.csv a success, yet some of the optional runners are green ✅ when running on GitHub or locally with Maven:

Details

I realised this, as I was working on a separate PR here: #468

I believe we have a common agreement that the tck_results.csv can be used to report IGNORED, not supported, and missing cases; this is very helpful when a Vendor wants to submit refreshed results and being explicit that a specific test case might relate to a feature not-yet-supported, that might eventually be supported, etc. beyond simply reporting the error.

On other hand, the semantic of the JUnit running with each test case, I believe should be the Java one: either a test case is preventively marked as ignored, or it should Fail if the test is Failing for whatever reason.

Any solution to this issue, would need careful considerations, also in order to strike a balance between: Vendors who can and are willing to share end-to-end open source solutions (both engine and runners), and Vendors who cannot share in full end-to-end open source solutions.

In my opinion, the current semantic of some of the runners (demonstrated by #469) that if a test is Failing, it is automatically marked as Ignored instead, and NO JUnit failures, imho:

• is potentially misleading for Users
• defies the purpose of having the runner as an Optional GHA Runner (will always be green!)

We have been using Optional GHA Runner as an helpful tool to catch potential problems, or difference of interpretation in the spec; but if a GHA runner is green because automatically assuming any Failure as Ignored, defies this rationale. At the risk of saying something tautological, just to be clear: Optional GHA Runners, as the name implies, are opt-in, and no Vendor is required to enlist the runner provided to the TCK in the list of GHA Runners.

I am happy to hear feedback both on my analysis if I missed to consider anything, and to potential solutions.

[saig0]

@tarilabs that is an interesting point. I get your idea but I see it a bit differently.

For example, the Camunda runners report test failures (by default) as ignored. The Camunda DMN engines don't support all features of DMN (and will probably not in the future).

If we change the behavior of the TCK runners to report the failures then the TCK runners will always fail and report ❌
This behavior could be confusing in combination with the GitHub actions. If I open a PR then the GitHub action of the runner will fail. As the author of the PR, I may think that the PR is not valid. But actually, it is a problem with the TCK runner / the DMN engine.

In this case, we could remove the GitHub actions for the runners. But I see value in them to discover failures in the TCK runners or the dependent TCK modules.

[tarilabs]

As discussed in the TCK meeting 2021-10-14 I will try to see if we can leverage the TCK+JUnit infrastructure so to provide ahead-of-time a list of ignored for each runner/vendor.

This could work as a compromise in-between the perspective of this #470 (comment) and this #470 (comment)

Will keep posted.

[baldimir]

Discussed changes on the webpage on the TCK meeting. See meeting notes https://docs.google.com/document/d/1GnXZcwSczIbyNtMDM8ZnITVNgpIVi9nGbyt-0WzKxec/edit