Feature: Add test-suite support to IReporter interface and JUnit-Reporter

[jenisys]

Description

Currently, the doctest::IReporter interface lacks methods to add test-suite information.

CURRENT STATE:

• The junit reporter just provides only <testsuite/> XML element(s) for the test-program that is executed
• JUnitXML supports nested testsuite(s), meaning a <testsuite/> XML element(s) within another <testsuite/> XML element.

BUT:

• Testsuites (as a general concept) provide a hierarchical grouping mechanism
• JUnitXML nested testsuites map well to the concept how testsuites are used in the C++ source code.

NOTES:

• IDE(s), like Jetbrains, etc. normally provide a TestRunner/TestExplorer where this hierarchical testsuite grouping is shown.

Steps to reproduce

MOTIVATING EXAMPLE:

// -- FILE: test_example_with_testsuites.cpp
#include "doctest/doctest.h"

TEST_SUITE("test_suite_1") {
    TEST_CASE("one")
    {
        MESSAGE("Part of testsuite_1: test=one");
    }

    TEST_CASE("two")
    {
        MESSAGE("Part of testsuite_1: test=two");
    }
}

TEST_SUITE("test_suite_2") {
    TEST_CASE("one")
    {
        MESSAGE("Part of testsuite_2: test=one");
    }

    TEST_CASE("two")
    {
        MESSAGE("Part of testsuite_2: test=two");
    }
}

When you build the test program and run it with ... --reporters=junit, you get the following JUnitXML report:

<!-- FILE: junit_test_report.xml -->
<testsuites>
  <testsuite name="test_example" errors="0" failures="0" tests="4" ... doctest_version="2.4.11">
    <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
    <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/>
    <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
    <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/> 
  <testsuite>
<testsuites>

PROBLEM:

• Currently, it is completely unclear why tests "one" and "two" occur twice
• Testsuite context is lost in the report

DESIRED OUTPUT:

<!-- FILE: junit_test_report.xml -->
<testsuites>
  <testsuite name="test_example" errors="0" failures="0" tests="4" ... doctest_version="2.4.11">
    <testsuite name="test_suite_1" errors="0" failures="0" tests="2" ... >
      <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
      <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/>
    </testsuite>
    <testsuite name="test_suite_2" errors="0" failures="0" tests="2" ... >
      <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
      <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/> 
    </testsuite>
  </testsuite>
</testsuites>

EXAMPLE 2: Test suite name provides context info for test-cases

// -- FILE: test_example2_with_testsuites.cpp
#include "doctest/doctest.h"

TEST_SUITE("Function A") {
    TEST_CASE("It should succeed if ...") { ... }
    TEST_CASE("It should fail if ...") { ... }
    ...
}

TEST_SUITE("Function B") {
    TEST_CASE("It should succeed if ...") { ... }
    TEST_CASE("It should fail if ...") { ... }
    ...
}

CONSIDERED ALTERNATIVES: TEST_CASE/SUB_CASE combination

• This disambiguates the example above
• This combines the TEST_CASE name with the SUB_CASE name in one <testcase name="{TEST_CASE.name}/{SUBCASE.name}" /> XML element
• The testcase.name can become rather large
• BUT: The grouping gets slightly lost or must be rediscovered from the name info.

Extra information

DESIRED CHANGE:

• Add methods to doctest::Reporter to add testsuite entry/exit.
• Default implementation of these methods could be empty (to avoid to break existing reporters).
• Call theses methods when a testsuite entry/exit is encountered by the doctest framework

NICE TO HAVE:

• An extended junit reporter implementation that uses this additional testsuite context information to create the test report with the desired information

VERSION INFO:

• doctest version: v2.4.11

[cdeln]

I agree with everything you have described, and believe the current behavior is not intentional.
Since the current output is completely broken when using test suites I think it's fine to break downstream who depend on it (or this should have been reported earlier).
I think most consumers will enjoy a fix for it.
Just want to check, since I am no JUnit expert, are there several JUnit formats available that needs to be considered?

[jenisys]

In my opinion, you need to decide first which XML schema you want to support (even if you do not use the complete functionality). That allows that user can build readable report-generator tools around it (for HTML, …).

CANDIDATES:

• JenkinsCI XUnit format: https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd
• JUnit5 refers to a slightly different XML format by now, defined by open-test-reporting
• …

NOTES:

• pytest, a Python based test framework, changed to its XML schema a while ago to the (newer) JenkinsCI XUnit format.

IMPORTANT ASPECTS (in my opinion):

• properties to be able to describe the context of the test run (which versions are used, tests ran on which host/target, which BUILD variant was used, …)
• Captured output (stdout, stderr, log-output) should be supported (in principle)
• Composibility: testsuiteXML elements can contain testcase XML elements and also other testsuite XML child elements.

SEE ALSO:

• https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd
• https://junit.org/junit5/docs/current/user-guide/#junit-platform-reporting-open-test-reporting
• https://github.com/ota4j-team/open-test-reporting/

[cdeln]

JUnit5 also supports "legacy XML": https://junit.org/junit5/docs/current/user-guide/#junit-platform-reporting-legacy-xml

Open Test Reporting support two different schemas as well:

• Event based: https://github.com/ota4j-team/open-test-reporting/?tab=readme-ov-file#event-based-format
• Hierarchical: https://github.com/ota4j-team/open-test-reporting/?tab=readme-ov-file#hierarchical-format

I agree, need to pick a schema. Question is, which one do you use? Doctest can't support any available schema out there, and it's unnecessary to implement reporters for schemas that no one uses.

I suggest you implement the desired functionality using the reporter API, and share your implementation here for others to use. If you one of the JUnit10 schema, then I suggest we call the corresponding reporter JUnit10Reporter in C++ and the CLI option junit10. Similarly, if you use the Open Test Reporting schema, the C++ classes could be called something like OpenTestEventReporter and OpenTestHierarchicalReporter respectively, and the corresponding CLI options ote and oth. Just suggestions, open for a discussion.

Whatever reporter schema is implemented we can make the junit reporter CLI option default to one of them.

If you can't implement this using the current reporter API as hinted in your initial description, then please elaborate on your suggested improvements to the IReporter interface that are necessary to implement the actual reporter implementation (code > text). These changes would be in two PRs to make it reviewable.

[jenisys]

I use the JenkinsCI JUnitXML format mostly.

NOTES:

• The Events format of open-test-reporting is primarily intended if you need to stream the events during test execution to another service that listens to these stream of events. This can have advantages when you have rather long test run durations, like: multiple hours or days.