trx logger does not include Description attribute

[AbhitejJohn]

Description

Test description attributes like Description, CssIteration, WorkItem, CssProjectStructure do not show up in the trx log. Only Owner and Priority are part of the log.

Steps to reproduce

1. Create an MSTest V1 based project.
2. Add a DescriptionAttribute on top of the test method.
3. Run tests with trx turned on.

Expected behavior

The trx should have the description string associated with the test in the log.

Actual behavior

The description string does not show up in the log.

Environment

I'm trying this on Dev15 prev-5 and on Windows 10.

[AbhitejJohn]

Tied to https://github.com/Microsoft/testfx/pull/16

[codito]

Description attribute is very specific to a particular test framework. Should there be a extensibility point for test frameworks contribute data into trx logger?

@Faizan2304 what do you think?

[codito]

@AbhitejJohn is this a regression since Dev14 Update 3?

[AbhitejJohn]

@codito : DescriptionAttribute was just an example. All these do not show up: Description, CssIteration, WorkItem, CssProjectStructure. They however do get filled in properly when run from mstest.exe though.

IMHO all traits that frameworks fill in need to be part of trx. The goal is to not hard code of-course.

This has been the behaviour in TP V1 for a long time though. We only put in Priority and Owner but the drawback for this is with reporting this in VSTS or any place else.

[codito]

@AbhitejJohn makes sense. A sample trx with Priority looks like following:

    <UnitTest name="TestWithPriority" storage="g:\tmp\unittestproject1.dll" priority="0" id="f2daad78-aff7-a545-f0ba-d1d27d5b0789">
      <Execution id="40c5e9e7-9349-4a0e-a885-393691579e2a" />
      <TestMethod codeBase="G:\tmp\UnitTestProject1.dll" executorUriOfAdapter="executor://mstestadapter/v2" className="SampleUnitTestProject.UnitTest1" name="TestWithPriority" />
    </UnitTest>

Would an output like below make sense? Note the Traits section.

    <UnitTest name="TestWithPriority" storage="g:\tmp\unittestproject1.dll" priority="0" id="f2daad78-aff7-a545-f0ba-d1d27d5b0789">
      <Execution id="40c5e9e7-9349-4a0e-a885-393691579e2a" />
      <TestMethod codeBase="G:\tmp\UnitTestProject1.dll" executorUriOfAdapter="executor://mstestadapter/v2" className="SampleUnitTestProject.UnitTest1" name="TestWithPriority" />
       <Traits>
            <Trait Name="Description" Value="Sample description" />
            <Trait Name="ComplexData" Value="{ jsonArray: [ 1, 2, 3] }" />
       </Traits>
    </UnitTest>

@pvlakshm @sbaid what do you think?

[AbhitejJohn]

We would want to figure out a schema that fits in with TRI extensibility as well so that these results show up on VSTS.

[pvlakshm]

Triage: will talk to customers to understand scenario and importance.

[dotMorten]

It's already in the TRX schema:

    <xs:element name="Properties" minOccurs="0">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="Property" minOccurs="0" maxOccurs="unbounded">
              <xs:complexType>
                <xs:sequence>
                  <xs:element name="Key" />
                  <xs:element name="Value" />
                </xs:sequence>
              </xs:complexType>
            </xs:element>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
      <xs:element name="WorkItemIDs" type="WorkItemIDsType" minOccurs="0">
        <xs:annotation>
          <xs:documentation>References to workitems (bugs, issues) related to this test on the team server.</xs:documentation>
        </xs:annotation>
      </xs:element>

Essentially any use of [TestProperty] should go in here. I was able to use it in my own TRX logger, and it passes schema validation

[smadala]

@dotMorten Thanks for info. Do you like to patch vstest TRX logger with your TRX logger?

[pvlakshm]

This ought to be validated across adapters to ensure there are no surprises.

[dotMorten]

@smadala My logger is completely different from yours. I would have preferred being able to reuse the TRX logger you have (a .NET Standard version that can serialize an IEnumerable<TestResult> would be appreciated !), but found it overly complex to deal with and would need a lot of forking.

I could try and take a stab at it, but yours is so much more complex :-)

[DmitryNaumovA1qa]

Is there a fix for the issue?
If so which version of .net core should I use to be able to include description attribute?

[dotMorten]

@DmitryNaumovA1qa no 😒

[dotMorten]

Just submitted a PR that ensures CSS*, Description, WorkItemIDs and TestProperties gets written to the TRX

[OsirisTerje]

The Trx logger does not write out categories from NUnit, because the category type name is hardcoded to "MSTestDiscoverer.TestCategory", whereas the NUnit name is "NUnit.TestCategory". The idea back in 2012 was that categories should be Traits that where not hardcoded to anything in particular. Can we a) get this fixed so that any trait show up in the trx file? or b) Add in the NUnit category type name ?

See issue nunit/nunit3-vs-adapter#506

We [NUnit] accepted a PR to include the MSTest name as a temporary workaround until this was fixed, but it turns out this results in duplicate categories in the Test Explorer. nunit/nunit3-vs-adapter#559

[dotMorten]

@OsirisTerje I'm not sure about the relevance wrt writing a TRX report? The xsd schema has a generic Category field. How these categories are transported around with which trait name seems sort of irrelevant, but if you want to use a Microsoft trx defined standard, you sort of have to adhere to that, or request their loggers have a more abstract way to define the various traits and their meaning.,

[OsirisTerje]

@dotMorten This code line https://github.com/Microsoft/vstest/blob/0341da456744f78bba8ff2a25c1cddfc89c87b08/src/Microsoft.TestPlatform.Extensions.TrxLogger/Utility/Converter.cs#L51 looks hardcoded to MSTest to me. The NUnit3TestAdapter doesn't write to the trx file, that is all up to vstest, it only provides the information, and the category type name is marking it as a category. There is no generic type name for the category on that side .

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform