DataRow execution is published as 'unspecified' with VSTest@3 task for Test plan

[gasparnagy]

Describe the bug

We are using the VSTest@3 task in Azure DevOps for executing the MsTest test associated to Test Cases which having data rows in the test method ([DataRow]).

Based on the issue suggestion at #992, we have disabled source discovery using [assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]

When we run the pipeline, it logs all data rows as executed, but in the TestRun it is reported as a single test with 'unspecified' result. (In the TRX file that is attached to the TestRun all data row execution is recorded.)

When I downgrade MSTest.TestAdapter and MSTest.TestFramework to v2.2.3 (and remove [TestDataSourceDiscovery]) it works fine.

Steps To Reproduce

1. Create a new MsTest project.
2. Add a test method like.

        [DataTestMethod]
        [DataRow(42)]
        [DataRow(43)]
        public void TestMethod1(int value)
        {
            Console.WriteLine($"Value: {value}");
        }

3. Add to the project: [assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]
4. Create a TestPlan, a Test Suite and a Test Case in ADO
5. Associate the TestMethod1 to the Test Case
6. Create a pipeline that builds the project and runs the tests based on the TestPlan/TestSuite:

- task: VSTest@3
  displayName: 'VsTest - testSuite'
  inputs:
    testSelector: testPlan
    testPlan: 552255
    testSuite: 552257
    testConfiguration: 15
    testRunTitle: 'CI1-Plan'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

7. Run pipeline and check "Tests" tab

Expected behavior

Displays the data row test as sub-tests, like with MsTest 2.2.3:

Actual behavior

It is shown as 'Unspecified':

Also in the related Test Run:

Additional context

I have also tried with VSTest@2, made no difference.

[nohwnd]

Hello, I've also found this discussion #1026 (comment) where our AzDO colleagues point out the need to enable a feature flag in case you are running on on-prem instance. Is this the case for you?

[nohwnd]

This should be an issue for the AzDO team, if you are still experiencing the issue, please file it here: https://developercommunity.visualstudio.com/AzureDevOps

Sorry for no response for a long time and then just redirecting you elsewhere 🙇

The change on MSTest side is the trigger for the new behavior, but it was communicated to AzDO and fix for it was implemented, so I am not sure why it does not work.

[gasparnagy]

@nohwnd Thx for the update. I (my clients) need this for the cloud version, so the feature flag is not an option for me. I will replicate the issue to the dev community.

[gasparnagy]

Related dev community issue: https://developercommunity.visualstudio.com/t/MsTest-DataRow-execution-is-published-as/10856785