Code Coverage : RunSettings : Cobertura format is not working, even tough it works with CLI

[p00j4]

I'm trying VSTest task to use my runsettings and generate a Cobertura report, which works fine on my local CLI, however when same ran on ADO pipeline, generates .coverage file instead of *.cobertra.xml file

Steps to Reproduce

1. Have a sample C++ project solution
2. Add runsetting

CodeCoverage.runsettings

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <DataCollectionRunSettings>
        
        <DataCollectors>
            <DataCollector friendlyName="Code Coverage">
                <Configuration>
                    <Format>Cobertura</Format>
                </Configuration>
            </DataCollector>
            
            
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

3. Create a sample ADO pipeline with VSTest task
   Task

      - task: VSTest@2
        displayName: Execute Unit Tests for $MySolution
        inputs:
          testAssemblyVer2: '**\*unittest*.dll'
          searchFolder: $(outputDirectory)  
          vsTestVersion: toolsInstaller
          runSettingsFile: Src\CodeCoverage.runsettings
          resultsFolder: '$(Agent.TempDirectory)\TestResults'
          diagnosticsEnabled: True
          codeCoverageEnabled: true
          collectDumpOn: Always
          testRunTitle: ""
          rerunFailedTests: true
          rerunMaxAttempts: 2
          condition: and(
                        succeeded() 
           )

4. Run it and observe, it generates dot coverage file. Instead is supposed to generate a cobertura file

5. The same works using CLI
   Cli run
   vstest.console.exe MySolution.UnitTests.dll --EnableCodeCoverage --settings:$MyPath\Src\CodeCoverage.runsettings
   

However, on ADO, the same config, generates a dot coverage file instead

Debug info:
The runsettings ADO has automatically modified to

But update: If I modify the runsetting (as of the ADO) on my local and run with CLI, it still works fine. It's only not working with the ADO task. What am I missing?

[fhnaseer]

Can you check which version of test platform is being used by ADO? It seems like it is using older version in which cobertura format is not supported.

[p00j4]

Thanks Faisal for checking this. Yes, I also came to the same conclusion. The Visual Studio on my ADO instance is on 2019 16.0.1

• From which exact version, it has started supporting? didn't find the info in the release notes
• If it's coming from VS 2022 (which looks high likely because my local had both 19 and 22).

Query: If I can't update the VS full version on the ADO box, will just replacing the vstest.console.exe to your recommended version would do the job and safe from the dependency tree's perspective?

Edit:
Can "testPlatformVersion" to 17.0.1 (which has latest vstest.console.exe) in VSPTI before running the test be an easy safe solution instead?
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer
I'm anyways, testing this out. will keep posted how it goes.
Any lead on cli syntax that I can log to know where VSPTI installs (i'm assuming it must be creating a sandbox kind of and doesn't replace anything inside the actual VS installed location in C Program Files\Microsoft Visual Studio because that would be undesirable.
It mentions tools cache, where and what's the duration of the cache to be alive? can a next build get affected by the updated cache by this one?

[fhnaseer]

Cobertura support was added in 17.1 (Visual Studio and Microsoft.TestPlafrom). https://docs.microsoft.com/en-us/visualstudio/releases/2022/release-notes
I assume tools cache means nuget cache. You can ask this question on ADO forums to get correct answer.

[p00j4]

Ok thanks, yet I have tested it 17.0.1 and it seem to generate the cobertura. However yes, I need to confirm that bit. will do on azure forum.

A follow up query on VsTest cobertura support: just like for .covx and .coverage, the Publish Test Result Task auto merge/aggregates the results (from multiple jobs withing the same build) at the end of the build. Would it do for cobertura as well?
Because the problem is it's not supporting cobertura and have to go to explicitely call Publish Code Coverage Results and the doc doesn't mention about the auto merge.

[jakubch1]

@p00j4 sorry for late response :) Azure DevOps Publish Code Coverage results tasks are not supporting merging. You should merge all coverage reports before you push it. You can use report generator to merge it for you.

[QushengSun]

@jakubch1 Are you saying the PublishCodeCoverageResults@2 task does not support merging? I kinda doubt that