TestExplorer cannot understand TestCase-naming-style

[Kohlroulade]

Description

I have an implementation for ITestDiscoverer as follows:

class MyClass : ITestDiscoverer
{
    public void DiscoverTests(IEnumerable<string> sources, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)
    {
        discoverySink.SendTestCase(new TestCase(
          "My.Namespace.MyType.MyMember.MyMember(1)", 
          new Uri("executor://myexecutor/v1"), 
          "path\\to\\my.test.assembly.dll"));
    }
}

Furtmerore my test-project has this class:

namespace My.Namespace
{
    [TestFixture]
    public class MyType
    {
        [TestCase(1)
        public void MyMember(int n) { ... }
    }
}

However VS TestExplorer cannot interpret the testcase's name and allways just assumes My.Namespace.MyType to be the namespace, instead of My.Namespace.

I omited the very first level of hierarchy, which applies to the assembly.
So there's no way to indicate where the namespace for a particular testcase ends and where its member starts.
Furthermore the test appears twice in TestExplorer. I assume this is because some default (?) test-discoverer also running.

Steps to reproduce

Create a new adapter and add it to your test-project as a reference. Then use TestExplorer to see the testcase appearing twice: once with namespace being My.Namespace.MyType and once with My.Namespace.

Using VS2022.

Expected behavior

TestCase should only appear once, with second hierarchy (=namespace) My.Namespace and third hierarchy (=type) MyType.

Actual behavior

TestCase appears twice, once with namespace being My.Namespace.MyType and once with My.Namespace

Environment

running VS2022 on Windows and NUnit 4.1

[nohwnd]

TE uses regex to parse (namespace)?.class.method(params), you name seems to repeat the method names twice, so it becomes the type name.

My.Namespace.MyType.MyMember.MyMember(1)

[Kohlroulade]

Well, that gets it, but it doesn't create a nesting for the different testcases within a specific test. So all TestCases with different parameters are flattened. So the max. depth of the hierarchy is four, while with the standard-adapter the depth seems to be five.

My test-explorer shows this:

What I want instead is this:

So the test still appears twice, while one of them cannot be executed, even if I right-click and say "Run" (indicated by the blue bulps in the screenshot). So how can I disable the "default"-adapter for my project? In vstest..console I could just provide the /TestAdapterPath-argument. However how would I do that in TestExplorer? I also provided a .runsettings-file with the /TestAdapterPaths-setting, but the "default" one is still run (together with mine).

And how would I go for getting five instead of four levels of nesting?

My test-code-file is this:

namespace My.Namespace
{
    [TestFixture]
    public class MyType
    {
        [TestCase(1)]
        [TestCase(2)]
        public void MyMember(int n) {  }
    }
}

[nohwnd]

I would go and enable trace logging in VS.

You can then inspect the messages that MSTest / NUnit in similar setup send, and compare your data and the other data.

That is probably the easiest way to troubleshoot this.

[nohwnd]

You will see where the logs are collected in the Test output window. It is in Temp.

[nohwnd]

@Kohlroulade Were you able to solve this by copying the data we post from mstest / xunit / nunit?

[Kohlroulade]

@nohwnd well, tbh, the log-files were incredibly huge and pretty unstructured. I didn't even get which places specifically to look for.

[nohwnd]

You can simply look into the *.host.* log, and search for the data sent as json, e.g. sending data from testhost (formatted):

TestRequestHandler.SendData: sending data from testhost: {
    "Version": 7,
    "MessageType": "TestExecution.Completed",
    "Payload": {
        "TestRunCompleteArgs": {
            "TestRunStatistics": {
                "ExecutedTests": 1,
                "Stats": {
                    "Passed": 1
                }
            },
            "IsCanceled": false,
            "IsAborted": false,
            "Error": null,
            "AttachmentSets": [],
            "InvokedDataCollectors": [],
            "ElapsedTimeInRunningTests": "00:00:00.1127182",
            "Metrics": {},
            "DiscoveredExtensions": {
                "TestDiscoverers": [
                    "Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestDiscoverer, Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
                ],
                "TestExecutors": [
                    "executor://MSTestAdapter/v2"
                ],
                "TestExecutors2": [],
                "TestSettingsProviders": []
            }
        },
        "LastRunTests": {
            "NewTestResults": [
                {
                    "TestCase": {
                        "Id": "65747f33-daed-a11c-25ff-09a600cdc21f",
                        "FullyQualifiedName": "mstest162.UnitTest1.TestMethod1",
                        "DisplayName": "TestMethod1",
                        "ExecutorUri": "executor://MSTestAdapter/v2",
                        "Source": "S:\\t\\mstest162\\bin\\Debug\\net9.0\\mstest162.dll",
                        "CodeFilePath": null,
                        "LineNumber": -1,
                        "Properties": [
                            {
                                "Key": {
                                    "Id": "TestCase.ManagedType",
                                    "Label": "ManagedType",
                                    "Category": "",
                                    "Description": "",
                                    "Attributes": 1,
                                    "ValueType": "System.String"
                                },
                                "Value": "mstest162.UnitTest1"
                            },
                            {
                                "Key": {
                                    "Id": "TestCase.Hierarchy",
                                    "Label": "Hierarchy",
                                    "Category": "",
                                    "Description": "",
                                    "Attributes": 2,
                                    "ValueType": "System.String[]"
                                },
                                "Value": [
                                    null,
                                    "mstest162",
                                    "UnitTest1",
                                    "TestMethod1"
                                ]
                            },
                            {
                                "Key": {
                                    "Id": "TestCase.ManagedMethod",
                                    "Label": "ManagedMethod",
                                    "Category": "",
                                    "Description": "",
                                    "Attributes": 1,
                                    "ValueType": "System.String"
                                },
                                "Value": "TestMethod1"
                            },
                            {
                                "Key": {
                                    "Id": "MSTestDiscoverer.TestClassName",
                                    "Label": "ClassName",
                                    "Category": "",
                                    "Description": "",
                                    "Attributes": 1,
                                    "ValueType": "System.String"
                                },
                                "Value": "mstest162.UnitTest1"
                            },
                            {
                                "Key": {
                                    "Id": "TestObject.Traits",
                                    "Label": "Traits",
                                    "Category": "",
                                    "Description": "",
                                    "Attributes": 5,
                                    "ValueType": "System.Collections.Generic.KeyValuePair`2[[System.String],[System.String]][]"
                                },
                                "Value": []
                            },
                            {
                                "Key": {
                                    "Id": "MSTest.TestIdGenerationStrategy",
                                    "Label": "TestIdGenerationStrategy",
                                    "Category": "",
                                    "Description": "",
                                    "Attributes": 1,
                                    "ValueType": "System.Int32"
                                },
                                "Value": 2
                            }
                        ]
                    },
                    "Attachments": [],
                    "Outcome": 1,
                    "ErrorMessage": null,
                    "ErrorStackTrace": null,
                    "DisplayName": "TestMethod1",
                    "Messages": [],
                    "ComputerName": "RDESKTOP",
                    "Duration": "00:00:00.0034517",
                    "StartTime": "2024-07-09T15:23:31.2494352+02:00",
                    "EndTime": "2024-07-09T15:23:31.2679196+02:00",
                    "Properties": [
                        {
                            "Key": {
                                "Id": "InnerResultsCount",
                                "Label": "InnerResultsCount",
                                "Category": "",
                                "Description": "",
                                "Attributes": 1,
                                "ValueType": "System.Int32"
                            },
                            "Value": 0
                        },
                        {
                            "Key": {
                                "Id": "ExecutionId",
                                "Label": "ExecutionId",
                                "Category": "",
                                "Description": "",
                                "Attributes": 1,
                                "ValueType": "System.Guid"
                            },
                            "Value": "00000000-0000-0000-0000-000000000000"
                        },
                        {
                            "Key": {
                                "Id": "ParentExecId",
                                "Label": "ParentExecId",
                                "Category": "",
                                "Description": "",
                                "Attributes": 1,
                                "ValueType": "System.Guid"
                            },
                            "Value": "00000000-0000-0000-0000-000000000000"
                        }
                    ]
                }
            ],
            "TestRunStatistics": {
                "ExecutedTests": 1,
                "Stats": {
                    "Passed": 1
                }
            },
            "ActiveTests": []
        },
        "RunAttachments": [],
        "ExecutorUris": [
            "executor://mstestadapter/v2"
        ]
    }
}

[Kohlroulade]

hmmm, that's strange:

"ExecutorUris": [
  "executor://nunit3testexecutor/",
  "executor://myexecutor/v1"
],

Although I only referenced my own adapter. The nunit3 adapter was referenced in another project in my solution in order to compare both settings.

[nohwnd]

Yes this is unfortunate behavior of VS / vstest.console, all adapters are scraped and unified and sent to all projects, probably because someone in the past did not want to install adapter and framework into every project, and also because tests were running directly in the vstest.console process in the beginning.

[Kohlroulade]

That's indeed pretty unfortunate and unexpected. In particular, it's really annoying to try to run a test without the correct adapter. My adapter starts another process, so there's little use in having a test run without that process. Isn't there a way to disable all other adapters so only mine is used? Otherwise all miy tests appear twice, which is really annoying in TestExplorer as mentioned earlier.

[nohwnd]

Otherwise all miy tests appear twice, which is really annoying in TestExplorer as mentioned earlier.

Are you using types from the other adapter (nunit)? Because otherwise I don't see a reason why the tests would appear twice. Maybe you are just returning different ID for the test each time (random guid), rather than a guid you determine from the test data and that remains the same across runs?

[Kohlroulade]

hmmm, I'm just referencing the NUnit-package, not the appropriate adapter in my own test-project. So this is my project-file:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
  <PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
    <ProjectGuid>{76632ADF-ADAC-430A-9872-55A51D8CCECF}</ProjectGuid>
    <OutputType>Library</OutputType>
    <AppDesignerFolder>Properties</AppDesignerFolder>
    <AssemblyName>MyTests</AssemblyName>
    <TargetFrameworkVersion>v4.7.2</TargetFrameworkVersion>
    <FileAlignment>512</FileAlignment>
    <Deterministic>true</Deterministic>
    <NuGetPackageImportStamp>
    </NuGetPackageImportStamp>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    <DebugSymbols>true</DebugSymbols>
    <DebugType>full</DebugType>
    <Optimize>false</Optimize>
    <OutputPath>bin\Debug\</OutputPath>
    <DefineConstants>DEBUG;TRACE</DefineConstants>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
    <PlatformTarget>x86</PlatformTarget>
  </PropertyGroup>
  <ItemGroup>
    <Reference Include="System" />
    <Reference Include="System.Core" />
    <Reference Include="System.Xml.Linq" />
    <Reference Include="System.Data.DataSetExtensions" />
    <Reference Include="Microsoft.CSharp" />
    <Reference Include="System.Data" />
    <Reference Include="System.Net.Http" />
    <Reference Include="System.Xml" />
  </ItemGroup>
  <ItemGroup>
    <Compile Include="Properties\AssemblyInfo.cs" />
    <Compile Include="MyType.cs" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\My.TestAdapter.csproj">
      <Project>{c6f51f56-89a5-49e4-9030-e6073abcb992}</Project>
      <Name>My.TestAdapter</Name>
    </ProjectReference>
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="NUnit">
      <Version>4.1.0</Version>
    </PackageReference>
  </ItemGroup>
  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
</Project>

[nohwnd]

Can you confirm that the GUID returned for the same test remains the same in Discovery and Run phases in your test adapter? How are you generating the test id?

[Kohlroulade]

well, I don't set any GUIDs in my code.

[nohwnd]

Then your discovered test and executed test will not map correctly together.

[Kohlroulade]

Eeeeehm, if I wanted that kind of mapping, I'd need to persist these GUIDs between discovery and execution in some way, right? That seems odd, IMHO, in particular as there's nothing in the docs that indicates that mapping.

Anyway, I set the id to a new Guid now and get the following message in tests-output:

Failed to add result for test 'MyType.MyMember' with ID 'e123c5da-9f79-4db7-b798-5142c9beaded'.

although I can confirm the ids to be identical between discovery and execution.

EDIT: Also tried it again without setting any ID myself. Seems like TestPlatform automatically assigns an id and guarantees them to be equal between dicsovery and execction, so the above seems to be irrelevant anyway.

EDIT2: The issue seems to appear only when using parameterized tests. For normal [Test]-decorated function the test is only dispplayed once in test-explorer.

[nohwnd]

Is your source code shared anywhere?

[Kohlroulade]

Not publicily available. I'm gonna check if I can share some demo-code. That however assumes, I can decouple it from my process...