Multiple versions of the same extension found

[nohwnd]

Description

Warning is logged to the screen but not further information that would help diagnose the issue is provided. Nothing is written to log.

Steps to reproduce

Not sure, cannot diagnose it.

Expected behavior

I get versions and paths from where the extensions were loaded. Or I get that information in diag log so I can fix my test project.

Actual behavior

I get warning that is unhelpful.

Diagnostic logs

Nothing about this is written there.

Environment

AB#2036316

[jiangyz7cc]

I have the same problem:

========== Starting test run ==========
Multiple versions of same extension found. Selecting the highest version.
  Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter : 14.0.7923.2
An exception occurred while invoking executor 'executor://mstestadapter/v2': Field not found: 'Microsoft.VisualStudio.TestTools.UnitTesting.TestContext.TestRunDirectoryLabel'.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestDeployment.GetDeploymentInformation(String source)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTestsInSource(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, String source, Boolean isDeploymentDone)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, Boolean isDeploymentDone)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken runCancellationToken)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionDecorators.SerialTestRunDecorator.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithTests.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUri, RunContext runContext, IFrameworkHandle frameworkHand

Environment:
VS 2022
Project: .NET Framework 4.6.1
MSTest.Framework : 2.2.10

It all runs well on .NET 5.0+
and the 1st time I created the .NET Framework Unit Test Project, and later some problems occurs including the above one, no ideals what's going on inside.

[jiangyz7cc]

I just try to create a demo solution of test projects.

After some tries I found it's successful when there's only 1 project, but when I add another project ( with different MSTest package version that higher than old one's ) , then the error Multiple versions of same extension found. Selecting the highest version. occurs, it's a strange behavior that is reverse to default package dependencies choose behavior: choose the lowest one, so probably the higher version of test dependency is the problem source.

[LeeWhite187]

Is this going to get fixed?
I see the same issue, when multiple projects (of different framework target versions) exist in the solution, that require different versions of MSTest.TestAdapter.

[nohwnd]

Time to reinvestigate! 🔎

[testplatform-bot]

✅ Successfully linked to Azure Boards work item(s):

• User Story 2036316

[nohwnd]

@Evangelink I am putting this on my board, but I think you told me that you already investigated this in the past. Let's get in touch, maybe it could also help mstest v2 v3 run side by side.

[nohwnd]

I can repro seeing the warning, but not having tests fail.

Vstest.console inspects all extensions from all projects and more places, and there can be conflict, and will be conflict when you use multiple versions of a test framework. This behavior (loading extensions from projects) is useful to add extension to vstest.console that are shipped in TestAdapter.dll, but in hindsight should not have been done.

But the case that @jiangyz7cc saw is curious, because I am not able to replicate the warning actually causing issues, maybe I am doing something wrong, and someone has a repro.

[LeeWhite187]

I get similar output to the initial issuer.
Here's the Output that I get when attempting to run a test from a Net Framework MSTest Project, in case it helps:

Building Test Projects
========== Starting test run ==========
Multiple versions of same extension found. Selecting the highest version.
  Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter : 14.0.8501.1
An exception occurred while invoking executor 'executor://mstestadapter/v2': Field not found: 'Microsoft.VisualStudio.TestTools.UnitTesting.TestContext.TestRunDirectoryLabel'.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.TestDeployment.GetDeploymentInformation(String source)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTestsInSource(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, String source, Boolean isDeploymentDone)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.ExecuteTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, Boolean isDeploymentDone)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken runCancellationToken)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionDecorators.SerialTestRunDecorator.RunTests(IEnumerable`1 tests, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithTests.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUri, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass46_0.<RunTestInternalWithExecutors>b__0()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.<Run>b__0()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInStaThread(Action action, Boolean waitForCompletion)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 110 ms ==========

[LeeWhite187]

And, I'm unable to access the Azure Board reference (https://devdiv.visualstudio.com/0bdbc590-a062-4c3f-b0f6-9383f67865ee/_workitems/edit/2036316).

Is that where this issue is being tracked?

[nohwnd]

The azdo link is just so it appears on my internal work board.

That output looks promising, so this is .net framework project, in a solution that has mix of versions of mstest? I see it fails in deployment, do you use [DeploymentItem] on any of your tests?

[nohwnd]

From what I see this seems to repro only on .NET Framework dlls, when you run them under 1 instance of vstest.console, that is either by: running in VS, running under dotnet test + dll (but not csproj or dll), running under vstest.console (it only allows passing dll).

This is caused by using a shared host, which is a "historical" performance optimization where the dlls will run in the same testhost.exe. This mode is enabled only for .NET Framework dlls, that don't enable process level parallelization, or that don't disable AppDomains.

At the moment you can force the run to be in non-shared mode by setting:

 -- RunConfiguration.DisableAppDomain=true
 or
 -- RunConfiguration.MaxCpuCount=0

Or using the equivalent runsettings, that specify at least one of these options. The maxcpucount can be anything that forces process-level parallelization, so any value that is 0, or more, but is NOT 1.

<RunSettings>
  <RunConfiguration>
    <DisableAppDomain>true</DisableAppDomain>
    <MaxCpuCount>0</MaxCpuCount> 
  </RunConfiguration>
</RunSettings>

I am thinking of putting this Shared mode under a runsetting and feature flag, and disabling it by default. Because it is undocumented and confusing.

[jiangyz7cc]

Thanks for reply.

I almost forgot the problem.

But still remind this case seems occurs when I use a new NET version of MSTest project refers an old NET framework project or mixed with NET Standard one.

It's like the shared host case, because the VS can not distinguish which version of Test frame (one) to use, then I update all projects to the new NET, I don't know how to re-make this happen now.

[LeeWhite187]

@nohwnd saw your reply, with switch settings to force non-shared mode.
It's not clear to me, where to set these, so I can include my older NET Framework targeted test project with the newer ones in the same test "batch".
When you get a chance, would you mind explaining where those settings would go, please?
I have a two-solution workaround in place, so my 4.5.2 tests can be discovered, so no rush.

Interestingly enough, the problem I'm seeing (having undiscovered tests in NET Framework targeted projects) seems to NOT affect NET Framework 4.8, while it does affect NET Framework 4.5.2.
I think this fits what you were saying, that it's a multiple test framework version problem, and NET 4.5.2 requires an older version than 4.8.
Thankfully, 4.8 is compatible with the same test framework as NET5, 6, and 7.
So, I only have to split off any NET 4.5.2 MSTest projects to their own solutions.

Here's my working list of relevant library dependencies for MSTest projects, if it helps:

[nohwnd]

It's not clear to me, where to set these, so I can include my older NET Framework targeted test project with the newer ones in the same test "batch".

The options would go to runsettings. https://learn.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2022

Interestingly enough, the problem I'm seeing (having undiscovered tests in NET Framework targeted projects) seems to NOT affect NET Framework 4.8, while it does affect NET Framework 4.5.2.

This is interesting, I saw multiple ways to have tests broken, so you are maybe facing a different issue.