Cancelling a test session may lead to a block

[dupdob]

Description

Running a test session with multiple test sources with parallelism disabled will lead to a blocked/hanged VsTest if the session is cancelled once the first sub session is completed.

Steps to reproduce

Note that is has been reproduced while working on Stryker.Net; note sure how to replicate via command line.
Start a VsTest session (via IVsTestConsoleWrapper.RunTestsWithCustomTestHost(...)) with two test assemblyies.
Cancel the test session (via IVsTestConsoleWrapper.CancelTestRun) after all test in the first assembly have been executed.

Expected behavior

TestRunComplete event should be triggered.
Session should stop as soon as possible.

Actual behavior

No TestRunComplete event is raised and session will hang or timeout (if a timeout has been set).

Diagnostic logs

Here are the parameters:
(TestExecution.GetTestRunnerProcessStartInfoForRunAll) -> {"Version":7,"MessageType":"TestExecution.GetTestRunnerProcessStartInfoForRunAll","Payload":{"Sources":["C:\\Users\\cyrilledupuydauby\\source\\repos\\stryker-net\\integrationtest\\TargetProjects\\NetCoreTestProject.NUnit\\bin\\Debug\\net6\\NetCoreTestProject.NUnit.dll","C:\\Users\\cyrilledupuydauby\\source\\repos\\stryker-net\\integrationtest\\TargetProjects\\NetCoreTestProject.XUnit\\bin\\Debug\\net6\\NetCoreTestProject.XUnit.dll"],"TestCases":null,"RunSettings":"<RunSettings>\r\n<RunConfiguration>\r\n <CollectSourceInformation>false</CollectSourceInformation>\r\n <MaxCpuCount>1</MaxCpuCount>\r\n<CollectDataForEachTestSeparately>true</CollectDataForEachTestSeparately><DisableParallelization>true</DisableParallelization>\r\n<DesignMode>false</DesignMode>\r\n</RunConfiguration>\r\n</RunSettings>","KeepAlive":false,"DebuggingEnabled":false,"TestPlatformOptions":{"TestCaseFilter":null,"FilterOptions":null,"CollectMetrics":false,"SkipDefaultAdapters":false},"TestSessionInfo":null}}
Log extract (focus on what appear to be relevant):

...
TpTrace Verbose: 0 : 16408, 5, 2023/06/03, 11:39:58.381, 640147486080, vstest.console.exe, ParallelProxyExecutionManager: Start execution. Total sources: 2
...
TpTrace Verbose: 0 : 16408, 6, 2023/06/03, 11:39:59.369, 640157275857, vstest.console.exe, ParallelProxyExecutionManager: Execution started. Started clients: 1
...
TpTrace Verbose: 0 : 16408, 10, 2023/06/03, 11:40:01.411, 640177680892, vstest.console.exe, ParallelProxyExecutionManager: HandlePartialRunComplete: Total completed clients = 1, Run complete = False, Run canceled: False.
...
TpTrace Verbose: 0 : 16408, 13, 2023/06/03, 11:40:01.438, 640177955740, vstest.console.exe, ParallelProxyExecutionManager: Execution started. Started clients: 1
...
TpTrace Verbose: 0 : 16408, 12, 2023/06/03, 11:40:01.539, 640179007126, vstest.console.exe, ParallelProxyExecutionManager: HandlePartialRunComplete: Total completed clients = 2, Run complete = False, Run canceled: True.
// note: Run complete should be true, as there is no other run to execute
...
TpTrace Verbose: 0 : 16408, 14, 2023/06/03, 11:40:08.630, 640249870357, vstest.console.exe, TestRunRequest.OnTestSessionTimeout: calling cancellation as test run exceeded testSessionTimeout 10250 milliseconds
// timeout triggers a few seconds later

Complete log file
Runner 2-log.txt

Environment

Windows 11 (running with Parallel Desktop on Mac OS 13.5
VsTest 17.6.1
VisualStudio 17.6.2

Further details:

Looking VsTest sources (more specifically Microsoft.VisualStudio.TestPlatform.CrossPlatEngine) it looks like the problem lies within 'ParallelProxyExecutionManager.HandlePartialRunComplete'
the stopping condition runCompletedClients == _runStartedClients does not make sense for me.

            allRunsCompleted = testRunCompleteArgs.IsCanceled || _abortRequested
                ? _runCompletedClients == _runStartedClients
                : _runCompletedClients == _availableWorkloads;

because _runStartedClient appears to be the number of concurrent test sessions (so matching concurrency settings), while _runCompletedClients is the number of test sessions completed so far. I am guessing that the stopping condition for cancellation is making sure there is no longer any active session.
Hope this helps

[nohwnd]

I tried to repro, but it never was hanging for me. I know that I've been doing some fixes in that code some time ago, so maybe it is fixed. This is the easiest wayt to try to reproduce probably:

main...nohwnd:vstest:cancel-run

(And then just run the Playground project).

Does it repro for you?

[dupdob]

Hi
thanks for the update. I had a look at vstest code, and it has been seriously revised. I had implemented a work around in Stryker; I will disable it and see it fixes things (hope I can find a problematic project, because it was a long time ago).
The workaround was to ensure we have a specific vstest session per test assembly.
I will try and keep you posted

[dupdob]

I removed the work around and used Stryker against a large and complex project that was likely to exhibit the issue.
Everything went fine. I close this issue.

[nohwnd]

Perfect, thanks for confirming.