Cancelling a test session may lead to a block #4527
Description
Description
Running a test session with multiple test sources with parallelism disabled will lead to a blocked/hanged VsTest if the session is cancelled once the first sub session is completed.
Steps to reproduce
Note that is has been reproduced while working on Stryker.Net; note sure how to replicate via command line.
Start a VsTest session (via IVsTestConsoleWrapper.RunTestsWithCustomTestHost(...)) with two test assemblyies.
Cancel the test session (via IVsTestConsoleWrapper.CancelTestRun) after all test in the first assembly have been executed.
Expected behavior
TestRunComplete event should be triggered.
Session should stop as soon as possible.
Actual behavior
No TestRunComplete event is raised and session will hang or timeout (if a timeout has been set).
Diagnostic logs
Here are the parameters:
(TestExecution.GetTestRunnerProcessStartInfoForRunAll) -> {"Version":7,"MessageType":"TestExecution.GetTestRunnerProcessStartInfoForRunAll","Payload":{"Sources":["C:\\Users\\cyrilledupuydauby\\source\\repos\\stryker-net\\integrationtest\\TargetProjects\\NetCoreTestProject.NUnit\\bin\\Debug\\net6\\NetCoreTestProject.NUnit.dll","C:\\Users\\cyrilledupuydauby\\source\\repos\\stryker-net\\integrationtest\\TargetProjects\\NetCoreTestProject.XUnit\\bin\\Debug\\net6\\NetCoreTestProject.XUnit.dll"],"TestCases":null,"RunSettings":"<RunSettings>\r\n<RunConfiguration>\r\n <CollectSourceInformation>false</CollectSourceInformation>\r\n <MaxCpuCount>1</MaxCpuCount>\r\n<CollectDataForEachTestSeparately>true</CollectDataForEachTestSeparately><DisableParallelization>true</DisableParallelization>\r\n<DesignMode>false</DesignMode>\r\n</RunConfiguration>\r\n</RunSettings>","KeepAlive":false,"DebuggingEnabled":false,"TestPlatformOptions":{"TestCaseFilter":null,"FilterOptions":null,"CollectMetrics":false,"SkipDefaultAdapters":false},"TestSessionInfo":null}}
Log extract (focus on what appear to be relevant):
...
TpTrace Verbose: 0 : 16408, 5, 2023/06/03, 11:39:58.381, 640147486080, vstest.console.exe, ParallelProxyExecutionManager: Start execution. Total sources: 2
...
TpTrace Verbose: 0 : 16408, 6, 2023/06/03, 11:39:59.369, 640157275857, vstest.console.exe, ParallelProxyExecutionManager: Execution started. Started clients: 1
...
TpTrace Verbose: 0 : 16408, 10, 2023/06/03, 11:40:01.411, 640177680892, vstest.console.exe, ParallelProxyExecutionManager: HandlePartialRunComplete: Total completed clients = 1, Run complete = False, Run canceled: False.
...
TpTrace Verbose: 0 : 16408, 13, 2023/06/03, 11:40:01.438, 640177955740, vstest.console.exe, ParallelProxyExecutionManager: Execution started. Started clients: 1
...
TpTrace Verbose: 0 : 16408, 12, 2023/06/03, 11:40:01.539, 640179007126, vstest.console.exe, ParallelProxyExecutionManager: HandlePartialRunComplete: Total completed clients = 2, Run complete = False, Run canceled: True.
// note: Run complete should be true, as there is no other run to execute
...
TpTrace Verbose: 0 : 16408, 14, 2023/06/03, 11:40:08.630, 640249870357, vstest.console.exe, TestRunRequest.OnTestSessionTimeout: calling cancellation as test run exceeded testSessionTimeout 10250 milliseconds
// timeout triggers a few seconds later
Complete log file
Runner 2-log.txt
Environment
Windows 11 (running with Parallel Desktop on Mac OS 13.5
VsTest 17.6.1
VisualStudio 17.6.2
Further details:
Looking VsTest sources (more specifically Microsoft.VisualStudio.TestPlatform.CrossPlatEngine) it looks like the problem lies within 'ParallelProxyExecutionManager.HandlePartialRunComplete'
the stopping condition runCompletedClients == _runStartedClients does not make sense for me.
allRunsCompleted = testRunCompleteArgs.IsCanceled || _abortRequested
? _runCompletedClients == _runStartedClients
: _runCompletedClients == _availableWorkloads;
because _runStartedClient appears to be the number of concurrent test sessions (so matching concurrency settings), while _runCompletedClients is the number of test sessions completed so far. I am guessing that the stopping condition for cancellation is making sure there is no longer any active session.
Hope this helps