Test Runner terminates before Logger completes

[davidguidali]

We are running multiple test suites on different azure devops test pipeline agents. Some of the test suites are very small, containing only one test. We also use a custom logger which inherits from ITestLoggerWithParameters. It implements some data enrichment and transformation logic and afterwards sends the results to an azure data explorer database. This implementation is done in the TestLoggerEvents (TestResults / TestRunComplete).

Now it seems that sometimes when only one test is executed, the test runner process terminates before the logger finished sending the test results to the cloud. All other scenarios with 2 or more tests run successfully.

It turned out that when setting a short Thread.Sleep in the AssemblyCleanup of the test adapter, this issue is solved, however this seems a little hacky to me.

Since the logger is in-proc with the runner, I am wondering why the runner does not wait until the logger tasks finished?

Is this a known issue and are there any better workarounds?

[Tyrrrz]

I seem to be having the same issue. TestRunComplete appears to trigger before all TestResult events trigger. Only seems to happen on macOS and Linux, but not on Windows. Not sure, but it also looks that it only happens when the test run finishes very quickly.

[nohwnd]

Either my approach is wrong, or this is not a problem anymore, I am trying in WSL on Ubuntu 20 04, dotnet 8.0.200,

// file Logger.cs
using Microsoft.VisualStudio.TestPlatform.ObjectModel;
using Microsoft.VisualStudio.TestPlatform.ObjectModel.Client;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace TestProject107.TestAdapter
{
    [FriendlyName("MyLogger")]
    [ExtensionUri("logger://Microsoft/TestPlatform/MyLogger/v1")]
    public class MyLogger : ITestLoggerWithParameters
    {
        public void Initialize(TestLoggerEvents events, Dictionary<string, string?> parameters)
        {
            events.TestRunComplete += (_, __) => Console.WriteLine("complete");
            events.TestResult += (_, __) => Console.WriteLine("result");
        }

        public void Initialize(TestLoggerEvents events, string testRunDirectory)
        {
            events.TestRunComplete += (_, __) => Console.WriteLine("complete");
            events.TestResult += (_, __) => Console.WriteLine("result");
        }
    }
}

<!-- file TestProject107.TestLogger.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="coverlet.collector" Version="6.0.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="3.1.1" />
    <PackageReference Include="MSTest.TestFramework" Version="3.1.1" />
  </ItemGroup>

  <ItemGroup>
    <Using Include="Microsoft.VisualStudio.TestTools.UnitTesting" />
  </ItemGroup>

</Project>

// file UnitTest1.cs
namespace TestProject107.TestAdapter
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
        }
    }
}

I am then running this in powershell in wsl and all the results are result followed by complete:

1..100 | % { "-$_-"; dotnet test bin/Debug/net8.0/TestProject107.TestLogger.dll --logger:MyLogger  }  |  sls "(-\d+-)|(complete)|(result)"

[Tyrrrz]

@nohwnd I don't remember exactly what I ended up doing, but I think I traced down the issue to the fact that I was using a test logger that came from the same assembly that I was actually testing, so it's probably a very rare use case.

[nohwnd]

That is what I am doing in the example as well. So It is probably fixed.