Is it possible to run each test isolated in a separate process?

[siegfriedpammer]

One feature that has been often requested in xUnit or NUnit (I am too lazy to look up/provide the relevant GH issues now) and is somewhat required when you work with an unmanaged component that is hard to isolate and you want to run tests in parallel, is the ability to execute each test in a separate process. Separate threads or AppDomains don't cut it, because a) AppDomains are no longer a thing in recent versions of .NET and b) the unmanaged part lives outside of the AppDomain anyway.

Having this feature would be a strong argument for moving to TUnit.

• Is this something that is possible out of the box using an attribute?
• Is this something that can be implemented using "Executors"?
• Would this also work when running tests in VS?
• Would it be possible to conveniently debug a single test from VS?

Thanks for your answers in advance!

[thomhurst]

It'd be slow spawning new processes - But my first thought would be you could create a custom executor which executes your test project again with a specific filter for that one test.

public class Tests
{
    [TestExecutor<SeparateProcessExecutor>]
    [Test]
    public async Task Test()
    {
        // ...
    }
}

public class SeparateProcessExecutor : ITestExecutor
{
    public static bool IsSeparateProcess => Environment.GetEnvironmentVariable("IS_SEPARATE_PROCESS") == "true";
    public Task ExecuteTest(TestContext context, Func<Task> action)
    {
        if (IsSeparateProcess)
        {
            return action();
        }
        
        var testDetails = context.TestDetails;

        var classInformation = testDetails.TestClass;
        
        return Process.Start(new ProcessStartInfo
        {
            FileName = "dotnet",
            ArgumentList = { "run", "--treenode-filter", $"/{classInformation.Assembly.Name}/{classInformation.Namespace}/{classInformation.Name}/{testDetails.TestMethod.Name}" },
            WorkingDirectory = GetProjectDirectory(),
            Environment = { ["IS_SEPARATE_PROCESS"] = "true" }
        })!.WaitForExitAsync();
    }
}

Edit:

Would it be possible to conveniently debug a single test from VS?

I don't think so (not easily at least). I've never delved into how debuggers attach to things. You'd need to attach a debugger to the newly spawned process, but I don't know how you'd do that!

[siegfriedpammer]

Thank you for the quick response and sample code. That's a good starting point to explore.

You'd need to attach a debugger to the newly spawned process

So, debugging needs System.Diagnostics.Debugger.Launch(); that's OK, I guess.

It'd be slow spawning new processes

Yeah, that's a trade-off I'd be willing to accept in my case, the feature is entirely opt-in. I guess, it should be possible to use limited parallelism, to not overload the machine with new processes.

Another idea would be to implement something like "sharding" - would be possible to randomly/dynamically distribute test cases across a fixed number of test runners?

[thomhurst]

Another idea would be to implement something like "sharding" - would be possible to randomly/dynamically distribute test cases across a fixed number of test runners?

This is something I want to do in the future via a custom filter. Something like:

dotnet run --batch-filter 1/6
dotnet run --batch-filter 2/6
etc.

But the filter mechanism is Microsoft.Testing.Platform code, and currently it's not extensible. So I can't write custom filters until they open it up.

[siegfriedpammer]

But the filter mechanism is Microsoft.Testing.Platform code, and currently it's not extensible. So I can't write custom filters until they open it up.

That's a bummer... thank you for the info... I have not looked into how filtering is done in dotnet test... I always assumed that it is implemented in the test framework, because I found that the filter syntax/capabilities are different in NUnit or xUnit...