Add runner support for v3 cross-process RPC

[bradwilson]

Given the v3 architecture where unit test projects are programs means we need to add support in v3 runner utility to be able to launch these separate processes for meta-runners (like xunit.v3.runner.console).

More details TBD.

[remcomulder]

Is it intended for the runner API to also use this mechanism? I ask as it will be difficult for us to implement integration with xunit v3 if the test execution happens in a sub-process outside of the outer runner's control.

Ideally, we'd want to work with xunit v3 in the same basic manner as v2 (i.e. invoking the v3 runner utility as a service which calls the test code inside the current process).

I accept the reasoning behind having the test code inside the user's own EXE process as this definitely solves issues with cross compatibility of so many platform versions. However, in the case of NCrunch (and likely other runners), we've already had to solve these problems. Calling into the runner directly would help with leveraging our existing infrastructure.

[bradwilson]

@remcomulder wrote:

Is it intended for the runner API to also use this mechanism?

Yes, this is the primary way it will be surfaced is through the runner API.

I ask as it will be difficult for us to implement integration with xunit v3 if the test execution happens in a sub-process outside of the outer runner's control.

Our current thought here is that the runner API will expose the notion that, when a sub-process has been launched, that process ID will be made available to the runner. At the moment, this is planned only for v3, but there is a possibility that for better performance and isolation, we may end up also supporting v2 projects running in a separate process (instead of just a separate app domain).

The notification here is critical for things like being able to attach debuggers or profilers.

invoking the v3 runner utility as a service which calls the test code inside the current process

Running in-process will not be an option for v3. I'm interested to know what other facilities you might need beyond just process ID that we could make available in the runner API.

in the case of NCrunch (and likely other runners), we've already had to solve these problems

It's just not possible for us to do this. The assembly loading and linking resolution issues are simply too complex for us to maintain; we tried in v2 with our own .NET Core-based console runner, and it was frankly a complete disaster. It also caused us to not be able to take any third party dependencies outside of the core framework itself; not even dependencies from Microsoft-owned projects that were imported via NuGet.

[remcomulder]

Running in-process will not be an option for v3. I'm interested to know what other facilities you might need beyond just process ID that we could make available in the runner API.

Unfortunately, this is not a simple question for me to answer directly. We've stacked a 14 year long project on the ability to have complete control over the environment in which a test is being executed. There's a ton of stuff we've built that I know won't work without this control, and probably at least as much again that I can't account for due to the scale of the change. It's a big paradigm shift for us and it comes with a fair share of unknown unknowns.

Because we run user code continuously in transiently wired environments, our reliability and performance constraints go far beyond those you would need to cater for in the standard xunit runners. If the xunit API takes responsibility for the process and environment, these constraints of ours get pushed down onto xunit. I doubt I could convince you to care about them and we won't be able to provide our current features without them being addressed.

It's just not possible for us to do this. The assembly loading and linking resolution issues are simply too complex for us to maintain; we tried in v2 with our own .NET Core-based console runner, and it was frankly a complete disaster. It also caused us to not be able to take any third party dependencies outside of the core framework itself; not even dependencies from Microsoft-owned projects that were imported via NuGet.

I understand the nightmare that comes with needing to host .NET Core (we have some serious war scars from this ourselves). For what it's worth, I think that your chosen architecture makes sense and I think you should absolutely do it. But what about providing a switch that allows a runner to instruct xunit to simply call directly into test execution instead of isolating this in a sub-process? To me it seems like a small abstraction could enable us to both have what we want here. I expect that other runners will probably want to have this option too.

A situation that I'm trying to avoid is one where we're forced to write our own xunit emulator because we can't integrate with the API in a way that gives our users the experience they can reasonably expect from NCrunch. Such an approach would feel like a tragic lost opportunity and would also dilute the value of work done to xunit itself. We had to do this with MSTest. I really don't want to do it again.

[davidmatson]

I really like the v3 choice of making tests projects compile as programs/exes - I've worked on integrating a number of test frameworks into an existing, complex, internal testing infrastructure, including GoogleTest, Catch2, and Boost.Test, and they also compile to exes and their "just run the exe to execute the tests" simplicity and ease launching/debugging/etc. is hard to beat.

Two key properties that have made integrating with existing testing infrastructure feasible for those cases has been:

1. Test output can be read programmatically in a streaming fashion.
   Catch2 and Boost.Test both support logging XML to stdout, and this XML streams in as the tests run. (So something like XmlReader or XmlLite can process the data as the test runs.)
   GoogleTest has something arguably nicer, with a simple "query string-like event data lines" socket output option used to communicate results to Eclipse while a test is running (though I'm still trying to get them to merge support for streaming output on Windows, and currently have to fall back to parsing stdout, which is rather ugly).
2. Streaming output support is available-by-default in any test program/exe, via a command-line switch.
   Existing, compiled test programs "just work" when run via another harness capturing this streaming output and reporting it back to another system. (For example, users can take an existing, compiled test program and see the results of the test, via the infrastructure access mechanisms, while the test is still running, or get partial output if it hangs).

For v3 RPC, have there been further thoughts on exactly what the RPC contract would be, and how it would be triggered from outside the test program?
It would be great if it would allow such streaming output during test execution, and if it's included in any xUnit exe by default, with an optional command-line switch, or something. (Perhaps similar to GoogleTest's --gtest_stream_result_to.)

[davidmatson]

Another angle to consider here is in designing the RPC contract is how easy it is to nest/delegate test execution. This discussion has one example of where that can provide elegant solutions for problems that are otherwise very challenging.

Stated another way, a desirable property here would be that the contract between the test program and any outside runner is something that's easy to pass across multiple process boundaries, such as:

1. results streamed via Console.Out/Console.Error
2. data passed to an observer via sockets/RPC, specified via a command-line parameter (so another process can communicate out via the same port/etc. specified on the command line)

If the protocol is also easy to "proxy" (add some output data from this process, then pass back anything coming from another process), that's a plus.

[bradwilson]

I've been discussing reusing the new VSTest protocol for our RPC. It's based on JSON-RPC, sometimes over TCP and sometimes over a named pipe.

[bradwilson]

It's very early, and as such the system is under-documented. https://github.com/microsoft/testfx/blob/main/docs/mstest-runner-protocol/001-protocol-intro.md

[davidmatson]

Interesting - yeah, I wondered if that would be a good option.

It might also be worth considering whether the command line + process output could serve the same purpose. That's the approach the major C++ testing frameworks seem to take (GoogleTest, Catch2, Boost.Test), and it seemes to work well there. For example, to list tests, just pass that command-line switch; to execute tests, just run the exe; if you want to run the exe under a debugger, do so, just like with any other exe; if you want to run a single test, pass the filter switch on the command line, etc.
For output, having it default to human-readable but also support machine-readable (again, via the command line) seems to be a common approach.

Since the exe command line, and output are needed anyway, re-using them for discover/execution/reporting could be a really simple solution, and it means any harness doesn't even need to have something like a JSON-RPC parser to make use of the exe - even a human can do it just by typing a command, running a PowerShell script, etc.

The only place I notice in the new MSTest protocol that looked like more that what the frameworks above handle would be attachments, though I'm not sure if xUnit does those anyway, and those could just be files written to the working directory, or something like that.

Thoughts?

[bradwilson]

I had thought about using the console instead of TCP, it just seemed like TCP would offer me a fairly transparent way to support remoting (the argument could just be a port, but could also be a hostname) and/or container-based testing. It certainly sounds like they're going to be supporting a variety of communication mechanisms (as previously mentioned, at least TCP and named pipes), so I'm in a little bit of a holding pattern waiting for them to start to formalize things a little better.

What they have right now is pretty hacked together and un(der)-documented, so I had to decompile some of their closed source packages in order to make anything work, but I did at least get integration working with their dotnet test hacked override in a branch: https://github.com/xunit/xunit/tree/microsoft-testing-platform. This supports their undocumented named pipe system for MSBuild which I'm not going to merge yet, since it's not clear whether I should be doing this at all yet. They were pretty excited to get started with their adapter for the Visual Studio runner but I'm less interested in this right now. It (theoretically) offers performance benefits, but it just seems like they're too early for me to commit to this strategy, without even knowing what the backward compatibility story is.