Unit of isolation = process and unit of scheduling = test class or custom container?

[rkieslinger]

Currently we are executing our unit and integration tests in parallel using the following settings in the .runsettings file:

<RunSettings>
    <RunConfiguration>
        <MaxCpuCount>0</MaxCpuCount>
    </RunConfiguration>
</RunSettings>

This enables parallelization, but just per assembly. In our case, this is not really ideal, because in some cases we have a few fast assemblies, and just 1 or 2 very large and long running assemblies. So parallelization per assembly does not help much there.

Here's a screenshot, how that looks like (every line is an assembly and every bar with a different color is a test):

I read here that it's possible to change the "unit of scheduling" from assembly to "test class" or even "test method". This seems exactly what we need, but the document states, that this will change the "unit of isolation" from process to thread and I'm pretty sure, our code will not support that (Singletons, static code, ...).

So what I'm asking is, as I wrote in the title: Is it possible to still use an unit of isolation of process, but change the unit of scheduling from assembly to test class or maybe be able to pass custom containers?

I also stumbled upon this tool, which seems to do exactly what we want, but it's outdated and it would be nice to not need an additional tool for that.

Thank you in advance!

[elgatov]

Try adding this to the .runsettings file

<MSTest>
    <Parallelize>
        <!--<Workers>0</Workers>-->
        <Scope>MethodLevel</Scope>
    </Parallelize>
</MSTest>

[rkieslinger]

@elgatov Wouldn't that change the Unit of isolation to thread? That's the problem, because that won't work with our code. What I'm looking for is a way to parallelize tests per test class or maybe a custom container and still have process isolation.

[nohwnd]

@rkieslinger If I get you right you want to discover your tests, and then split them to groups per assembly, and then further split them to several groups from each assembly based on some rule. And then each group should run in a separate process.

So e.g.:

dll1- 1000 tests, dll2 - 10 tests, dll3 - 3000 tests

Would become:

dll1- 1000 tests,
dll2 - 10 tests,
dll3 - 1000 tests,
dll3 - 1000 tests,
dll3 - 1000 tests

And would run as:

testhost.exe - dll1- 1000 tests,
testhost.exe - dll2 - 10 tests,
testhost.exe - dll3 - 1000 tests,
testhost.exe - dll3 - 1000 tests,
testhost.exe - dll3 - 1000 tests

This is not currently (easily) possible, for few reasons:

1. vstest.console runs tests directly as it finds them when running in console mode. We could however add that possibility, if we can come up with a good enough reason to do so. And a strategy that would work reasonably well for everyone.
2. vstest.console internally splits testcases into groups based on filename, even if you provide it a bunch of testcases from IDE, and will run 1 process for file name. I am doing some baby steps toward changing that in Run multiple target frameworks and architectures in single vstest.console #3412
3. Test frameworks don't have a good way to control what they want to parallelize on process level, because that needs to be orchestrated by test platform (vstest), not the test framework itself (mstest).

[nohwnd]

@rkieslinger moved it under test platform, because that is where it would have to be implemented.

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform