Unable to filter to just tests with default TestCategory

[ghost]

Description

IMPORTANT: if the defect is reproduced only in a workflow from within the Visual Studio IDE then do not report the issue here - instead, please report it using Visual Studio's "Send Feedback" option that can be accessed from the Help menu OR using this link https://developercommunity.visualstudio.com.

For a defect reproducable from the vstest command line, describe the issue you've observed.

First reported by a customer in 2013 before it was part of dotnetsdk, the --filter option doesn't permit empty string when setting TestCategory filter. This prevents uses from selecting just those test where TestCategory was not specified. As there's no way to require TestCategory to be set, this effectively prevents users from running just the "default" test of unit tests with a simple self contained TestCategory query.

Steps to reproduce

$ dotnet new test
The template "MSTest Test Project (.NET Core)" was created successfully.

$ dotnet test --filter "TestCategory="
Determining projects to restore...
All projects are up-to-date for restore.
You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
test -> C:\scratch\test\bin\Debug\net5.0\test.dll
Test run for C:\scratch\test\bin\Debug\net5.0\test.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.9.0-preview-20210125-03
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Incorrect format for TestCaseFilter Error: Invalid Condition 'TestCategory='. Specify the correct format and try again. Note that > the incorrect format can lead to no test getting executed.
No test matches the given testcase filter TestCategory= in C:\scratch\test\bin\Debug\net5.0\test.dll

[Expected]
.

[Actual]

Expected behavior

Run the one test, and passes

Actual behavior

Fails, see above log

Diagnostic logs

N/A

Environment

$ ver

Microsoft Windows [Version 10.0.19042.867]

$ dotnet --list-sdks
2.1.514 [C:\Program Files\dotnet\sdk]
2.1.518 [C:\Program Files\dotnet\sdk]
2.1.522 [C:\Program Files\dotnet\sdk]
2.1.814 [C:\Program Files\dotnet\sdk]
3.1.302 [C:\Program Files\dotnet\sdk]
3.1.407 [C:\Program Files\dotnet\sdk]
5.0.103 [C:\Program Files\dotnet\sdk]
5.0.104 [C:\Program Files\dotnet\sdk]
5.0.200-preview.21077.7 [C:\Program Files\dotnet\sdk]
5.0.200 [C:\Program Files\dotnet\sdk]
5.0.201 [C:\Program Files\dotnet\sdk]
6.0.100-preview.1.21103.13 [C:\Program Files\dotnet\sdk]

[Sanan07]

@aaronla-ms I didn't fully understand your concern. What do you want to achieve by running that command?

If you want to run all tests which do not have TestCategory attribute you can use this:

dotnet test test.dll --filter TestCategory!~[A-Za-z]*

Please use this doc for more info.

[ghost]

No dice. That still executed some tests I was trying not to run. E.g.:

[TestMethod]
public void TestMethod1()
{
}

[TestMethod]
[TestCategory("RequiresSpecialEnvironment")]
public void TestMethod2()
{
    Assert.AreEqual("SPECIAL", System.Environment.GetEnvironmentVariable("SPECIAL"));
}

aaronla@aaronla-rabbit:test$ dotnet test --filter TestCategory!~[A-Za-z]*
Determining projects to restore...
All projects are up-to-date for restore.
You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
test -> C:\scratch\test\bin\Debug\net5.0\test.dll
Test run for C:\scratch\test\bin\Debug\net5.0\test.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.9.0-preview-20210125-03
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed TestMethod2 [24 ms]
Error Message:
Assert.AreEqual failed. Expected:. Actual:<(null)>.
Stack Trace:
at test.UnitTest1.TestMethod2() in C:\scratch\test\UnitTest1.cs:line 17

Failed! - Failed: 1, Passed: 1, Skipped: 0, Total: 2, Duration: 27 ms - test.dll (net5.0)

I've got thousands of tests. I'd rather not have to go through and specially annotate them. Not only would that be unnecessary churn, but it would be error prone to expect devs to specially annotate all normal unit tests.

Previously, we used a different runner that let us just filter to only run the tests with no special categories set, but we're trying to migrate to retail tools.

[Sanan07]

You are right, it checks for usual string.contains, I thought it had regular expression support.

Not sure how to do it now, but let us check, if we can do this in other way with already implemented functionality.

[Evangelink]

Relates to #1724

[jochenwezel]

I'd like my tests to run with dotnet sdk 7 and nunit framework and still all tests are running (instead of just running all tests without category).

This is my filter expression to select all empty categories: TestCategory!~.+

Would be great to see some progress at this issue.

[faiteanu]

Using a RunSettings file in a dotnet 8 SDK project, I found the following syntax to work for tests without an assigned TestCategory:
TestCategory = null

A minimal example of a RunSettings file:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
	<TestCaseFilter>(TestCategory = null) | ((TestCategory != Integration) &amp; (TestCategory != UnfinishedFeature))</TestCaseFilter>
    </RunConfiguration>
</RunSettings>

[jochenwezel]

Using a RunSettings file in a dotnet 8 SDK project, I found the following syntax to work for tests without an assigned TestCategory: TestCategory = null

A minimal example of a RunSettings file:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
	<TestCaseFilter>(TestCategory = null) | ((TestCategory != Integration) &amp; (TestCategory != UnfinishedFeature))</TestCaseFilter>
    </RunConfiguration>
</RunSettings>

I immediately tested this good news from @faiteanu, so I updated my dotnet test command like this
dotnet test -v minimal "./Test.dll" --filter "(TestCategory = null)"

Unfortunately, it doesn't work out for my scenario: it doesn't find my unassigned tests :-(
No test matches the given testcase filter 'TestCategory=null'

[eFail]

+1 on needing this. When I run a PR Build, I would like to specify that all regular unit tests (those without an assigned category) get run. But there is no way to do this, outside of specifying a != condition for every category that has been used. :(

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform