Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestCategory no longer works for VsTest #4782

Closed
aaronportier opened this issue Dec 7, 2023 · 10 comments
Closed

TestCategory no longer works for VsTest #4782

aaronportier opened this issue Dec 7, 2023 · 10 comments

Comments

@aaronportier
Copy link

I changed my appium dotnet version and had to add yaml run settings files for the cloud provider I am using. I am running this in Azure Pipeline with VsTest. The output seems like it knows how many tests should run with the Category but gives this message:
now IsValidServiceResponse: Received None command..Service Workflow is not active
Creating run for selected test assemblies with following parameters
SourceFilter: ThisWontBeUsed TestCaseFilter: TestCategory=UserSubscription
Run title: TestRun_TestAutomationBSIntegration_Release-598

Why will it not be used? I even tried adding to the Runsettigns filter (TestCategory = UserSubscription) but that still gives me the same error

@nohwnd
Copy link
Member

nohwnd commented Dec 13, 2023

I don't think I follow what is not working. Can you please give a list of simple steps to reproduce this?

@aaronportier
Copy link
Author

For a repo of steps just use the above chnages to your appium project and then TestCategory added in Test Criteria for vsTest will no longer work.
I can see in the logs it says "This Wont Be Used" but why will it not be used?

Creating run for selected test assemblies with following parameters
SourceFilter: ThisWontBeUsed TestCaseFilter: TestCategory=UserSubscription
Run title: TestRun_TestAutomationBSIntegration_Release-606

@aaronportier
Copy link
Author

I am using Run Settings and I have tried putting User Subscription in there as well but still it runs all tests not just the category I choose. The correct amount of tests is 16 but they all run without Category filtering them.
2023-12-14T19:59:47.0011882Z Test filter criteria : TestCategory=UserSubscription...
2023-12-14T19:59:47.9808239Z Number of test cases per batch : 16...
Creating run for selected test assemblies with following parameters
SourceFilter: ThisWontBeUsed TestCaseFilter: TestCategory=UserSubscription
Run title: TestRun
In the VsTest-testAssemblies on Azure Pipelines my Agent Job I have Test filter criteria TestCategory=UserSubscription.
What am I missing why will it not filter by Category for me? It used to work until I had to create Run Settings file and upgrade my dotnet appium to the 5.0 version.

@aaronportier
Copy link
Author

Does anyone know where this line SourceFilter: ThisWontBeUsed might be coming from? In the past I never saw it now I have run setting files but they should not be doing anything to source filters.

@aaronportier
Copy link
Author

I was able to get everything to run by turning Batch off. Unfortunately turning off Batch options makes my pipeline unable to re-run failed tests.I know that was a work around was the issue ever fixed.

@aaronportier
Copy link
Author

Is anyone else experiencing or reporting this issue. It was working but now its not and I cant figure out why? The batching if clicked will not run using your Category Filter but If you turn batching off it will not re-run failed tests. Saying that it could not find them.

@aaronportier
Copy link
Author

This is the state of my VSTest any ideas on next steps?

@nohwnd
Copy link
Member

nohwnd commented Jul 8, 2024

Could you sum this up into a simple repro please?

I think what is happening is that running with batches is running your tests with DTAgent, and that ignores commandline parameters like the filter.

Are you setting the filter on commandline or in runsettings file?

@aaronportier
Copy link
Author

aaronportier commented Jul 8, 2024 via email

@nohwnd
Copy link
Member

nohwnd commented Jul 8, 2024

Please share a repro project and a command that shows the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants