Issue With Skipping Tests Resurfaced in 13.8.1

[wakeboardfit]

Current behavior

We run in an Azure pipeline and after upgrading to 13.8.1, we are experiencing that many tests are just being skipped. This did not happen before and the code did not change outside of a few locator updates and whatnot.

When running in 13.6.4, our executions would look like

You can see the drop off where we upgraded versions

I saw there was an issue that was closed regarding something similar back in version 10.

Desired behavior

Cypress should not just skip tests for not reason.

Test code to reproduce

Everything runs locally so this seems only on Azure. We are running "windows-latest" agent.

Cypress Version

13.8.1

Node version

18.18.2

Operating System

Windows-latest Azure Pipeline

Debug Logs

I will have to get this setup in our pipeline and will add when I can.

Other

No response

[jennifer-shehane]

@wakeboardfit Are you recording to the Cloud? Could you link to a URL of the run if so?

Could you provide the command you use to run Cypress?

I'm not familiar with anything that would cause this. This is very strange. Can you link to the old issue?

[wakeboardfit]

Old issue: #22276

Command:

npx cypress run --spec cypress/e2e/*/**/*.feature --config video=true,screenshotOnRunFailure=true --env username=$(USERNAME),insurerUser=$(INSUREDNAME),password=$env:MAPPED_PASS,ls_api_username=$(USERNAME),ls_api_password=$env:MAPPED_PASS,tags=@admitted

We are not using the cypress cloud. I am still trying to figure out how to set the Debug flag in Azure pipeline as it says you need to run in command line (not powershell) and I need to see if I can make that run as admin.

[wakeboardfit]

Unless you guys have another way, I am not seeing a way to run Command Line as and Admin in Azure without self hosted agents. I put the debug command in our Power Shell step but I don't believe it did anything and your documentation pretty specifically says not Power Shell.

[jennifer-shehane]

@wakeboardfit I do see you are passing a tags flag. What plugin are you using to read in tags and run/not run tests based on that? Is that was would be causing some tests to be skipped? If they don't match the tag?

[wakeboardfit]

I need to keep digging in to this. The Pending correctly indicates all the tests that are != tag. It gets weird though with the skipping and potentially it's a error that is happening silently. I have yet to be able to trigger it on a local run but also need have more time to experiment and try to see what's going on. I am reverting our pipeline to 12.17.4 for now and will experiment with 13.8.X to see why that is failing and skipping when it didn't in the past.

[wakeboardfit]

Finally getting a chance to get back to this. I noticed this when examining the latest Skipped test.


While executing, the "Mocha Tests" will just fire and then all tests after that are Skipped. I'm theorizing it has something to do with a failure although if it were consistent then we would never get executions on a set of tests with 1 or more failures. I am currently attempting to remove any code from the After in the tests just to see if that is potentially causing the tests to consider themselves "Done" in the middle of a run.