Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

17.11 - error output is treated as error and fails tests #5184

Open
birarroshan opened this issue Aug 21, 2024 · 8 comments · Fixed by #5192
Open

17.11 - error output is treated as error and fails tests #5184

birarroshan opened this issue Aug 21, 2024 · 8 comments · Fixed by #5192
Labels

Comments

@birarroshan
Copy link

birarroshan commented Aug 21, 2024

ℹ️ workarounds

Set ForwardStandardOutput to false in your runsettings:

<RunSettings>
  <RunConfiguration>
    <ForwardStandardOutput>false</ForwardStandardOutput>
  </RunConfiguration>
</RunSettings>

or

set VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1 environment variable.


Description

We are running Appium-WinAppdriver UI tests with ADO pipeline. After upgrading to 17.11.0, VStest task is reporting appium debug logs as error.

Steps to reproduce

  1. Install 17.11.0 for VSTestPlatform - Latest Stable is 17.11.0
  • task: VisualStudioTestPlatformInstaller@1
    inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'latestStable'
  1. Run test using
    • task: VSTest@2
      inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: "Test.dll"
      searchFolder: $(Pipeline.Workspace)
      testFiltercriteria: "$(testFilterExpression)"
      vsTestVersion: 'toolsInstaller'

Expected behavior

Tests should run as usual without any error logs

Actual behavior

Appium DEBUG logs are being reported as Error.
image
Note - We have not enabled any debug logging for appium

Also, specFlow logs are reported as Warning.

Diagnostic logs

Could not collect Diagnostic logs. Will upload it one collected.

Environment

OS - Hosted Pool with windows 10 base image

Aditional Notes -

ADO job is passing, but all errors are reported under Error tab of pipeline run

@shapea
Copy link

shapea commented Aug 28, 2024

I can confirm the same issue in vstest.console.exe as of version 17.11.0. As soon as the update was released, I had ADO build pipelines with no other changes start to fail with stderr output. Digging further into it, there seems to be a bug with some of the new output forwarding changes in the 17.11.0 release as noted here: https://github.com/microsoft/vstest/releases.

@nohwnd if bugs were ducks, this certainly walks, talks and quacks like a duck. From what I can tell, it appears that stdout output from tests gets erroneously forwarded to stderr instead. This causes benign test output to become errors. I can even verify this by manually calling vstest.console.exe with tests exhibiting this behaving and sending stderr output to nul (i.e. 2> nul).

I managed to sleuth out a workaround for this issue using the information here: #4998

For folks impacted by this, it seems you can update your test .runsettings to include this to disable the fowarding causing the issues:

<RunSettings>
  <RunConfiguration>
    <ForwardStandardOutput>false</ForwardStandardOutput>
  </RunConfiguration>
</RunSettings>

@birarroshan this may help unblock you, it did for me in ADO for the same reason. :)

@nohwnd
Copy link
Member

nohwnd commented Aug 29, 2024

The change was introduced here:

135b4f2#diff-d494c9e9058f3e044a79a007c308ad13da804fd3fb6d404462c09e3cd8003e50R59

This change then gets enabled to default in 17.11.

The ForwardStandardOutput option set to false is effective because it prevents forwarding the error stream as error messages.

it appears that stdout output from tests gets erroneously forwarded to stderr instead.

The appium logs go into error stream, not into standard output. All the streams get forwarded correctly, but you are right that this is not the desired behavior and the error stream should be sent as info messages instead, not as errors, because applications like to use error stream as output for logs, or info messages.

@birarroshan there is also VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1 env variable, if that is easier for you to set. as documented in #4998

@nohwnd
Copy link
Member

nohwnd commented Aug 29, 2024

Re-opening for awareness.

This bug affects VS17.11, TestPlatform 17.11, net8.0, and all newer versions.

Each product has it's own servicing calendar, will update this thread when fixes are released.

  • release 17.11.1 to nuget
  • insert 17.11.1 in VS
  • insert 17.11.1 into net8
  • insert 17.12.x into net9
  • insert 17.12.x into net10

@shapea
Copy link

shapea commented Aug 29, 2024

@nohwnd in my case it's not coming from Appium, but rather an internal-only MS thing.

Edit: It took a while to pull in symbols and sources to step deeper into their native pinvoke'd code, but it's doing the same sort of thing and writing non-error output to stderr as well.

@nohwnd
Copy link
Member

nohwnd commented Sep 5, 2024

@nohwnd in my case it's not coming from Appium, but rather an internal-only MS thing.

Edit: It took a while to pull in symbols and sources to step deeper into their native pinvoke'd code, but it's doing the same sort of thing and writing non-error output to stderr as well.

Thanks for confirming.

@nohwnd nohwnd changed the title 17.11 - Appium Debug logs are treated as error 17.11 - error output is treated as error and fails tests Sep 6, 2024
@pavelhorak pavelhorak added sprint and removed sprint labels Sep 9, 2024
@bradyclifford
Copy link

If you add VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING: "1" to your runner, it does fix the issue.

However, it does prevent output from ITestOutputHelper - if you are even using it. Which is helpful when running tests because it groups the logs and shows them in each test, segregated from the other logs. It does the same when creating a report in GitHub on the runner.

If you use GitHub Actions, you can instead use the suggested GitHub Actions test logger mentioned in .Net's docs. Using this test logger, I was able to avoid declaring the VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING env. var and still output errors in my logs.

@nohwnd
Copy link
Member

nohwnd commented Oct 7, 2024

@bradyclifford this should be fixed in 17.11.1 do you still see the issue in that version?

@carstencodes
Copy link

Hi @nohwnd

we had the same issue here. The tests were failing with .NET SDK 8.0.401, which is shipping VsTest 17.11.0.

All tests passed, but the entire test suite was reported as failed due to messages on stderr.

I was not able to reproduce the behavior on my machine using .NET SDK 8.0.403, which ships VsTest 17.11.1

To me, the issue looks solved.

Regards

Carsten

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants