-
Notifications
You must be signed in to change notification settings - Fork 323
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
17.11 - error output is treated as error and fails tests #5184
Comments
I can confirm the same issue in vstest.console.exe as of version 17.11.0. As soon as the update was released, I had ADO build pipelines with no other changes start to fail with stderr output. Digging further into it, there seems to be a bug with some of the new output forwarding changes in the 17.11.0 release as noted here: https://github.com/microsoft/vstest/releases. @nohwnd if bugs were ducks, this certainly walks, talks and quacks like a duck. From what I can tell, it appears that stdout output from tests gets erroneously forwarded to stderr instead. This causes benign test output to become errors. I can even verify this by manually calling vstest.console.exe with tests exhibiting this behaving and sending stderr output to nul (i.e. 2> nul). I managed to sleuth out a workaround for this issue using the information here: #4998 For folks impacted by this, it seems you can update your test <RunSettings>
<RunConfiguration>
<ForwardStandardOutput>false</ForwardStandardOutput>
</RunConfiguration>
</RunSettings> @birarroshan this may help unblock you, it did for me in ADO for the same reason. :) |
The change was introduced here: 135b4f2#diff-d494c9e9058f3e044a79a007c308ad13da804fd3fb6d404462c09e3cd8003e50R59 This change then gets enabled to default in 17.11. The ForwardStandardOutput option set to false is effective because it prevents forwarding the error stream as error messages.
The appium logs go into error stream, not into standard output. All the streams get forwarded correctly, but you are right that this is not the desired behavior and the error stream should be sent as info messages instead, not as errors, because applications like to use error stream as output for logs, or info messages. @birarroshan there is also |
Re-opening for awareness. This bug affects VS17.11, TestPlatform 17.11, net8.0, and all newer versions. Each product has it's own servicing calendar, will update this thread when fixes are released.
|
@nohwnd in my case it's not coming from Appium, but rather an internal-only MS thing. Edit: It took a while to pull in symbols and sources to step deeper into their native pinvoke'd code, but it's doing the same sort of thing and writing non-error output to stderr as well. |
Thanks for confirming. |
If you add However, it does prevent output from ITestOutputHelper - if you are even using it. Which is helpful when running tests because it groups the logs and shows them in each test, segregated from the other logs. It does the same when creating a report in GitHub on the runner. If you use GitHub Actions, you can instead use the suggested GitHub Actions test logger mentioned in .Net's docs. Using this test logger, I was able to avoid declaring the |
@bradyclifford this should be fixed in 17.11.1 do you still see the issue in that version? |
Hi @nohwnd we had the same issue here. The tests were failing with .NET SDK 8.0.401, which is shipping VsTest 17.11.0. All tests passed, but the entire test suite was reported as failed due to messages on stderr. I was not able to reproduce the behavior on my machine using .NET SDK 8.0.403, which ships VsTest 17.11.1 To me, the issue looks solved. Regards Carsten |
ℹ️ workarounds
Set ForwardStandardOutput to false in your runsettings:
or
set
VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1
environment variable.Description
We are running Appium-WinAppdriver UI tests with ADO pipeline. After upgrading to 17.11.0, VStest task is reporting appium debug logs as error.
Steps to reproduce
inputs:
packageFeedSelector: 'nugetOrg'
versionSelector: 'latestStable'
inputs:
testSelector: 'testAssemblies'
testAssemblyVer2: "Test.dll"
searchFolder: $(Pipeline.Workspace)
testFiltercriteria: "$(testFilterExpression)"
vsTestVersion: 'toolsInstaller'
Expected behavior
Tests should run as usual without any error logs
Actual behavior
Appium DEBUG logs are being reported as Error.
Note - We have not enabled any debug logging for appium
Also, specFlow logs are reported as Warning.
Diagnostic logs
Could not collect Diagnostic logs. Will upload it one collected.
Environment
OS - Hosted Pool with windows 10 base image
Aditional Notes -
ADO job is passing, but all errors are reported under Error tab of pipeline run
The text was updated successfully, but these errors were encountered: