test Mix task should exit with 1 when --warnings-as-errors opt is used #14019
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously some warnings were not detected as errors, exiting the command with 1. An example is wronly naming a test file with the extension .ex instead of .exs A test app has been uploaded to: https://github.com/eksperimental-debug/debug_elixir_test_wrong_ext Previously: ❯ mix test --warnings-as-errors; echo $? Compiling 1 file (.ex) Generated debug_elixir_test_wrong_ext app warning: test/foo_test.ex does not match "*_test.exs" and won't be loaded Running ExUnit with seed: 194476, max_cases: 20 .. Finished in 0.00 seconds (0.00s async, 0.00s sync) 1 doctest, 1 test, 0 failures 0 Now: ❯ mix test --warnings-as-errors; echo $? Compiling 1 file (.ex) Generated debug_elixir_test_wrong_ext app warning: test/foo_test.ex does not match "*_test.exs" and won't be loaded Running ExUnit with seed: 560339, max_cases: 20 .. Finished in 0.00 seconds (0.00s async, 0.00s sync) 1 doctest, 1 test, 0 failures ERROR! Test suite aborted after successful execution due to warnings while using the --warnings-as-errors option 1 All tests will be executed, the warning is printed, but it prints an error message and exits with 1.
eksperimental
commented
Nov 25, 2024
Comment on lines
+121
to
+122
| capture_io(:stderr, fn -> | ||
| capture_io(fn -> |
There was a problem hiding this comment.
I wasn't able to find a way to catch both stdio and stderr in one go, so I resorted to this:
josevalim
reviewed
Nov 25, 2024
|
|
||
| ensure_touched(file) | ||
| assert {:ok, []} = Mix.Tasks.Compile.Erlang.run([]) | ||
| end) |
There was a problem hiding this comment.
Sorry, why was this file changed? It seems unrelated to this PR?
There was a problem hiding this comment.
It was part of the other PR #14018. It ended up here when I split up the commits.
|
Thank you for the PR. The --warnings-as-errors flag applies to compiler errors, not any warning that may be emitted, as mentioned in the docs. For example, if you get warnings in your test_helper.exs, those won't fail either. So we don't have plans to change this for now, just make it consistent with the compiler flag itself. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously some warnings were not detected as errors, exiting the command with 1.
A real case example is wrongly naming a test file with the extension .ex instead of .exs. This was making the CI flow not to fail because of this.
A test app has been uploaded to: https://github.com/eksperimental-debug/debug_elixir_test_wrong_ext
Previously:
Now:
All tests will be executed, the warning is printed, but it prints an error message and exits with 1.
Screenshot with this fix:
