Suppressing all expected warnings from tests

[wjbenfold]

Running the Iris tests generates a number of warnings that are expected, e.g.

/iris/lib/iris/fileformats/_nc_load_rules/helpers.py:645: UserWarning: Ignoring netCDF variable 'time' invalid units 'wibble' warnings.warn(msg)

There are also others that are less clearly expected. The number of these errors makes it likely one might miss a new or unexpected warning (e.g. a numpy deprecation, as spotted in #4374)

In discussion with @trexfeathers yesterday, we considered that the expected warnings could be suppressed such that it's easier to spot important errors. We could additionally test to make sure that code caused the warnings it's meant to cause, and/or have unsuppressed warnings cause tests to fail so that we notice them.

Is this something that seems valuable enough to make an issue and start working towards implementing (initially picking off warnings as tests are revisited for other reasons, then pushing to get the remainder at the end)?

[rcomer]

My initial thought is just to make sure whatever is done also works as expected when running under pytest, since we have an aspiration to eventually adopt that (#3835). I have found that warnings will appear when running tests under pytest even when they have been suppressed using filterwarnings in the main package. Having said that, I've never tried suppressing them in the test module itself.

[vsherratt]

Just to clarify, pytest does respect filterwarnings as long as it's set in the test itself (ie not at the module level).

[vsherratt]

Related: pytest picks up on absolutely all warnings, in particular it includes those that python by default suppresses (DeprecationWarning and PendingDeprecationWarning). Which is good - otherwise we might never see them - but unfortunately this includes deprecations from 3rd party packages that we can't do anything about.

So should probably add something like this to pyproject.toml:

[tool.pytest.ini_options]
filterwarnings = [
    "ignore::DeprecationWarning",
    "default::DeprecationWarning:iris",
    "ignore::PendingDeprecationWarning",
    "default::PendingDeprecationWarning:iris",
]

[pp-mo]

If you didn't know.. I think the obvious method is to use the "catch_warnings" context manager
There is a bit of a nasty potential hole in that thought, as the context manager, and I think warnings controls in general, are not thread-safe. So I guess whether that is a problem may depend on the test framework we use (unittest/nose/pytest).

Sometime back, we added such a context manager to all our gallery/example tests, to ensure that the examples don't raise deprecation warnings.

It seems that Pytest can filter warnings, also at the module level, but I think we need to commit to Pytest as our standard framework before we can use this.
We really should "migrate to PyTest" !

[trexfeathers]

This is now an active issue: #5466