diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index 697e0fb69eed..5c7ec2240574 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -1214,6 +1214,13 @@ Miscellaneous stubs at the end of the run, but only if any missing modules were detected. + It is not recommended to use this option in `CI/CD + `_, as it will make your + dependencies less reproducible. Instead, you should require and + install type dependencies like you would any other (test) dependency, + such as by using a dependency section in your `pyproject.toml + `_. + .. note:: This is new in mypy 0.900. Previous mypy versions included a @@ -1234,6 +1241,10 @@ Miscellaneous stub packages were found, they are installed and then another run is performed. + It is not recommended to use ``--install-types --non-interactive`` + in `CI/CD `_; see the other + flag for more details. + .. option:: --junit-xml JUNIT_XML Causes mypy to generate a JUnit XML test result document with diff --git a/docs/source/error_code_list.rst b/docs/source/error_code_list.rst index 49cb8a0c06c1..b4c8442f855b 100644 --- a/docs/source/error_code_list.rst +++ b/docs/source/error_code_list.rst @@ -640,7 +640,8 @@ Check for an issue with imports [import] ---------------------------------------- Mypy generates an error if it can't resolve an `import` statement. -This is a parent error code of `import-not-found` and `import-untyped` +This is a parent error code of `import-not-found`, `import-untyped`, +and `import-untyped-stubs-available` See :ref:`ignore-missing-imports` for how to work around these errors. @@ -664,7 +665,7 @@ See :ref:`ignore-missing-imports` for how to work around these errors. .. _code-import-untyped: Check that import target can be found [import-untyped] --------------------------------------------------------- +------------------------------------------------------ Mypy generates an error if it can find the source code for an imported module, but that module does not provide type annotations (via :ref:`PEP 561 `). @@ -673,7 +674,7 @@ Example: .. code-block:: python - # Error: Library stubs not installed for "bs4" [import-untyped] + # Error: Library stubs not installed for "bs4" [import-untyped-stubs-available] import bs4 # Error: Skipping analyzing "no_py_typed": module is installed, but missing library stubs or py.typed marker [import-untyped] import no_py_typed @@ -681,6 +682,27 @@ Example: In some cases, these errors can be fixed by installing an appropriate stub package. See :ref:`ignore-missing-imports` for more details. +.. _code-import-untyped-stubs-available: + +Check that import target with known stubs can be found [import-untyped-stubs-available] +--------------------------------------------------------------------------------------- + +Like :ref:`code-import-untyped`, but used when mypy knows there is an appropriate +type stub package corresponding to the library, which you could install. + +Example: + +.. code-block:: python + + # Error: Library stubs not installed for "bs4" [import-untyped-stubs-available] + import bs4 + # Error: Skipping analyzing "no_py_typed": module is installed, but missing library stubs or py.typed marker [import-untyped] + import no_py_typed + +These errors can be fixed by installing the appropriate +stub package. See :ref:`ignore-missing-imports` for more details. + + .. _code-no-redef: Check that each name is defined once [no-redef] diff --git a/docs/source/running_mypy.rst b/docs/source/running_mypy.rst index 9f7461d24f72..2cc2983f309e 100644 --- a/docs/source/running_mypy.rst +++ b/docs/source/running_mypy.rst @@ -382,26 +382,29 @@ the library, you will get a message like this: main.py:1: note: (or run "mypy --install-types" to install all missing stub packages) You can resolve the issue by running the suggested pip commands. + If you're running mypy in CI, you can ensure the presence of any stub packages you need the same as you would any other test dependency, e.g. by adding them to -the appropriate ``requirements.txt`` file. +the appropriate ``requirements.txt`` file or dependency section of ``pyproject.toml``. -Alternatively, add the :option:`--install-types ` -to your mypy command to install all known missing stubs: +The :option:`--install-types ` flag +makes mypy list and (after a prompt) install all known missing stubs: .. code-block:: text mypy --install-types This is slower than explicitly installing stubs, since it effectively -runs mypy twice -- the first time to find the missing stubs, and +runs mypy twice — the first time to find the missing stubs, and the second time to type check your code properly after mypy has installed the stubs. It also can make controlling stub versions harder, -resulting in less reproducible type checking. +resulting in less reproducible type checking — it might even install +incompatible versions of your project's non-type dependencies, if the +type stubs require them! By default, :option:`--install-types ` shows a confirmation prompt. Use :option:`--non-interactive ` to install all suggested -stub packages without asking for confirmation *and* type check your code: +stub packages without asking for confirmation *and* then type check your code. If you've already installed the relevant third-party libraries in an environment other than the one mypy is running in, you can use :option:`--python-executable diff --git a/mypy/build.py b/mypy/build.py index 355ba861385e..874c06f87147 100644 --- a/mypy/build.py +++ b/mypy/build.py @@ -2780,11 +2780,10 @@ def module_not_found( msg, notes = reason.error_message_templates(daemon) if reason == ModuleNotFoundReason.NOT_FOUND: code = codes.IMPORT_NOT_FOUND - elif ( - reason == ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS - or reason == ModuleNotFoundReason.APPROVED_STUBS_NOT_INSTALLED - ): + elif reason == ModuleNotFoundReason.FOUND_WITHOUT_TYPE_HINTS: code = codes.IMPORT_UNTYPED + elif reason == ModuleNotFoundReason.APPROVED_STUBS_NOT_INSTALLED: + code = codes.IMPORT_UNTYPED_STUBS_AVAILABLE else: code = codes.IMPORT errors.report(line, 0, msg.format(module=target), code=code) diff --git a/mypy/errorcodes.py b/mypy/errorcodes.py index c22308e4a754..a3753b085097 100644 --- a/mypy/errorcodes.py +++ b/mypy/errorcodes.py @@ -34,6 +34,9 @@ def __init__( sub_code_map[sub_code_of.code].add(code) error_codes[code] = self + def is_import_related_code(self) -> bool: + return IMPORT in (self.code, self.sub_code_of) + def __str__(self) -> str: return f"" @@ -113,6 +116,12 @@ def __hash__(self) -> int: IMPORT_UNTYPED: Final = ErrorCode( "import-untyped", "Require that imported module has stubs", "General", sub_code_of=IMPORT ) +IMPORT_UNTYPED_STUBS_AVAILABLE: Final = ErrorCode( + "import-untyped-stubs-available", + "Require that imported module (with known stubs) has stubs", + "General", + sub_code_of=IMPORT, +) NO_REDEF: Final = ErrorCode("no-redef", "Check that each name is defined once", "General") FUNC_RETURNS_VALUE: Final = ErrorCode( "func-returns-value", "Check that called function returns a value in value context", "General" diff --git a/mypy/errors.py b/mypy/errors.py index 6aa19ed7c5a0..d5a6950fd4d5 100644 --- a/mypy/errors.py +++ b/mypy/errors.py @@ -10,7 +10,7 @@ from mypy import errorcodes as codes from mypy.error_formatter import ErrorFormatter -from mypy.errorcodes import IMPORT, IMPORT_NOT_FOUND, IMPORT_UNTYPED, ErrorCode, mypy_error_codes +from mypy.errorcodes import ErrorCode, mypy_error_codes from mypy.options import Options from mypy.scope import Scope from mypy.util import DEFAULT_SOURCE_OFFSET, is_typeshed_file @@ -530,7 +530,7 @@ def _add_error_info(self, file: str, info: ErrorInfo) -> None: self.error_info_map[file].append(info) if info.blocker: self.has_blockers.add(file) - if info.code in (IMPORT, IMPORT_UNTYPED, IMPORT_NOT_FOUND): + if info.code is not None and info.code.is_import_related_code(): self.seen_import_error = True def _filter_error(self, file: str, info: ErrorInfo) -> bool: @@ -576,7 +576,7 @@ def add_error_info(self, info: ErrorInfo) -> None: self.only_once_messages.add(info.message) if ( self.seen_import_error - and info.code not in (IMPORT, IMPORT_UNTYPED, IMPORT_NOT_FOUND) + and (info.code is None or (not info.code.is_import_related_code())) and self.has_many_errors() ): # Missing stubs can easily cause thousands of errors about diff --git a/mypy/report.py b/mypy/report.py index 39cd80ed38bf..f90ca3333732 100644 --- a/mypy/report.py +++ b/mypy/report.py @@ -26,7 +26,7 @@ from mypy.version import __version__ try: - from lxml import etree # type: ignore[import-untyped] + from lxml import etree LXML_INSTALLED = True except ImportError: diff --git a/mypy/test/testcheck.py b/mypy/test/testcheck.py index fb2eb3a75b9b..ca5d7c773fd4 100644 --- a/mypy/test/testcheck.py +++ b/mypy/test/testcheck.py @@ -5,6 +5,9 @@ import os import re import sys +from importlib.util import find_spec as import_exists + +import pytest from mypy import build from mypy.build import Graph @@ -24,14 +27,6 @@ ) from mypy.test.update_data import update_testcase_output -try: - import lxml # type: ignore[import-untyped] -except ImportError: - lxml = None - - -import pytest - # List of files that contain test case descriptions. # Includes all check-* files with the .test extension in the test-data/unit directory typecheck_files = find_test_files(pattern="check-*.test") @@ -55,7 +50,7 @@ class TypeCheckSuite(DataSuite): files = typecheck_files def run_case(self, testcase: DataDrivenTestCase) -> None: - if lxml is None and os.path.basename(testcase.file) == "check-reports.test": + if not import_exists("lxml") and os.path.basename(testcase.file) == "check-reports.test": pytest.skip("Cannot import lxml. Is it installed?") incremental = ( "incremental" in testcase.name.lower() diff --git a/mypy/test/testcmdline.py b/mypy/test/testcmdline.py index 11d229042978..17437c79665c 100644 --- a/mypy/test/testcmdline.py +++ b/mypy/test/testcmdline.py @@ -10,6 +10,9 @@ import re import subprocess import sys +from importlib.util import find_spec as import_exists + +import pytest from mypy.test.config import PREFIX, test_temp_dir from mypy.test.data import DataDrivenTestCase, DataSuite @@ -19,13 +22,6 @@ normalize_error_messages, ) -try: - import lxml # type: ignore[import-untyped] -except ImportError: - lxml = None - -import pytest - # Path to Python 3 interpreter python3_path = sys.executable @@ -38,7 +34,7 @@ class PythonCmdlineSuite(DataSuite): native_sep = True def run_case(self, testcase: DataDrivenTestCase) -> None: - if lxml is None and os.path.basename(testcase.file) == "reports.test": + if not import_exists("lxml") and os.path.basename(testcase.file) == "reports.test": pytest.skip("Cannot import lxml. Is it installed?") for step in [1] + sorted(testcase.output2): test_python_cmdline(testcase, step) diff --git a/mypy/test/testreports.py b/mypy/test/testreports.py index f638756ad819..7ea8cb1cc7fc 100644 --- a/mypy/test/testreports.py +++ b/mypy/test/testreports.py @@ -3,27 +3,23 @@ from __future__ import annotations import textwrap +from importlib.util import find_spec as import_exists + +import pytest from mypy.report import CoberturaPackage, get_line_rate from mypy.test.helpers import Suite, assert_equal -try: - import lxml # type: ignore[import-untyped] -except ImportError: - lxml = None - -import pytest - class CoberturaReportSuite(Suite): - @pytest.mark.skipif(lxml is None, reason="Cannot import lxml. Is it installed?") + @pytest.mark.skipif(not import_exists("lxml"), reason="Cannot import lxml. Is it installed?") def test_get_line_rate(self) -> None: assert_equal("1.0", get_line_rate(0, 0)) assert_equal("0.3333", get_line_rate(1, 3)) - @pytest.mark.skipif(lxml is None, reason="Cannot import lxml. Is it installed?") + @pytest.mark.skipif(not import_exists("lxml"), reason="Cannot import lxml. Is it installed?") def test_as_xml(self) -> None: - import lxml.etree as etree # type: ignore[import-untyped] + import lxml.etree as etree cobertura_package = CoberturaPackage("foobar") cobertura_package.covered_lines = 21 diff --git a/pyproject.toml b/pyproject.toml index 1870e0931407..8d3782d7f584 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -63,6 +63,10 @@ python2 = [] reports = ["lxml"] install-types = ["pip"] faster-cache = ["orjson"] +dev = [ + "pip-tools", # Not strictly needed for building per se, but pip-compile from this package is needed if you want to update the requirement files. (Word on the street is you can also use uv pip compile instead.) + "pip<24.3" # Needed to update the requirement files correctly with pip-tools ay ay ay https://github.com/jazzband/pip-tools/issues/2131 +] [project.urls] Homepage = "https://www.mypy-lang.org/" diff --git a/test-data/unit/check-errorcodes.test b/test-data/unit/check-errorcodes.test index d6e3366401dd..46fa3519f1e0 100644 --- a/test-data/unit/check-errorcodes.test +++ b/test-data/unit/check-errorcodes.test @@ -535,7 +535,10 @@ if int() is str(): # E: Non-overlapping identity check (left operand type: "int [builtins fixtures/primitives.pyi] [case testErrorCodeMissingModule] -from defusedxml import xyz # E: Library stubs not installed for "defusedxml" [import-untyped] \ +# Note: it was too difficult for me to figure out how to test [import-untyped] here, +# (ideally, it would!) +# but testNamespacePkgWStubs does test that, anyway. +from defusedxml import xyz # E: Library stubs not installed for "defusedxml" [import-untyped-stubs-available] \ # N: Hint: "python3 -m pip install types-defusedxml" \ # N: (or run "mypy --install-types" to install all missing stub packages) from nonexistent import foobar # E: Cannot find implementation or library stub for module named "nonexistent" [import-not-found] diff --git a/test-requirements.in b/test-requirements.in index 666dd9fc082c..0c2007828c8e 100644 --- a/test-requirements.in +++ b/test-requirements.in @@ -6,6 +6,7 @@ attrs>=18.0 filelock>=3.3.0 lxml>=5.3.0; python_version<'3.14' +lxml-stubs psutil>=4.0 pytest>=8.1.0 pytest-xdist>=1.34.0 diff --git a/test-requirements.txt b/test-requirements.txt index bcdf02319306..245c16fa4368 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -8,6 +8,8 @@ attrs==25.3.0 # via -r test-requirements.in cfgv==3.4.0 # via pre-commit +colorama==0.4.6 + # via pytest coverage==7.8.2 # via pytest-cov distlib==0.3.9 @@ -24,6 +26,8 @@ iniconfig==2.1.0 # via pytest lxml==5.4.0 ; python_version < "3.14" # via -r test-requirements.in +lxml-stubs==0.5.1 + # via -r test-requirements.in mypy-extensions==1.1.0 # via -r mypy-requirements.txt nodeenv==1.9.1