Autodoc fails to import docstrings from C extension modules, imports stub file instead

[rhpvorderman]

Describe the bug

['Traceback (most recent call last):\n', '  File "/home/rhpvorderman/PycharmProjects/python-isal/.tox/docs/lib/python3.11/site-packages/sphinx/ext/autodoc/importer.py", line 181, in import_module\n    spec.loader.exec_module(module)\n', '  File "<frozen importlib._bootstrap_external>", line 940, in exec_module\n', '  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed\n', '  File "/home/rhpvorderman/PycharmProjects/python-isal/.tox/docs/lib/python3.11/site-packages/isal/isal_zlib.pyi", line 46, in <module>\n    level: int = ISAL_DEFAULT_COMPRESSION,\n                 ^^^^^^^^^^^^^^^^^^^^^^^^\n', "NameError: name 'ISAL_DEFAULT_COMPRESSION' is not defined\n", '\nThe above exception was the direct cause of the following exception:\n\n', 'Traceback (most recent call last):\n', '  File "/home/rhpvorderman/PycharmProjects/python-isal/.tox/docs/lib/python3.11/site-packages/sphinx/ext/autodoc/importer.py", line 269, in import_object\n    module = import_module(modname, try_reload=True)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/rhpvorderman/PycharmProjects/python-isal/.tox/docs/lib/python3.11/site-packages/sphinx/ext/autodoc/importer.py", line 187, in import_module\n    raise ImportError(exc, traceback.format_exc()) from exc\n', 'ImportError: (NameError("name \'ISAL_DEFAULT_COMPRESSION\' is not defined"), \'Traceback (most recent call last):\\n  File "/home/rhpvorderman/PycharmProjects/python-isal/.tox/docs/lib/python3.11/site-packages/sphinx/ext/autodoc/importer.py", line 181, in import_module\\n    spec.loader.exec_module(module)\\n  File "<frozen importlib._bootstrap_external>", line 940, in exec_module\\n  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed\\n  File "/home/rhpvorderman/PycharmProjects/python-isal/.tox/docs/lib/python3.11/site-packages/isal/isal_zlib.pyi", line 46, in <module>\\n    level: int = ISAL_DEFAULT_COMPRESSION,\\n                 ^^^^^^^^^^^^^^^^^^^^^^^^\\nNameError: name \\\'ISAL_DEFAULT_COMPRESSION\\\' is not defined\\n\')\n'] [autodoc.import_object]

The problem is that it imports the pyi stub file instead. I checked the install directory and the extension module files are present for importing.

How to Reproduce

sudo apt install nasm  # nasm needs to be installed
brew install nasm # if on MacOS
git clone --recursive https://github.com/pycompression/python-isal.git
cd python-isal
virtualenv .venv
source .venv/bin/activate
pip install sphinx sphinx-argparse sphinx-rtd-theme
pip install .
mkdir -p docs/_static
sphinx-build -a -W -n --keep-going docs docs/_build

Environment Information

Platform:              linux; (Linux-6.1.0-31-amd64-x86_64-with-glibc2.36)
Python version:        3.11.2 (main, Nov 30 2024, 21:22:50) [GCC 12.2.0])
Python implementation: CPython
Sphinx version:        8.2.3
Docutils version:      0.21.2
Jinja2 version:        3.1.5
Pygments version:      2.19.1

Sphinx extensions

`extensions = ["sphinx.ext.autodoc", 'sphinxarg.ext']`

Additional context

The problem does not occur in 8.1.0

[AA-Turner]

Do you have a reproducible error that works on Windows?

A

[rhpvorderman]

Sure replace pip install . with pip install isal and that should work. The nasm installation is not needed in that case.

[AA-Turner]

The relevant change is #13253 (#7630). For your case, you would want to preference importing a native module over stubs? Does the rationale in the issue apply here re signatures?

sphinx/sphinx/ext/autodoc/importer.py

Lines 169 to 181 in 54ab557

	spec = find_spec(modname)
	if spec is None:
	msg = f'No module named {modname!r}'
	raise ModuleNotFoundError(msg, name=modname) # NoQA: TRY301
	spec, pyi_path = _find_type_stub_spec(spec, modname)
	if pyi_path is None:
	module = importlib.import_module(modname)
	else:
	if spec.loader is None:
	msg = 'missing loader'
	raise ImportError(msg, name=spec.name) # NoQA: TRY301
	sys.modules[modname] = module = module_from_spec(spec)
	spec.loader.exec_module(module)

[rhpvorderman]

While I would love to document in the stub file. This essentially means double work. Using help(mymodule.my_c_function) would not work anymore as the docstrings would not be present if I did it in the stub file.

But I can understand the sentiment from #7630 too.

[AA-Turner]

It would be useful to have a tool to automatically update stub files with docstrings from C, but I don't know if anything like this exists.

Does your current documentation render the function signatures with arguments?

[rhpvorderman]

Well, it does...

https://python-isal.readthedocs.io/en/stable/#module-isal.isal_zlib

But only because I documented the function signature myself (or in this case modified it from existing CPython code):
https://github.com/pycompression/python-isal/blob/5e320f523676c831cd3e99ae89d6e84e1fec86d4/src/isal/isal_zlibmodule.c#L256C1-L271C1

I rather have the function and the documentation in one place. The fact that function signatures cannot be inferred from C code is not really an argument here as the signature can be documented in the docstring. This is not essentially different from documenting it in a stub file.

This is also how CPython generates function signatures in their documentation.

[AA-Turner]

For the given crc32_combine function, the online documentation doesn't seem to render the signature (perhaps related to the missing final comma before / in crc32_combine($module, crc1, crc2, crc2_length /)).

This is also how CPython generates function signatures in their documentation.

CPython doesn't use autodoc, the signatures are explicitly documented.

I rather have the function and the documentation in one place. The fact that function signatures cannot be inferred from C code is not really an argument here as the signature can be documented in the docstring. This is not essentially different from documenting it in a stub file.

I appreciate not wanting to duplicate this. I had hoped to avoid a configuration value here, but I will think a little more.

[rhpvorderman]

Thanks for spotting the typo. The rest of the functions work though.

CPython doesn't use autodoc, the signatures are explicitly documented.

Oh yes that is true. Sorry, but at least it does work with the help function. That can't be said for stub files. So I am still convinced that documenting stuff in the C module is the best place. And it does work. Issue #7630 highlights an issue that does not exist. His problem could also have been solved by writing the docstrings in C.

EDIT: Sorry, I could have worded that a bit more nicely. It is just frustrating when documentation gets broken to fix an issue that you never had, and turns out to be fixable by something else than changing the code. I don't have a problem with also documenting in the stub files per se, but that having that stuff in two places is almost a guarantee for them being out of sync.