ModuleNotFoundError: No module named '_multiprocessing' -- when `_multiprocessing` is built-in #8111

JonathonReinhart · 2023-11-17T17:01:36Z

Description of the issue

It appears that PyInstaller (perhaps rthooks/pyi_rth_multiprocessing.py) fails when _multiprocessing is built-in to Python, rather than a module.

Context information (for bug reports)

Output of pyinstaller --version: 6.2.0
Version of Python: Python 3.11.3+chromium.29 (from CIPD package infra/3pp/tools/cpython3/linux-amd64/3ywmFUOaOaFOgNvtlG9w1GSzYXssvcYo2UvqnTfZsNsC)
Platform: Linux

A minimal example program which shows the error

testapp.py

print("Hello world")
import multiprocessing.connection
print("import ok:")
print(f"multiprocessing.connection: {multiprocessing.connection}")
print(f"multiprocessing.connection._multiprocessing: {multiprocessing.connection._multiprocessing}")

When run directly using Python from CIPD (in a virtualenv):

$ python --version
Python 3.11.3+chromium.29

$ python testapp.py 
Hello world
import ok:
multiprocessing.connection: <module 'multiprocessing.connection' from '.../environment/cipd/packages/python311/lib/python3.11/multiprocessing/connection.py'>
multiprocessing.connection._multiprocessing: <module '_multiprocessing' (built-in)>

👉 Note that _multiprocessing is (built-in). 👈

When run directly using Python from Debian:

$ which python3
/usr/bin/python3

$ python3 --version
Python 3.11.5

$ python3 testapp.py 
Hello world
import ok:
multiprocessing.connection: <module 'multiprocessing.connection' from '/usr/lib/python3.11/multiprocessing/connection.py'>
multiprocessing.connection._multiprocessing: <module '_multiprocessing' from '/usr/lib/python3.11/lib-dynload/_multiprocessing.cpython-311-x86_64-linux-gnu.so'>

👉 Note that _multiprocessing is a .so module. 👈

Stacktrace / full error message

Using Python from CIPD:

$ pyinstaller --onedir testapp.py
140 INFO: PyInstaller: 6.2.0
140 INFO: Python: 3.11.3+chromium.29
143 INFO: Platform: Linux-6.5.6-1rodete4-amd64-x86_64-with-glibc2.37
143 INFO: wrote .../pyi-test/testapp.spec
146 INFO: Extending PYTHONPATH with paths
['.../pyi-test']
255 INFO: checking Analysis
258 INFO: checking PYZ
314 INFO: checking PKG
315 INFO: Bootloader .../environment/pigweed-venv/lib/python3.11/site-packages/PyInstaller/bootloader/Linux-64bit-intel/run
315 INFO: checking EXE
315 INFO: checking COLLECT
315 INFO: Building COLLECT COLLECT-00.toc
327 INFO: Building COLLECT COLLECT-00.toc completed successfully.

$ ./dist/testapp/testapp 
Traceback (most recent call last):
  File "PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py", line 107, in <module>
  File "PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py", line 95, in _pyi_rthook
  File "PyInstaller/loader/pyimod02_importers.py", line 419, in exec_module
  File "multiprocessing/popen_forkserver.py", line 7, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 419, in exec_module
  File "multiprocessing/forkserver.py", line 11, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 419, in exec_module
  File "multiprocessing/connection.py", line 21, in <module>
ModuleNotFoundError: No module named '_multiprocessing'
[382207] Failed to execute script 'pyi_rth_multiprocessing' due to unhandled exception!

Using Python from Debian:

$ pyinstaller --onedir testapp.py 
180 INFO: PyInstaller: 6.2.0
180 INFO: Python: 3.11.5
182 INFO: Platform: Linux-6.5.6-1rodete4-amd64-x86_64-with-glibc2.37
182 INFO: wrote .../pyi-test/testapp.spec
184 INFO: Extending PYTHONPATH with paths
['.../pyi-test']
797 INFO: checking Analysis
800 INFO: Building because hookspath changed
800 INFO: Initializing module dependency graph...
801 INFO: Caching module graph hooks...
808 INFO: Analyzing base_library.zip ...
1425 INFO: Loading module hook 'hook-encodings.py' from '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks'...
1883 INFO: Loading module hook 'hook-heapq.py' from '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks'...
3181 INFO: Loading module hook 'hook-pickle.py' from '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks'...
4492 INFO: Caching module dependency graph...
4565 INFO: Running Analysis Analysis-00.toc
4565 INFO: Looking for Python shared library...
4614 INFO: Using Python shared library: /lib/x86_64-linux-gnu/libpython3.11.so.1.0
4614 INFO: Analyzing .../pyi-test/testapp.py
4646 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks'...
5071 INFO: Loading module hook 'hook-xml.py' from '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks'...
5316 INFO: Processing module hooks...
5548 INFO: Looking for ctypes DLLs
5560 INFO: Analyzing run-time hooks ...
5562 INFO: Including run-time hook '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
5563 INFO: Including run-time hook '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
5565 INFO: Processing pre-find module path hook _pyi_rth_utils from '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks/pre_find_module_path/hook-_pyi_rth_utils.py'.
5566 INFO: Loading module hook 'hook-_pyi_rth_utils.py' from '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks'...
5566 INFO: Including run-time hook '/usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
5572 INFO: Looking for dynamic libraries
5881 INFO: Warnings written to .../pyi-test/build/testapp/warn-testapp.txt
5893 INFO: Graph cross-reference written to .../pyi-test/build/testapp/xref-testapp.html
5902 INFO: checking PYZ
5903 INFO: Building because toc changed
5903 INFO: Building PYZ (ZlibArchive) .../pyi-test/build/testapp/PYZ-00.pyz
6156 INFO: Building PYZ (ZlibArchive) ../pyi-test/build/testapp/PYZ-00.pyz completed successfully.
6172 INFO: checking PKG
6173 INFO: Building because toc changed
6173 INFO: Building PKG (CArchive) testapp.pkg
6182 INFO: Building PKG (CArchive) testapp.pkg completed successfully.
6183 INFO: Bootloader /usr/local/google/home/jrreinhart/.local/lib/python3.11/site-packages/PyInstaller/bootloader/Linux-64bit-intel/run
6183 INFO: checking EXE
6183 INFO: Building because toc changed
6183 INFO: Building EXE from EXE-00.toc
6184 INFO: Copying bootloader EXE to .../pyi-test/build/testapp/testapp
6184 INFO: Appending PKG archive to custom ELF section in EXE
6193 INFO: Building EXE from EXE-00.toc completed successfully.
6194 INFO: checking COLLECT
6194 INFO: Building COLLECT COLLECT-00.toc
6215 INFO: Building COLLECT COLLECT-00.toc completed successfully.

$ ./dist/testapp/testapp 
Hello world
import ok:
multiprocessing.connection: <module 'multiprocessing.connection' from '.../pyi-test/dist/testapp/_internal/multiprocessing/connection.pyc'>
multiprocessing.connection._multiprocessing: <module '_multiprocessing' from '.../pyi-test/dist/testapp/_internal/lib-dynload/_multiprocessing.cpython-311-x86_64-linux-gnu.so'>

The text was updated successfully, but these errors were encountered:

rokm · 2023-11-17T17:56:37Z

Can you open build/testapp/Analysis-00.toc for the CIPD-python build, and check where the python shared library was collected from?

I suspect that the CIPD python doesn't actually have a shared library (and thus cannot work with PyInstaller), and instead we end up collecting shared library from the Debian python, and hence those supposed built-ins end up missing.

JonathonReinhart · 2023-11-17T18:10:48Z

Can you open build/testapp/Analysis-00.toc for the CIPD-python build, and check where the python shared library was collected from?

I suspect that the CIPD python doesn't actually have a shared library (and thus cannot work with PyInstaller), and instead we end up collecting shared library from the Debian python, and hence those supposed built-ins end up missing.

Bingo.

...
  ('xml.sax.xmlreader',
   '.../environment/cipd/packages/python311/lib/python3.11/xml/sax/xmlreader.py',
   'PYMODULE'),
  ('http.client',
   '.../environment/cipd/packages/python311/lib/python3.11/http/client.py',
   'PYMODULE')]
[('libpython3.11.so.1.0',
   '/lib/x86_64-linux-gnu/libpython3.11.so.1.0',
   'BINARY'),
  ('libz.so.1', '/lib/x86_64-linux-gnu/libz.so.1', 'BINARY'),
  ('libexpat.so.1', '/lib/x86_64-linux-gnu/libexpat.so.1', 'BINARY')],

$ ldd environment/cipd/packages/python311/bin/python3
        linux-vdso.so.1 (0x00007ffdfa9e9000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f32ec31f000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f32ec31a000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f32ec23b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f32ec236000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f32ec231000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f32ec04d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f32ec33e000)

Interestingly, my Debian python3 doesn't seem to use libpython3.11.so either:

$ ldd /usr/bin/python3
        linux-vdso.so.1 (0x00007ffdecffc000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f662846f000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6628450000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f6628425000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6628243000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6628568000)

Does this mean that PyInstaller has to search/hope that there is a libpython.so which corresponds to the Python instance currently executing PyInstaller? (And this heuristic got it wrong here? because the CIPD & Debian versions happened to be the same?)

bwoodsend · 2023-11-17T19:21:27Z

Interestingly, my Debian python3 doesn't seem to use libpython3.11.so either:

I think Debian does some weird thing where it creates the libpython.so but also statically links the bin/python3 executable so that it doesn't use the libpython.so. It gives some tiny speed boost at the expense of making bin/python3 several megabytes instead of a few hundred kilobytes.

Does this mean that PyInstaller has to search/hope that there is a libpython.so which corresponds to the Python instance currently executing PyInstaller? (And this heuristic got it wrong here? because the CIPD & Debian versions happened to be the same?)

Not sure if the matching versions had a hand in this but yes to the rest of it.

JonathonReinhart · 2023-11-17T20:15:51Z

Hmm, well I'm not sure how an issue like this can be fixed unless PyInstaller makes a substantial change where it re-executes Python (using its own front-end executable?!) using the "found" libpython.so to perform the analysis phase...

bwoodsend · 2023-11-17T21:09:16Z

Fixing this issue would just mean raising the libpython.so not found error. PyInstaller's bootloaders need that library to run.

rokm · 2023-11-17T22:22:00Z

Does this mean that PyInstaller has to search/hope that there is a libpython.so which corresponds to the Python instance currently executing PyInstaller? (And this heuristic got it wrong here? because the CIPD & Debian versions happened to be the same?)

Yes. We typically infer the location of libpython3.x.so.1.0 from the environment's python executable (sys.executable). But like you observed, some python interpreters are actually not using the shared library, so we need additional heuristics to find that library in standard library locations.

In this case, CIPD python did not have shared lib at all - normally, this would raise the "libpython.so not found" error. But because system-provided python was exact match version-wise, its shared lib was picked up instead. I don't think we can come up with a reasonable heuristic to counter that, nor I think it is really worth it, since CIPD python is not compatible with PyInstaller anyway due to lack of the shared lib.

bwoodsend · 2023-11-18T16:00:08Z

The CIPD version is built with --enable-py-version-override=3.11.3+chromium.29. We could skip the heuristic search if "chromium" in sys.version? It feels very woolly...

rokm · 2023-11-18T16:29:41Z

I suppose we could try detecting known problematic python builds based on sys.version, and raise an error immediately; i.e., something along the lines of "This version of Python is not supported by PyInstaller." instead of the more generic "libpython.so not found" error.

JonathonReinhart added the triage Please triage and relabel this issue label Nov 17, 2023

rokm removed the triage Please triage and relabel this issue label Nov 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ModuleNotFoundError: No module named '_multiprocessing' -- when `_multiprocessing` is built-in #8111

ModuleNotFoundError: No module named '_multiprocessing' -- when `_multiprocessing` is built-in #8111

JonathonReinhart commented Nov 17, 2023

rokm commented Nov 17, 2023

JonathonReinhart commented Nov 17, 2023 •

edited

bwoodsend commented Nov 17, 2023

JonathonReinhart commented Nov 17, 2023

bwoodsend commented Nov 17, 2023

rokm commented Nov 17, 2023 •

edited

bwoodsend commented Nov 18, 2023

rokm commented Nov 18, 2023 •

edited

ModuleNotFoundError: No module named '_multiprocessing' -- when _multiprocessing is built-in #8111

ModuleNotFoundError: No module named '_multiprocessing' -- when _multiprocessing is built-in #8111

Comments

JonathonReinhart commented Nov 17, 2023

Description of the issue

Context information (for bug reports)

A minimal example program which shows the error

Stacktrace / full error message

rokm commented Nov 17, 2023

JonathonReinhart commented Nov 17, 2023 • edited

bwoodsend commented Nov 17, 2023

JonathonReinhart commented Nov 17, 2023

bwoodsend commented Nov 17, 2023

rokm commented Nov 17, 2023 • edited

bwoodsend commented Nov 18, 2023

rokm commented Nov 18, 2023 • edited

ModuleNotFoundError: No module named '_multiprocessing' -- when `_multiprocessing` is built-in #8111

ModuleNotFoundError: No module named '_multiprocessing' -- when `_multiprocessing` is built-in #8111

JonathonReinhart commented Nov 17, 2023 •

edited

rokm commented Nov 17, 2023 •

edited

rokm commented Nov 18, 2023 •

edited