Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Pyodide 0.27 and bump to Python >=3.12 #57

Draft
wants to merge 36 commits into
base: main
Choose a base branch
from

Conversation

agriyakhetarpal
Copy link
Member

@agriyakhetarpal agriyakhetarpal commented Mar 14, 2025

Closes #47
Closes #45

@rth
Copy link
Member

rth commented Mar 14, 2025

Thanks @agriyakhetarpal for taking over the maintenance!

@agriyakhetarpal agriyakhetarpal changed the title Support Pyodide 0.27 Support Pyodide 0.27 and bump to Python >=3.12 Mar 14, 2025
@agriyakhetarpal
Copy link
Member Author

Pyodide 0.26.4 and 0.27.3 fail with a LookupError claiming an unsupported cp437 encoding after being stripped. I've traced this in Pyodide to the fact that the stripped stdlib also excludes the various encoding implementations from python_stdlib/encodings/, and CPython raises this error when it can't find them (as zipfile tries to decode with either a specified encoding or cp437 – and the latter is undefined).

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I have to note that 0.25.1 and 0.27.3 fail with different errors, so it is also likely that changes in the order of the imports, etc., are causing things to fail. Importing encodings before zip file will not help here, as the encoding itself is lost. We'll need a way to avoid stripping out this specific encoding from this folder. I think the way to do this would be to dive deeper into what discovery.js is doing.

@rth
Copy link
Member

rth commented Mar 14, 2025

Regarding encodings I don't remember the exact details, but indeed the overall idea was that they were taking space so utf8 + a few specific ones should be enough to start Python. But maybe indeed the list of minimal required encodings changed.

so it is also likely that changes in the order of the imports, etc., are causing things to fail
I think the way to do this would be to dive deeper into what discovery.js is doing.

Yeah sorry part of it might also be relying on internal emscripten APIs so if those parts (in discovery.js) changed that would explain the breakage.

@agriyakhetarpal
Copy link
Member Author

Thanks! I fixed the LookupError, now one more error remains, so we're almost there!

There is no proper error message in the CI logs, but I am able to reproduce the failure locally, which does have a stack trace:

pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

Tap to expand full error message
PythonError: Traceback (most recent call last):
  File "/home/pyodide/pyodide_pack_loader.py", line 10, in setup
    await _module.API.loadDynlib(path, bool(is_shared))
pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

    at new_error (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:10009)
    at wasm://wasm/0268a37a:wasm-function[308]:0x16e384
    at wasm://wasm/0268a37a:wasm-function[507]:0x17855f
    at wasm://wasm/0268a37a:wasm-function[4586]:0x328642
    at wasm://wasm/0268a37a:wasm-function[1151]:0x1c44db
    at wasm://wasm/0268a37a:wasm-function[3622]:0x2ca38d
    at wasm://wasm/0268a37a:wasm-function[2184]:0x20c2d8
    at wasm://wasm/0268a37a:wasm-function[1159]:0x1c4bc2
    at wasm://wasm/0268a37a:wasm-function[1162]:0x1c4ed1
    at wasm://wasm/0268a37a:wasm-function[1163]:0x1c4f4f
    at wasm://wasm/0268a37a:wasm-function[3428]:0x2a0d61
    at wasm://wasm/0268a37a:wasm-function[3429]:0x2a72bf
    at wasm://wasm/0268a37a:wasm-function[1165]:0x1c508f
    at wasm://wasm/0268a37a:wasm-function[1160]:0x1c4cf8
    at wasm://wasm/0268a37a:wasm-function[494]:0x177bbf
    at callPyObjectKwargs (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:62939)
    at Module.callPyObjectMaybePromising (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:64121)
    at Immediate.wrapper (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:27499)
    at process.processImmediate (node:internal/timers:511:21) {
  type: 'JsException',
  __error_address: 15818000
}

Node.js v23.6.0
FAILED

============================================================================== FAILURES ==============================================================================
_______________________________________________________________________ test_all[scikit-learn] _______________________________________________________________________

example_dir = PosixPath('/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/scikit-learn')
tmp_path = PosixPath('/private/var/folders/b3/2bq1m1_50bs4c7305j8vxcqr0000gn/T/pytest-of-agriyakhetarpal/pytest-230/test_all_scikit_learn_0')

    @pytest.mark.parametrize("example_dir", gen_all_examples())
    def test_all(example_dir, tmp_path):
        stdout = StringIO()
        os.chdir(tmp_path)
        # Assumes there is a node_modules in the current directory
        (tmp_path / "node_modules").symlink_to(
            BASE_DIR / "node_modules", target_is_directory=True
        )
        with redirect_stdout(stdout):
>           cli.main(
                example_path=example_dir / "app.py",
                config_path=None,
                verbose=False,
                include_paths=None,
                write_debug_map=True,
            )

/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/test_examples.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/cli.py:248: in main
    runner.run()
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/runners/node.py:39: in run
    subprocess.run(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

I'm not sure how to debug that so far as it's quite cryptic, so any suggestions would be welcome in case you're familiar with it. :)

@rth
Copy link
Member

rth commented Mar 14, 2025

I think I did encounter it the past, but I'm not 100% sure/remember the cause.

It's raised here but in this case the error is misleading, because we are not trying to load file_packager (legacy pre-wheel) files.

Probably something is not quite right with the sequence in which dynamic libraries are loaded (like if something changed in pyodide in that respect). If you don't find it, ask Hood :)

@rth
Copy link
Member

rth commented Mar 14, 2025

Maybe related to pyodide/pyodide#4726, see @hoodmane's comments in the code there. It looks like _module.API.loadDynlib would be calling locateFile after we are expecting it should no longer happen.

I don't know maybe monkeypatching locateFile back to the original value would be enough to make it work?

@agriyakhetarpal
Copy link
Member Author

agriyakhetarpal commented Mar 14, 2025

One mention of file_packager is here: https://github.com/pyodide/micropip/blob/71bc2e3a5eec9c842f466dd0b5944631feb3b96c/tests/test_install.py#L426-L450, and I see that pyodide/pyodide#2027 removed this format, but it's probably still used for shared libraries.

In scikit-learn's case, shared libraries that need to be imported would be OpenBLAS (due to SciPy). I thought that adding SciPy as an explicit dependency would help, but that's not the case, unfortunately – as OpenBLAS is already included.

The error message is here: https://github.com/pyodide/pyodide/blob/0f3e5f029275482882174b20770caf00290ec4f2/src/js/pyodide.ts#L284-L287


Edit: ah, thanks, let me take a look at your comments!

@rth
Copy link
Member

rth commented Mar 14, 2025

I see that pyodide/pyodide#2027 removed this format

Nah I think it was working in 2023 and that PR is from 2021. But this part does keep breaking with emscripten or pyodide updates, I last fixed in it #34

@agriyakhetarpal
Copy link
Member Author

In the meantime, I can confirm that pyodide/pyodide#4726 did not have an effect, as it first turned up in 0.26.0a5 and this test passes with Pyodide 0.26.4 locally (I'll confirm that with CI in a moment).

@ryanking13
Copy link
Member

pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

This generally happens when we try to locate the shared library from the file system but when it fails (if file is not there, or there was an error while loading the library, etc). As Roman said, emscripten will fallback to locateFile when it happens, but locateFile is not the way we want to load the library, hence the error. So patching the locateFile will not work, instead, I think you need to figure out why the library loading failed.

When building pyodide.asm.js, setting PYODIDE_DEBUG_JS=1 (pyodide flag) and -sDYLINK_DEBUG= (emscripten flag, but it is quite noisy) may help you understand what is happening when loading the libraries.

@agriyakhetarpal
Copy link
Member Author

Thanks for the insights, @rth and @ryanking13! I've been able to isolate the issue – it's coming from the fact that OpenBLAS isn't being correctly bundled against Pyodide 0.27.3:

                                        Packing..                                        
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ No ┃ Package                        ┃ All files ┃ .so libs ┃    Size (MB) ┃ Reduction ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│  0 │ stdlib                         │ 554 → 186 │          │  2.29 → 0.55 │    76.1 % │
│  1 │ joblib-1.4.0-py3-none-any.whl  │   51 → 17 │    0 → 0 │  0.17 → 0.03 │    79.3 % │
│  2 │ numpy-2.0.2-cp312-cp312-pyodi… │ 338 → 142 │  19 → 13 │  3.05 → 2.49 │    18.4 % │
│  3 │ openblas-0.3.26.zip            │     1 → 0 │    1 → 0 │  1.91 → 0.00 │   100.0 % │
│  4 │ scikit_learn-1.6.1-cp312-cp31… │ 511 → 183 │  68 → 43 │  6.53 → 3.12 │    52.2 % │
│  5 │ scipy-1.14.1-cp312-cp312-pyod… │ 730 → 446 │ 110 → 84 │ 13.31 → 7.73 │    41.9 % │
│  6 │ threadpoolctl-3.5.0-py3-none-… │     5 → 1 │    0 → 0 │  0.02 → 0.01 │    65.6 % │
└────┴────────────────────────────────┴───────────┴──────────┴──────────────┴───────────┘

As it can be seen above, libopenblas.so isn't being bundled properly. It's reduced by 100% and lost after modification. This does not happen with Pyodide 0.26.4, which keeps OpenBLAS intact as it was being loaded globally.

Later, this shows up here: [DEBUG] Loading dynamic library: /usr/lib/libopenblas.so, global: false, handle: undefined. However, for Pyodide 0.26.4, we have [DEBUG] Loading dynamic library: /usr/lib/libopenblas.so, global: true, handle: undefined – i.e., global: true instead of false.

This manifests later when we try to import extension modules in SciPy that rely on OpenBLAS, such as scipy.integrate._dop (https://github.com/pyodide/pyodide/blob/8bfd7bfd6a1b33dbf00b4fcf160b78c7307819e4/packages/scipy/patches/0010-Link-openblas-with-modules-that-require-f2c.patch#L18-L30).

Here is a brief summary from my extended logging:

Pyodide 0.26.4

[DEBUG-LOADER] Successfully loaded /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so

and

[DEBUG] Library /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so has 1 accessed symbols

and later

[DEBUG] Symbol accessed: /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so.PyInit__dop

Pyodide 0.27.3

[DEBUG-LOADER] Error loading /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files!

and in this case, the error message is indeed not helpful; it hides the traceback. To be more specific, the actual error is masked a few layers underneath by this error:

Tap to show error message
Fatal error: PythonError: Traceback (most recent call last):
  File "/lib/python312.zip/_pyodide/_base.py", line 141, in eval_code_async
    return await CodeRunner(source, return_mode=return_mode, quiet_trailing_semicolon=quiet_trailing_semicolon, filename=filename, flags=flags, dont_inherit=dont_inherit, optimize=optimize).compile().run_async(globals, locals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/_pyodide/_base.py", line 130, in run_async
    coroutine = eval(self.code, globals, locals)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<exec>", line 5, in <module>
  File "/lib/python3.12/site-packages/sklearn/__init__.py", line 11, in <module>
    from .base import clone
  File "/lib/python3.12/site-packages/sklearn/base.py", line 12, in <module>
    from .utils._estimator_html_repr import _HTMLDocumentationLinkMixin, estimator_html_repr
  File "/lib/python3.12/site-packages/sklearn/utils/__init__.py", line 8, in <module>
    from ._chunking import gen_batches, gen_even_slices
  File "/lib/python3.12/site-packages/sklearn/utils/_chunking.py", line 6, in <module>
    from ._param_validation import Interval, validate_params
  File "/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 12, in <module>
    from .validation import _is_arraylike_not_scalar
  File "/lib/python3.12/site-packages/sklearn/utils/validation.py", line 14, in <module>
    from ..utils._array_api import _asarray_with_order, _is_numpy_namespace, get_namespace
  File "/lib/python3.12/site-packages/sklearn/utils/_array_api.py", line 10, in <module>
    from .fixes import parse_version
  File "/lib/python3.12/site-packages/sklearn/utils/fixes.py", line 7, in <module>
    import scipy.stats
  File "/lib/python3.12/site-packages/scipy/stats/__init__.py", line 2, in <module>
    from ._stats_py import *
  File "/lib/python3.12/site-packages/scipy/stats/_stats_py.py", line 13, in <module>
    from . import distributions
  File "/lib/python3.12/site-packages/scipy/stats/distributions.py", line 1, in <module>
    from ._distn_infrastructure import rv_discrete, rv_continuous, rv_frozen
  File "/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py", line 13, in <module>
    from scipy import integrate
  File "/lib/python3.12/site-packages/scipy/__init__.py", line 34, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/__init__.py", line 50, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/scipy/integrate/__init__.py", line 4, in <module>
    from ._ode import *
  File "/lib/python3.12/site-packages/scipy/integrate/_ode.py", line 6, in <module>
    from . import _dop
ImportError: dynamic module does not define module export function (PyInit__dop)

    at new_error (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:10009)
    at wasm://wasm/0268a37a:wasm-function[308]:0x16e384
    at wasm://wasm/0268a37a:wasm-function[507]:0x17855f
    at wasm://wasm/0268a37a:wasm-function[4586]:0x328642
    at wasm://wasm/0268a37a:wasm-function[1151]:0x1c44db
    at wasm://wasm/0268a37a:wasm-function[3622]:0x2ca38d
    at wasm://wasm/0268a37a:wasm-function[2184]:0x20c2d8
    at wasm://wasm/0268a37a:wasm-function[1159]:0x1c4bc2
    at wasm://wasm/0268a37a:wasm-function[1162]:0x1c4ed1
    at wasm://wasm/0268a37a:wasm-function[1163]:0x1c4f4f
    at wasm://wasm/0268a37a:wasm-function[3428]:0x2a0d61
    at wasm://wasm/0268a37a:wasm-function[3429]:0x2a72bf
    at wasm://wasm/0268a37a:wasm-function[1165]:0x1c508f
    at wasm://wasm/0268a37a:wasm-function[1160]:0x1c4cf8
    at wasm://wasm/0268a37a:wasm-function[494]:0x177bbf
    at callPyObjectKwargs (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:62939)
    at Module.callPyObjectMaybePromising (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:64121)
    at Immediate.wrapper (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:27499)
    at process.processImmediate (node:internal/timers:511:21) {
  type: 'ImportError',
  __error_address: 60631024
}

I did not need to use debug builds of Pyodide to get these traces so far. I can put together more logging in a new PR after this one through a pyodide pack --verbose option, as it is currently unused.

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Copy link
Member Author

@agriyakhetarpal agriyakhetarpal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a self-review: I'm not fully sure whether my changes are correct, but I'm open to suggestions as this is the first time I'm going through our dynamic library loading procedure and trying to understand it better.

Either way, this is ready for review and the tests are passing, marking it as such!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this test because we haven't had any other tests like it so far. Here, we have both a shared library (libhdf5) and a static library (libnetcdf), a package that depends on both of them (h5py), and eventually netCDF4 – i.e., we weren't testing the integration with any static libraries, just shared ones (SciPy + OpenBLAS). I can drop it if you suggest I do so.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to ask if it would be okay if I could address missing coverage in a further PR? The change doesn't do much; it just fails more gracefully by printing a warning when loading a dynlib. What I understand here that this loader is an elementary check and that we don't load the libs in the order they are supposed to be loaded in – we just run over what comes first in the list. It could be the case that I might have made a mistake somewhere.

With this, the netcdf4 test does display this locally:

Warning: Failed to load /lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files!
Warning: Failed to load /lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files!

but at the same time, the code in examples/netcdf4/app.py executes and runs completely fine till the end, which I don't understand.

@@ -123,7 +123,7 @@ def test_strip_module_docstrings():
)


@settings(deadline=300)
@settings(deadline=None)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't get this test to pass within 500ms :)

@agriyakhetarpal agriyakhetarpal marked this pull request as ready for review March 15, 2025 14:45
Copy link
Member

@rth rth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @agriyakhetarpal . Changes wise LGTM, but I haven't followed latest pyodide development.

# Always mark shared libraries as shared=True to ensure they're loaded
# globally, for libraries that need these global symbols
if extension == ".so":
dll.shared = True
Copy link
Member

@rth rth Mar 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure about this? Is far as I remember cython generated objects are also .so and they don't need to be managed as global symbol.

I think the idea was more to look whether the shared_library tag in packages https://github.com/pyodide/pyodide/blob/main/packages/openblas/meta.yaml#L6 and in which case load it.

But yeah, in any case this part is suboptimal so feel free to merge as is, and investigate this separately (if you are interested in this topic). Because updating to the latest Pyodide version is probably more important.

@agriyakhetarpal
Copy link
Member Author

🤔 I tried the netcdf4 example I added with Pyodide 0.26.4 and it seems to pass without needing to ignore unloadable dynlibs, which makes me feel that there is something that I'm missing in the equation. I'll mark it as a draft for now till I investigate more about why it fails with 0.27.3.

Thanks for the review in the meantime!

@agriyakhetarpal agriyakhetarpal marked this pull request as draft March 15, 2025 17:23
@agriyakhetarpal
Copy link
Member Author

agriyakhetarpal commented Mar 15, 2025

Some more investigation for this particular example (netcdf4) reveals that:

  1. the number of accessed modules is getting reduced with Pyodide 0.27.3 in comparison to 0.26.4; i.e., the latter accesses 5 modules instead of 15 for the former. Modules from numpy.fft and numpy.random do not show up in the bundle .sos list for 0.27.3.
  2. the order of the accessed modules is incorrect – libhdf5_hl.so in h5py.libs/ shows up at the end of the list, rather than before other h5py-specific modules that rely on it. This disarrangement is likely what is causing file_packager errors when loading, as dependent dynlibs will fail to load until their parent dynlibs are loaded.

bundle-so-list.txt from Pyodide 0.26.4

Tap to expand

/lib/python3.12/site-packages/numpy/core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_internal.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,True
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,True
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False

bundle-so-list.txt with Pyodide 0.27.3

Tap to expand

/lib/python3.12/site-packages/numpy/_core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5_hl.so,False


Unfortunately, installing my debug build of Pyodide does not work here for getting extra logging, as I keep encountering function signature mismatches in the installation stage (with micropip) itself...

@ryanking13
Copy link
Member

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

@agriyakhetarpal
Copy link
Member Author

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

Sounds good, I'll leave that for exploration in another PR!

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

Yes, I've adjusted for it. I haven't found a solution to how the netCDF4 test can be fixed. It's good that I added such a test, though, as the rest of the tests were passing and we would have merged with broken functionality. The netCDF4 test passes on 0.24.1, so the failure with 0.27.1 is indeed related.

Do we have a guarantee that the libraries in mypackage.libs/ will always be loaded first, and then the ones insidemypackage/ and its subfolders? If yes, I can tweak the order of how the libraries are added to the bundled .sos list and we should be good to go. Otherwise, I think it needs more investigation. I didn't find any changes we made upstream that might have changed the order of the library detection/import – just that the libraries are imported locally, which I addressed. Maybe @hoodmane would know about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test against Pyodide 0.27 Require Python >=3.12
3 participants