Support Pyodide 0.27 and bump to Python >=3.12 #57

agriyakhetarpal · 2025-03-14T07:08:13Z

Closes #47
Closes #45

rth · 2025-03-14T07:30:02Z

Thanks @agriyakhetarpal for taking over the maintenance!

agriyakhetarpal · 2025-03-14T12:22:17Z

Pyodide 0.26.4 and 0.27.3 fail with a LookupError claiming an unsupported cp437 encoding after being stripped. I've traced this in Pyodide to the fact that the stripped stdlib also excludes the various encoding implementations from python_stdlib/encodings/, and CPython raises this error when it can't find them (as zipfile tries to decode with either a specified encoding or cp437 – and the latter is undefined).

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I have to note that 0.25.1 and 0.27.3 fail with different errors, so it is also likely that changes in the order of the imports, etc., are causing things to fail. Importing encodings before zip file will not help here, as the encoding itself is lost. We'll need a way to avoid stripping out this specific encoding from this folder. I think the way to do this would be to dive deeper into what discovery.js is doing.

rth · 2025-03-14T13:21:37Z

Regarding encodings I don't remember the exact details, but indeed the overall idea was that they were taking space so utf8 + a few specific ones should be enough to start Python. But maybe indeed the list of minimal required encodings changed.

so it is also likely that changes in the order of the imports, etc., are causing things to fail
I think the way to do this would be to dive deeper into what discovery.js is doing.

Yeah sorry part of it might also be relying on internal emscripten APIs so if those parts (in discovery.js) changed that would explain the breakage.

agriyakhetarpal · 2025-03-14T17:08:50Z

Thanks! I fixed the LookupError, now one more error remains, so we're almost there!

There is no proper error message in the CI logs, but I am able to reproduce the failure locally, which does have a stack trace:

pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

Tap to expand full error message

PythonError: Traceback (most recent call last):
  File "/home/pyodide/pyodide_pack_loader.py", line 10, in setup
    await _module.API.loadDynlib(path, bool(is_shared))
pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

    at new_error (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:10009)
    at wasm://wasm/0268a37a:wasm-function[308]:0x16e384
    at wasm://wasm/0268a37a:wasm-function[507]:0x17855f
    at wasm://wasm/0268a37a:wasm-function[4586]:0x328642
    at wasm://wasm/0268a37a:wasm-function[1151]:0x1c44db
    at wasm://wasm/0268a37a:wasm-function[3622]:0x2ca38d
    at wasm://wasm/0268a37a:wasm-function[2184]:0x20c2d8
    at wasm://wasm/0268a37a:wasm-function[1159]:0x1c4bc2
    at wasm://wasm/0268a37a:wasm-function[1162]:0x1c4ed1
    at wasm://wasm/0268a37a:wasm-function[1163]:0x1c4f4f
    at wasm://wasm/0268a37a:wasm-function[3428]:0x2a0d61
    at wasm://wasm/0268a37a:wasm-function[3429]:0x2a72bf
    at wasm://wasm/0268a37a:wasm-function[1165]:0x1c508f
    at wasm://wasm/0268a37a:wasm-function[1160]:0x1c4cf8
    at wasm://wasm/0268a37a:wasm-function[494]:0x177bbf
    at callPyObjectKwargs (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:62939)
    at Module.callPyObjectMaybePromising (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:64121)
    at Immediate.wrapper (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:27499)
    at process.processImmediate (node:internal/timers:511:21) {
  type: 'JsException',
  __error_address: 15818000
}

Node.js v23.6.0
FAILED

============================================================================== FAILURES ==============================================================================
_______________________________________________________________________ test_all[scikit-learn] _______________________________________________________________________

example_dir = PosixPath('/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/scikit-learn')
tmp_path = PosixPath('/private/var/folders/b3/2bq1m1_50bs4c7305j8vxcqr0000gn/T/pytest-of-agriyakhetarpal/pytest-230/test_all_scikit_learn_0')

    @pytest.mark.parametrize("example_dir", gen_all_examples())
    def test_all(example_dir, tmp_path):
        stdout = StringIO()
        os.chdir(tmp_path)
        # Assumes there is a node_modules in the current directory
        (tmp_path / "node_modules").symlink_to(
            BASE_DIR / "node_modules", target_is_directory=True
        )
        with redirect_stdout(stdout):
>           cli.main(
                example_path=example_dir / "app.py",
                config_path=None,
                verbose=False,
                include_paths=None,
                write_debug_map=True,
            )

/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/test_examples.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/cli.py:248: in main
    runner.run()
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/runners/node.py:39: in run
    subprocess.run(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

I'm not sure how to debug that so far as it's quite cryptic, so any suggestions would be welcome in case you're familiar with it. :)

rth · 2025-03-14T17:25:56Z

I think I did encounter it the past, but I'm not 100% sure/remember the cause.

It's raised here but in this case the error is misleading, because we are not trying to load file_packager (legacy pre-wheel) files.

Probably something is not quite right with the sequence in which dynamic libraries are loaded (like if something changed in pyodide in that respect). If you don't find it, ask Hood :)

rth · 2025-03-14T17:34:21Z

Maybe related to pyodide/pyodide#4726, see @hoodmane's comments in the code there. It looks like _module.API.loadDynlib would be calling locateFile after we are expecting it should no longer happen.

I don't know maybe monkeypatching locateFile back to the original value would be enough to make it work?

agriyakhetarpal · 2025-03-14T22:38:26Z

In the meantime, I can confirm that pyodide/pyodide#4726 did not have an effect, as it first turned up in 0.26.0a5 and this test passes with Pyodide 0.26.4 locally (I'll confirm that with CI in a moment).

ryanking13 · 2025-03-15T04:07:05Z

pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

This generally happens when we try to locate the shared library from the file system but when it fails (if file is not there, or there was an error while loading the library, etc). As Roman said, emscripten will fallback to locateFile when it happens, but locateFile is not the way we want to load the library, hence the error. So patching the locateFile will not work, instead, I think you need to figure out why the library loading failed.

When building pyodide.asm.js, setting PYODIDE_DEBUG_JS=1 (pyodide flag) and -sDYLINK_DEBUG= (emscripten flag, but it is quite noisy) may help you understand what is happening when loading the libraries.

agriyakhetarpal · 2025-03-15T09:58:18Z

Thanks for the insights, @rth and @ryanking13! I've been able to isolate the issue – it's coming from the fact that OpenBLAS isn't being correctly bundled against Pyodide 0.27.3:

                                        Packing..                                        
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ No ┃ Package                        ┃ All files ┃ .so libs ┃    Size (MB) ┃ Reduction ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│  0 │ stdlib                         │ 554 → 186 │          │  2.29 → 0.55 │    76.1 % │
│  1 │ joblib-1.4.0-py3-none-any.whl  │   51 → 17 │    0 → 0 │  0.17 → 0.03 │    79.3 % │
│  2 │ numpy-2.0.2-cp312-cp312-pyodi… │ 338 → 142 │  19 → 13 │  3.05 → 2.49 │    18.4 % │
│  3 │ openblas-0.3.26.zip            │     1 → 0 │    1 → 0 │  1.91 → 0.00 │   100.0 % │
│  4 │ scikit_learn-1.6.1-cp312-cp31… │ 511 → 183 │  68 → 43 │  6.53 → 3.12 │    52.2 % │
│  5 │ scipy-1.14.1-cp312-cp312-pyod… │ 730 → 446 │ 110 → 84 │ 13.31 → 7.73 │    41.9 % │
│  6 │ threadpoolctl-3.5.0-py3-none-… │     5 → 1 │    0 → 0 │  0.02 → 0.01 │    65.6 % │
└────┴────────────────────────────────┴───────────┴──────────┴──────────────┴───────────┘

As it can be seen above, libopenblas.so isn't being bundled properly. It's reduced by 100% and lost after modification. This does not happen with Pyodide 0.26.4, which keeps OpenBLAS intact as it was being loaded globally.

Later, this shows up here: [DEBUG] Loading dynamic library: /usr/lib/libopenblas.so, global: false, handle: undefined. However, for Pyodide 0.26.4, we have [DEBUG] Loading dynamic library: /usr/lib/libopenblas.so, global: true, handle: undefined – i.e., global: true instead of false.

This manifests later when we try to import extension modules in SciPy that rely on OpenBLAS, such as scipy.integrate._dop (https://github.com/pyodide/pyodide/blob/8bfd7bfd6a1b33dbf00b4fcf160b78c7307819e4/packages/scipy/patches/0010-Link-openblas-with-modules-that-require-f2c.patch#L18-L30).

Here is a brief summary from my extended logging:

Pyodide 0.26.4

[DEBUG-LOADER] Successfully loaded /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so

and

[DEBUG] Library /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so has 1 accessed symbols

and later

[DEBUG] Symbol accessed: /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so.PyInit__dop

Pyodide 0.27.3

[DEBUG-LOADER] Error loading /lib/python3.12/site-packages/scipy/integrate/_dop.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files!

and in this case, the error message is indeed not helpful; it hides the traceback. To be more specific, the actual error is masked a few layers underneath by this error:

Tap to show error message

Fatal error: PythonError: Traceback (most recent call last):
  File "/lib/python312.zip/_pyodide/_base.py", line 141, in eval_code_async
    return await CodeRunner(source, return_mode=return_mode, quiet_trailing_semicolon=quiet_trailing_semicolon, filename=filename, flags=flags, dont_inherit=dont_inherit, optimize=optimize).compile().run_async(globals, locals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/_pyodide/_base.py", line 130, in run_async
    coroutine = eval(self.code, globals, locals)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<exec>", line 5, in <module>
  File "/lib/python3.12/site-packages/sklearn/__init__.py", line 11, in <module>
    from .base import clone
  File "/lib/python3.12/site-packages/sklearn/base.py", line 12, in <module>
    from .utils._estimator_html_repr import _HTMLDocumentationLinkMixin, estimator_html_repr
  File "/lib/python3.12/site-packages/sklearn/utils/__init__.py", line 8, in <module>
    from ._chunking import gen_batches, gen_even_slices
  File "/lib/python3.12/site-packages/sklearn/utils/_chunking.py", line 6, in <module>
    from ._param_validation import Interval, validate_params
  File "/lib/python3.12/site-packages/sklearn/utils/_param_validation.py", line 12, in <module>
    from .validation import _is_arraylike_not_scalar
  File "/lib/python3.12/site-packages/sklearn/utils/validation.py", line 14, in <module>
    from ..utils._array_api import _asarray_with_order, _is_numpy_namespace, get_namespace
  File "/lib/python3.12/site-packages/sklearn/utils/_array_api.py", line 10, in <module>
    from .fixes import parse_version
  File "/lib/python3.12/site-packages/sklearn/utils/fixes.py", line 7, in <module>
    import scipy.stats
  File "/lib/python3.12/site-packages/scipy/stats/__init__.py", line 2, in <module>
    from ._stats_py import *
  File "/lib/python3.12/site-packages/scipy/stats/_stats_py.py", line 13, in <module>
    from . import distributions
  File "/lib/python3.12/site-packages/scipy/stats/distributions.py", line 1, in <module>
    from ._distn_infrastructure import rv_discrete, rv_continuous, rv_frozen
  File "/lib/python3.12/site-packages/scipy/stats/_distn_infrastructure.py", line 13, in <module>
    from scipy import integrate
  File "/lib/python3.12/site-packages/scipy/__init__.py", line 34, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/__init__.py", line 50, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.12/site-packages/scipy/integrate/__init__.py", line 4, in <module>
    from ._ode import *
  File "/lib/python3.12/site-packages/scipy/integrate/_ode.py", line 6, in <module>
    from . import _dop
ImportError: dynamic module does not define module export function (PyInit__dop)

    at new_error (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:10009)
    at wasm://wasm/0268a37a:wasm-function[308]:0x16e384
    at wasm://wasm/0268a37a:wasm-function[507]:0x17855f
    at wasm://wasm/0268a37a:wasm-function[4586]:0x328642
    at wasm://wasm/0268a37a:wasm-function[1151]:0x1c44db
    at wasm://wasm/0268a37a:wasm-function[3622]:0x2ca38d
    at wasm://wasm/0268a37a:wasm-function[2184]:0x20c2d8
    at wasm://wasm/0268a37a:wasm-function[1159]:0x1c4bc2
    at wasm://wasm/0268a37a:wasm-function[1162]:0x1c4ed1
    at wasm://wasm/0268a37a:wasm-function[1163]:0x1c4f4f
    at wasm://wasm/0268a37a:wasm-function[3428]:0x2a0d61
    at wasm://wasm/0268a37a:wasm-function[3429]:0x2a72bf
    at wasm://wasm/0268a37a:wasm-function[1165]:0x1c508f
    at wasm://wasm/0268a37a:wasm-function[1160]:0x1c4cf8
    at wasm://wasm/0268a37a:wasm-function[494]:0x177bbf
    at callPyObjectKwargs (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:62939)
    at Module.callPyObjectMaybePromising (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:64121)
    at Immediate.wrapper (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:27499)
    at process.processImmediate (node:internal/timers:511:21) {
  type: 'ImportError',
  __error_address: 60631024
}

I did not need to use debug builds of Pyodide to get these traces so far. I can put together more logging in a new PR after this one through a pyodide pack --verbose option, as it is currently unused.

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

agriyakhetarpal

Here is a self-review: I'm not fully sure whether my changes are correct, but I'm open to suggestions as this is the first time I'm going through our dynamic library loading procedure and trying to understand it better.

Either way, this is ready for review and the tests are passing, marking it as such!

.github/workflows/main.yml

agriyakhetarpal · 2025-03-15T14:33:15Z

examples/netcdf4/app.py

I added this test because we haven't had any other tests like it so far. Here, we have both a shared library (libhdf5) and a static library (libnetcdf), a package that depends on both of them (h5py), and eventually netCDF4 – i.e., we weren't testing the integration with any static libraries, just shared ones (SciPy + OpenBLAS). I can drop it if you suggest I do so.

pyodide_pack/ast_rewrite.py

agriyakhetarpal · 2025-03-15T14:41:43Z

pyodide_pack/loader/pyodide_pack_loader.py

I'd like to ask if it would be okay if I could address missing coverage in a further PR? The change doesn't do much; it just fails more gracefully by printing a warning when loading a dynlib. What I understand here that this loader is an elementary check and that we don't load the libs in the order they are supposed to be loaded in – we just run over what comes first in the list. It could be the case that I might have made a mistake somewhere.

With this, the netcdf4 test does display this locally:

Warning: Failed to load /lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files! Warning: Failed to load /lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files!

but at the same time, the code in examples/netcdf4/app.py executes and runs completely fine till the end, which I don't understand.

agriyakhetarpal · 2025-03-15T14:42:53Z

pyodide_pack/tests/test_ast_rewrite.py

@@ -123,7 +123,7 @@ def test_strip_module_docstrings():
    )


-@settings(deadline=300)
+@settings(deadline=None)


I can't get this test to pass within 500ms :)

rth

Thanks a lot @agriyakhetarpal . Changes wise LGTM, but I haven't followed latest pyodide development.

rth · 2025-03-15T15:58:37Z

pyodide_pack/runtime_detection.py

+            # Always mark shared libraries as shared=True to ensure they're loaded
+            # globally, for libraries that need these global symbols
+            if extension == ".so":
+                dll.shared = True


Are you sure about this? Is far as I remember cython generated objects are also .so and they don't need to be managed as global symbol.

I think the idea was more to look whether the shared_library tag in packages https://github.com/pyodide/pyodide/blob/main/packages/openblas/meta.yaml#L6 and in which case load it.

But yeah, in any case this part is suboptimal so feel free to merge as is, and investigate this separately (if you are interested in this topic). Because updating to the latest Pyodide version is probably more important.

agriyakhetarpal · 2025-03-15T17:22:25Z

🤔 I tried the netcdf4 example I added with Pyodide 0.26.4 and it seems to pass without needing to ignore unloadable dynlibs, which makes me feel that there is something that I'm missing in the equation. I'll mark it as a draft for now till I investigate more about why it fails with 0.27.3.

Thanks for the review in the meantime!

agriyakhetarpal · 2025-03-15T19:19:51Z

Some more investigation for this particular example (netcdf4) reveals that:

the number of accessed modules is getting reduced with Pyodide 0.27.3 in comparison to 0.26.4; i.e., the latter accesses 5 modules instead of 15 for the former. Modules from numpy.fft and numpy.random do not show up in the bundle .sos list for 0.27.3.
the order of the accessed modules is incorrect – libhdf5_hl.so in h5py.libs/ shows up at the end of the list, rather than before other h5py-specific modules that rely on it. This disarrangement is likely what is causing file_packager errors when loading, as dependent dynlibs will fail to load until their parent dynlibs are loaded.

`bundle-so-list.txt` from Pyodide 0.26.4

Tap to expand

/lib/python3.12/site-packages/numpy/core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_internal.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,True
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,True
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False

`bundle-so-list.txt` with Pyodide 0.27.3

Tap to expand

/lib/python3.12/site-packages/numpy/_core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5_hl.so,False

Unfortunately, installing my debug build of Pyodide does not work here for getting extra logging, as I keep encountering function signature mismatches in the installation stage (with micropip) itself...

ryanking13 · 2025-03-18T12:57:38Z

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

agriyakhetarpal · 2025-03-19T00:26:33Z

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

Sounds good, I'll leave that for exploration in another PR!

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

Yes, I've adjusted for it. I haven't found a solution to how the netCDF4 test can be fixed. It's good that I added such a test, though, as the rest of the tests were passing and we would have merged with broken functionality. The netCDF4 test passes on 0.24.1, so the failure with 0.27.1 is indeed related.

Do we have a guarantee that the libraries in mypackage.libs/ will always be loaded first, and then the ones insidemypackage/ and its subfolders? If yes, I can tweak the order of how the libraries are added to the bundled .sos list and we should be good to go. Otherwise, I think it needs more investigation. I didn't find any changes we made upstream that might have changed the order of the library detection/import – just that the libraries are imported locally, which I addressed. Maybe @hoodmane would know about it.

ryanking13 · 2025-03-20T08:41:41Z

Do we have a guarantee that the libraries in mypackage.libs/ will always be loaded first, and then the ones insidemypackage/ and its subfolders?

Yes. After we land RPATH in emscripten, the order of the import should be:

LD_LIBRARY_PATH (which will be something like /usr/lib:/lib)
paths in the RPATH field.

we can adjust the order in RPATH, so I think we have options to choose the load order.

agriyakhetarpal · 2025-03-20T10:02:18Z

Sounds good! Though, I don't think we should wait for RPATH support to unblock this PR, as that could likely be after Pyodide 0.28 (which will then break compatibility with pyodide-pack again).

agriyakhetarpal added 7 commits March 14, 2025 12:36

Bump to Node 22

f15d0e9

Test against Pyodide 0.27.3

db199bb

Fix typo

3940244

Update CHANGELOG for pyodide#57

3ce8480

Note compatibility with Pyodide 0.27 and later

ccac8f9

Use Python 3.12 everywhere

eacee75

Downgrade to Pyodide 0.26.4

7c83198

agriyakhetarpal added 6 commits March 14, 2025 13:42

Drop check for number of paths

72a091f

Go back to Node 20

dc0cfc2

Fix RTD builds

9fe3724

Fix coverage path

4b88204

Drop deprecated ast.Str

8b6c1ce

Upgrade to py312 for pyupgrade

6e35b9f

agriyakhetarpal changed the title ~~Support Pyodide 0.27~~ Support Pyodide 0.27 and bump to Python >=3.12 Mar 14, 2025

agriyakhetarpal added 6 commits March 14, 2025 13:58

Use -n auto for parallel testing

9b29a0e

Don't depend on tomli for Python <3.11

b0e99f3

Update to Python 3.12 everywhere

5551c85

Bump up Hypothesis deadline

6481503

Bisect; go down further to Pyodide 0.25.1

e16d2fe

Bump up test suite verbosity

490c3f1

agriyakhetarpal added 4 commits March 14, 2025 22:08

Import encodings.cp437

8462777

Update runtime module elimination docs

4ec4bdf

Bump to Pyodide 0.27.3 again

af67dcc

Bump to latest Node.js LTS again

1cd3e48

Test scikit-learn example with Pyodide 0.26.4

886a1bb

agriyakhetarpal force-pushed the update/pyodide-0.27 branch from 93b917b to 886a1bb Compare March 14, 2025 22:39

agriyakhetarpal added 9 commits March 15, 2025 17:20

Drop Hypothesis deadline for stdlib processing

02f65fb

All shared libraries are locally loaded

af067e7

Drop explicit NumPy from scikit-learn example

87b5478

Fix DynamicLib loading test

5441b38

Bump to Pyodide 0.27.3

58f7016

Add a more complex test

8795efc

Handle duplicate shared libraries gracefully?

ed4bde3

Merge branch 'main' into update/pyodide-0.27

4b57e0f

Mark compatibility with Python 3.12 in README

3cc8c03

agriyakhetarpal commented Mar 15, 2025

View reviewed changes

agriyakhetarpal marked this pull request as ready for review March 15, 2025 14:45

agriyakhetarpal requested review from rth and ryanking13 March 15, 2025 14:45

rth approved these changes Mar 15, 2025

View reviewed changes

agriyakhetarpal marked this pull request as draft March 15, 2025 17:23

agriyakhetarpal added 3 commits March 16, 2025 01:08

Fix coverage directory

f70fa86

Revert graceful checking for dynlib duplication

7c6c3e2

Drop shared=True markers

31a3f52

agriyakhetarpal force-pushed the update/pyodide-0.27 branch from 4e3717d to 31a3f52 Compare March 15, 2025 19:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Pyodide 0.27 and bump to Python >=3.12 #57

Support Pyodide 0.27 and bump to Python >=3.12 #57

agriyakhetarpal commented Mar 14, 2025 •

edited

Loading

rth commented Mar 14, 2025

agriyakhetarpal commented Mar 14, 2025

rth commented Mar 14, 2025

agriyakhetarpal commented Mar 14, 2025

rth commented Mar 14, 2025

rth commented Mar 14, 2025

agriyakhetarpal commented Mar 14, 2025

ryanking13 commented Mar 15, 2025

agriyakhetarpal commented Mar 15, 2025

agriyakhetarpal left a comment

agriyakhetarpal Mar 15, 2025

agriyakhetarpal Mar 15, 2025

agriyakhetarpal Mar 15, 2025

rth left a comment

rth Mar 15, 2025 •

edited

Loading

agriyakhetarpal commented Mar 15, 2025

agriyakhetarpal commented Mar 15, 2025 •

edited

Loading

ryanking13 commented Mar 18, 2025

agriyakhetarpal commented Mar 19, 2025

ryanking13 commented Mar 20, 2025

agriyakhetarpal commented Mar 20, 2025

Support Pyodide 0.27 and bump to Python >=3.12 #57

Are you sure you want to change the base?

Support Pyodide 0.27 and bump to Python >=3.12 #57

Conversation

agriyakhetarpal commented Mar 14, 2025 • edited Loading

rth commented Mar 14, 2025

agriyakhetarpal commented Mar 14, 2025

rth commented Mar 14, 2025

agriyakhetarpal commented Mar 14, 2025

rth commented Mar 14, 2025

rth commented Mar 14, 2025

agriyakhetarpal commented Mar 14, 2025

ryanking13 commented Mar 15, 2025

agriyakhetarpal commented Mar 15, 2025

Pyodide 0.26.4

Pyodide 0.27.3

agriyakhetarpal left a comment

Choose a reason for hiding this comment

agriyakhetarpal Mar 15, 2025

Choose a reason for hiding this comment

agriyakhetarpal Mar 15, 2025

Choose a reason for hiding this comment

agriyakhetarpal Mar 15, 2025

Choose a reason for hiding this comment

rth left a comment

Choose a reason for hiding this comment

rth Mar 15, 2025 • edited Loading

Choose a reason for hiding this comment

agriyakhetarpal commented Mar 15, 2025

agriyakhetarpal commented Mar 15, 2025 • edited Loading

bundle-so-list.txt from Pyodide 0.26.4

bundle-so-list.txt with Pyodide 0.27.3

ryanking13 commented Mar 18, 2025

agriyakhetarpal commented Mar 19, 2025

ryanking13 commented Mar 20, 2025

agriyakhetarpal commented Mar 20, 2025

agriyakhetarpal commented Mar 14, 2025 •

edited

Loading

rth Mar 15, 2025 •

edited

Loading

agriyakhetarpal commented Mar 15, 2025 •

edited

Loading

`bundle-so-list.txt` from Pyodide 0.26.4

`bundle-so-list.txt` with Pyodide 0.27.3