What is the origin of the platlib/_preload_exname_extver.pth file?

[ndw]

In January, I used a cibuildwheel task on GitHub to build wheels for an external C library. A few days ago, I used it again to build wheels for an updated version of the library.

The version built a few days ago includes an additional file in the wheel:

saxoncpe-12.1.0.data/platlib/_preload_saxoncpe_12.1.0.pth

That file isn't present in the "12.0.0" version of the wheel and it appears to cause problems with virtual environments on Windows:

Error processing line 1 of C:\Users\Norman Walsh\saxon-test-1\lib\site-packages\_preload_saxonche_12.1.0.pth:

  Traceback (most recent call last):
    File "C:\Python39\lib\site.py", line 169, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<string>", line 18, in <module>
  FileNotFoundError: .load-order-saxonche-12.1.0 not found

Remainder of file ignored
SaxonC-HE 12.1 from Saxonica

I can't work out if this is caused by something that changed in the configuration of our build, something that changed in cibuildwheel or something else.

Hints, suggestions, workarounds, etc. most humbly solicited.

[joerick]

Your build script still has v2.11.1 pinned by the looks of it, so I'm guessing the change will be in the code you're building. I can't seem to access that, is it accessible somewhere?

[ndw]

We're not building from exclusively the open-source editions of the product, so I don't have all the sources in an easily accessible place, but here's a link to what the cibuildwheel tool builds from: https://dev.saxonica.com/_wheels/

(The github action downloads and unpacks that before it begins.)

I don't think we made any changes to the configuration, but I've been wrong before.

[henryiii]

That's an editable install artifact, I'm going to bet. Some backends use generated pth files. Is there an editable install happening somewhere, like for coverage or something? What build backend are you using?

[ndw]

I can certainly try to pin to more recent versions of setuptools and cython and see what happens. I don't recall precisely how we arrived at those numbers (though I have a vague recollection that it was intentional).

[henryiii]

Setuptools is easy - NumPy recommended capping to <60. It was a horrible recommendation, basically due to internal details and NumPy not planning on supporting setuptools anymore (and can currently be fixed with an envvar setting instead of a cap!), but that caused a ton of libraries to add caps, even if it was completely irrelevant to them.

Though I don't expect either of those to add the file - but don't see anything else that would add it - setuptools looks like it's the only thing running that would have the necessary info to add such a file with the name and version in it. (well, pip too). FYI, setuptools has native Cython support, you don't need cythonize.

Maybe try setting the build frontend to build instead of pip?

[ndw]

Can you point me to some documentation that explains what "setting the build frontend to build instead of pip" means? I am a little out of my depth here and that's probably not helping.

[henryiii]

Well, not important now, but https://cibuildwheel.readthedocs.io/en/stable/options/#build-frontend was what I was referring to.

[ndw]

Updating setuptools had no effect, but that's not a surprise given the dicoveries about delvewheel below.

[joerick]

I did a little sleuthing on that filename and it looks like it's a feature of delvewheel...

https://github.com/adang1345/delvewheel/blob/master/delvewheel/_wheel_repair.py#L816-L835

[ndw]

Nice sleuthing!

2023-03-20T15:26:05.4254532Z ##[group]Repairing wheel...
2023-03-20T15:26:05.4254870Z 
2023-03-20T15:26:05.4257123Z + delvewheel repair --add-path libs/win -w C:\Users\runneradmin\AppData\Local\Temp\cibw-run-fh7ny6cj\cp38-win_amd64\repaired_wheel C:\Users\runneradmin\AppData\Local\Temp\cibw-run-fh7ny6cj\cp38-win_amd64\built_wheel\saxoncee-12.1.0-cp38-cp38-win_amd64.whl
2023-03-20T15:26:10.4275582Z C:\Users\runneradmin\AppData\Local\Temp\cibw-run-fh7ny6cj\cp38-win_amd64\build\venv\lib\site-packages\delvewheel\_wheel_repair.py:819: UserWarning: 
2023-03-20T15:26:10.4276735Z repairing C:\Users\runneradmin\AppData\Local\Temp\cibw-run-fh7ny6cj\cp38-win_amd64\built_wheel\saxoncee-12.1.0-cp38-cp38-win_amd64.whl
2023-03-20T15:26:10.4277548Z The wheel contains a top-level extension module and has a delay-load
2023-03-20T15:26:10.4277919Z finding DLL dependencies
2023-03-20T15:26:10.4278442Z dependency, so we are injecting some code that runs at Python startup
2023-03-20T15:26:10.4279009Z copying DLLs into saxoncee-12.1.0.data\platlib
2023-03-20T15:26:10.4279514Z to preload the dependencies. End users who have Python processes that
2023-03-20T15:26:10.4279856Z mangling DLL names
2023-03-20T15:26:10.4280306Z were running before the wheel was installed should start new Python
2023-03-20T15:26:10.4280609Z calculating DLL load order
2023-03-20T15:26:10.4281055Z instances after installing the wheel.
2023-03-20T15:26:10.4281397Z updating saxoncee-12.1.0.dist-info\RECORD
2023-03-20T15:26:10.4281835Z   warnings.warn(textwrap.fill(
2023-03-20T15:26:10.4282134Z repackaging wheel
2023-03-20T15:26:10.4282752Z fixed wheel written to C:\Users\runneradmin\AppData\Local\Temp\cibw-run-fh7ny6cj\cp38-win_amd64\repaired_wheel\saxoncee-12.1.0-cp38-cp38-win_amd64.whl
2023-03-20T15:26:10.7709588Z ##[endgroup]

Now what the heck is a delay-load, why was it not true in 12.0, and how do I make it go away :-(

[henryiii]

Fantastic! Totally forgot about delvewheel running on this. :)

[henryiii]

Well, as an emergency workaround, you could pin an older version, since it was working before. Might be worth opening an issue over on delvewheel.

[henryiii]

The info on delvewheel is at https://github.com/adang1345/delvewheel. I see it discusses delay load a bit, but not how to disable it AFAICT.

[ndw]

I had a brief hope that maybe using a different version of GraalVM (22.3.1 instead of 22.3.0) was relevant, but no :-(

I don't see where I specify an particular version of delvewheel, is that under my control?

[henryiii]

Yes, you are pip installing it, just add ==<some_version>.

Edit: looks like you found that https://github.com/Saxonica/BuildWheels/blob/614c922637da20c04cdc578704a7a5947802f526/.github/workflows/wheels.yml#L43 :)

[ndw]

Yes. Thank you. blush Trying 1.2.0 now which I think was the current version last time our build succeeded.

[ndw]

Right. Going back to delvewheel 1.2.0 fixes the problem in as much as there are no .pth files. Whether there are other possible consequences, I'm not sure.

Thank you, everyone, for your help! I really appreciate it!