PyInstaller executable does not find Qt WebEngine Process

[0xWOLAND]

I have a project that uses PyQt6, specifically its QtWebSockets server and I have built it with Pyinstaller with windowed mode activated. If I extract the executable from a zipped target file produced by Tufup, it works. However, if I try to let Tufup update an older version of the package and run the newly produced executable, PyQt throws:

The following paths were searched for Qt WebEngine Process:
  /Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess
  /.../my_app.app/Contents/MacOS/QtWebEngineProcess
but could not find it.
You may override the default search path by using QTWEBENGINEPROCESS_PATH environment variable.

To Reproduce
Steps to reproduce the behavior:

• Build tufup-example using QtWebSockets

Expected behavior
I expect the executable built by the auto-updater to work the same way as the target zip package that contains the PyQt app with the latest version.

System info (please complete the following information):

• OS: MacOS
• OS Version Sonoma 14.5
• Python version 3.9

Any help is much appreciated! Please let me know if I can provide any more details

[0xWOLAND]

I'm not sure if this is relevant

[dennisvang]

@0xWOLAND Thanks for posting.

At first glance, this looks like a configuration issue, perhaps related to the PyInstaller spec, or perhaps a macOS-specific issue.
I guess your link to pyinstaller/pyinstaller#2276 could indeed be relevant.

However, your statement

If I extract the executable from a zipped target file produced by Tufup, it works. However, if I try to let Tufup update an older version of the package and run the newly produced executable, PyQt throws [...]

Suggests there may be something else going on.

In essence, tufup is just a fancy file-mover. It does not do anything with the contents of the app bundle, other than copy them into the location you specify as app_install_dir. So, tufup itself does not know (nor care) that your app uses QtWebSockets.

The basic steps taken by the update client on macOS are as follows:

1. check for new update
2. download new update (if available), i.e. either a full archive or a patch
3. in case of patch: patch current archive to reconstruct new archive
4. extract new archive into temporary dir
   

   tufup/src/tufup/client.py

   Lines 316 to 318 in 4bb16ad

   	shutil.unpack_archive(
   	filename=self.new_archive_local_path, extract_dir=self.extract_dir
   	)

5. move extracted files from temporary dir to app_install_dir
   

   tufup/src/tufup/utils/platform_specific.py

   Line 259 in 4bb16ad

   shutil.copytree(src_dir, dst_dir, dirs_exist_ok=True)

6. restart the app
   

   tufup/src/tufup/utils/platform_specific.py

   Lines 265 to 266 in 4bb16ad

   	subprocess.Popen(sys.executable, shell=True) # nosec
   	sys.exit(0)

Perhaps something is going wrong during one of the last steps on macOS...

Some questions to get more insight:

• Which version of tufup are you using?
• Could you share the complete PyInstaller call, so we can see exactly which options are used, etc.?
• Are tufup patches enabled?
  

  tufup/src/tufup/client.py

  Line 143 in 4bb16ad

  patch: bool = True,

• Did you try manually starting the executable after the update was applied?
• Did you inspect the content of the app_install_dir and compare it with the content of the manually-extracted archive, i.e. the one that did work?
• Could you be a bit more specific in the "steps to reproduce"? A minimal example of the app code including instantiation of the QWebSocketServer would be helpful.

[dennisvang]

Another thing you could try:

• Manually remove the contents of the app_install_dir, then extract the content of your "working" tufup archive into that dir, and try to run the executable from there.

[0xWOLAND]

Here is my spec file:
main.txt. I built my package with patch = False because it took a very long time to build, maybe this is the source of the issue? I'm currently rebuilding with patch enabled. What does patch do?

Manually remove the contents of the app_install_dir, then extract the content of your "working" tufup archive into that dir, and try to run the executable from there.

If I do this, the executable works.

[0xWOLAND]

Update: doesn't work with patch enabled

[0xWOLAND]

If I run diff -r on the newly formed executable and the newest version in targets/repository, the packages are identical.

[0xWOLAND]

It seems like the file myapp.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess exists, not sure why the executable isn't checking here.

[dennisvang]

Here is my spec file:
main.txt. I built my package with patch = False because it took a very long time to build, maybe this is the source of the issue? I'm currently rebuilding with patch enabled. What does patch do?

@0xWOLAND: tufup supports two types of updates:

• full, i.e. patch=False: for every update, the complete new archive (.tar.gz) is downloaded
• patch, i.e. patch=True: the new archive is reconstructed by applying a binary patch to the current archive (which is already available on the client side)

Patch update files are typically much smaller, and therefore much quicker to download. The only drawback is that creation of the binary patch, on the repo side, takes a long time and a lot of memory, as explained in #105. However, you only need to do that once for every release. Patch application, on the client side, is (relatively) fast.

After patch application, the resulting archive must be byte-by-byte identical to the new archive. This is verified by checking the length and hash. So, in the end, as long as the patch procedure succeeds, it should not matter whether you do a patch update or a full update, because you end up with the exact same file bundle.

If you do see a difference in operation between full mode and patch mode, perhaps some error is being silenced inadvertently. Could you post the command line output of a failed update attempt on the client-side, with logging level set to DEBUG? (make sure to remove any sensitive info)

[dennisvang]

It seems like the file myapp.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess exists, not sure why the executable isn't checking here.

@0xWOLAND Just to be sure: Did you try setting QTWEBENGINEPROCESS_PATH as suggested in the original error message?

PyInstaller<6.0 used to have a specific runtime hook doing just that, but this has been removed in pyinstaller 6.0.

See pyinstaller/pyinstaller#7619 and the pyinstaller 6.0 changelog.

Also see understanding-pyinstaller-hooks and changing-runtime-behavior for more info.

In any case, pyinstaller has a long history of issues related to QtWebEngine as discussed e.g. here.

[dennisvang]

Another question: what is the exact pyinstaller command you are using? For example, do you use --windowed and --onefile together?

[dennisvang]

...

Manually remove the contents of the app_install_dir, then extract the content of your "working" tufup archive into that dir, and try to run the executable from there.

If I do this, the executable works.

Seeing that your pyinstaller bundle does appear to work when you do it manually, but then does not work after tufup copies it into place, suggests that something strange may be going on with the paths. For example, an absolute path issue, or something with relative paths that do not take into account the app being in a subfolder.

Tufup does not modify the files or directories inside your app bundle, but it does determine where the bundle itself is located.

Did you verify that the directory structure in the app_install_dir was exactly the same after the manual procedure above and after the automatic update?

[0xWOLAND]

...

Manually remove the contents of the app_install_dir, then extract the content of your "working" tufup archive into that dir, and try to run the executable from there.

If I do this, the executable works.

Seeing that your pyinstaller bundle does appear to work when you do it manually, but then does not work after tufup copies it into place, suggests that something strange may be going on with the paths. For example, an absolute path issue, or something with relative paths that do not take into account the app being in a subfolder.

Tufup does not modify the files or directories inside your app bundle, but it does determine where the bundle itself is located.

Did you verify that the directory structure in the app_install_dir was exactly the same after the manual procedure above and after the automatic update?

I ran diff -r on the two directories (newly built one and the newest version in target/, and the directory structures are identical

[dennisvang]

I ran diff -r on the two directories (newly built one and the newest version in target/, and the directory structures are identical

Not sure this is what I'm looking for. Let me clarify:

1. Try the automatic update:

• install your old app version
• let your app update using tufup
• run the updated app (manually) with logging level set to DEBUG, and verify that the error is raised
• post the command line output here (make sure to remove any sensitive info) (!)
• take note of the resulting directory structure in app_install_dir

2. Then do a "manual install":

• manually remove all content of app_install_dir
• manually extract the content of the new archive from your tufup targets dir into the app_install_dir
• make sure the resulting directory structure is an exact match to the previously noted one, including any nesting
• run the app and ensure the error is not raised

In any case, I cannot do much else without seeing a complete, minimal reproducible example. Note the emphasis on complete and minimal. ;-)

It also helps to start reproducing the issue from a completely clean slate.

[0xWOLAND]

Hey, I'm currently working on building a minimal reproducible example. I think I tried manually setting the QTWEBENGINEPROCES_PATH as

os.environ["QTWEBENGINEPROCESS_PATH"] = os.path.normpath(
    os.path.join(
        sys._MEIPASS,
        "PySide6",
        "Qt",
        "lib",
        "QtWebEngineCore.framework",
        "Helpers",
        "QtWebEngineProcess.app",
        "Contents",
        "MacOS",
        "QtWebEngineProcess",
    )
)
os.environ["QTWEBENGINE_RESOURCES_PATH"] = os.path.normpath(
    os.path.join(
        sys._MEIPASS,
        "..",
        "Resources",
        "PySide6",
        "Qt",
        "lib",
        "QtWebEngineCore.framework",
        "Resources",
    )
)
os.environ["QTWEBENGINE_LOCALES_PATH"] = os.path.normpath(
    os.path.join(
        sys._MEIPASS,
        "..",
        "Resources",
        "PySide6",
        "Qt",
        "lib",
        "QtWebEngineCore.framework",
        "Resources",
        "qtwebengine_locales",
    )
)

which seems to get rid of the issue. However, a new issue arises:

[87489:86787:0609/221319.040978:ERROR:child_process_launcher_helper_mac.cc(166)] Failed to compile sandbox policy: empty subpath pattern
[87489:86787:0609/221319.155533:ERROR:child_process_launcher_helper_mac.cc(166)] Failed to compile sandbox policy: empty subpath pattern

Do you have any thoughts on how to resolve this?

[dennisvang]

@0xWOLAND This again looks like something related to PyInstaller, QtWebEngine, and Chromium on macOS.

Perhaps you can find some clues in one of the following issues/prs:

• hooks: disable QtWebEngine sandboxing for Qt6 on macOS pyinstaller/pyinstaller#6903
• Qt 6: Make sure macOS sandboxing is reenabled qutebrowser/qutebrowser#7278
• sandboxing, macOS, and PKG_CONFIG_PATH ocaml/opam#3806

There's also an unanswered question on SO that may be related, although this is not very helpful.

Just to verify: In your original post you mentioned using PyQt6, but now I see you are using PySide6? Their APIs may be nearly identical, but it's an important detail.

[0xWOLAND]

Thanks for the quick reply! Here's the minimal example

[0xWOLAND]

I forgot to mention that the same behavior as earlier occurs, except this error takes place instead.

[dennisvang]

Thanks for the quick reply! Here's the minimal example

That's a good start, but minimal implies remove everything that is not required to reproduce the issue.

For example:

• do you need the whole React app, or could you reproduce the issue with just a plain HTML file?
• does the error occur if you leave out entire backend package? (e.g. visit example.com instead)
• and what if you leave out the DeskTopApp part?
• and what if you only import the QWebEngineView and instantiate it without calling load()?
• ...

and so on and so forth, until you have the absolute minimum necessary to reproduce the issue.

[0xWOLAND]

Looks like simply disabling sandbox (following the docs) works. All I had to do is os.environ["QTWEBENGINE_DISABLE_SANDBOX"] = "1". However, yet another bug has spawned 😭. It seems like the relative paths are messed when tufup moves the package to app_install_dir.

dyld[40500]: Library not loaded: @rpath/QtWebEngineCore
  Referenced from: 
 /Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess
  Reason: tried: 
'/Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/../../../../../../../../../../QtWebEngineCore' (no such file), 
'/Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/../../../../../../../../../../QtWebEngineCore' (no such file), 
'/usr/local/lib/QtWebEngineCore' (no such file),
'/usr/lib/QtWebEngineCore' (no such file, not in dyld cache)
dyld[40501]: Library not loaded: @rpath/QtWebEngineCore
  Referenced from: 
 /Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess
  Reason: tried: 
'/Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/../../../../../../../../../../QtWebEngineCore' (no such file), 
'/Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/../../../../../../../../../../QtWebEngineCore' (no such file),
'/usr/local/lib/QtWebEngineCore' (no such file), 
'/usr/lib/QtWebEngineCore' (no such file, not in dyld cache)

[dennisvang]

As mentioned above, tufup only does shutil.unpack_archive(...) followed by shutil.copytree(...).

It's not immediately clear to me how that would cause this relative path issue.

Can you locate QtWebEngineCore manually in the tree under app_install_dir, before and after the update?

[0xWOLAND]

The files that exist are:

./myapp Runtime.app/Contents/Resources/PySide6/Qt/lib/QtWebEngineCore.framework/Versions/A/QtWebEngineCore
./myapp Runtime.app/Contents/Resources/PySide6/Qt/lib/QtWebEngineCore.framework/Versions/Current/QtWebEngineCore
./myapp Runtime.app/Contents/Resources/PySide6/Qt/lib/QtWebEngineCore.framework/QtWebEngineCore
./myapp Runtime.app/Contents/Resources/QtWebEngineCore
./myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Versions/A/QtWebEngineCore
./myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Versions/Current/QtWebEngineCore
./myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/QtWebEngineCore
./myapp Runtime.app/Contents/Frameworks/QtWebEngineCore

But the path

'/Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/../../../../../../../../../../QtWebEngineCore' (no such file), '/Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/../../../../../../../../../../QtWebEngineCore' (no such file), '/usr/local/lib/QtWebEngineCore' 

doesn't exist if I check manually. The long path above is equivalent to /Users/USER/Applications/myapp/myapp Runtime.app/

Relevant: pyinstaller/pyinstaller#2276 (comment)

[dennisvang]

Sounds more and more like an exotic issue that is out-of-scope for tufup.

Anyway, I don't have access to a mac (other than github runners) so cannot reproduce.

Perhaps these can help:

• https://www.mikeash.com/pyblog/friday-qa-2009-11-06-linking-and-install-names.html
• https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html

[dennisvang]

So it's looking for a path relative to the location of QtWebEngineProcess (which you specified using QTWEBENGINEPROCESS_PATH)

./myapp Runtime.app/Contents/Frameworks/PySide6/Qt/lib/QtWebEngineCore.framework/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess

with all the ../../../ and so on that results in

./myapp Runtime.app/QtWebEngineCore

but the actual file is in

./myapp Runtime.app/Contents/Frameworks/QtWebEngineCore

The difference is Contents/Frameworks/, which is not explicitly part of your QTWEBENGINEPROCESS_PATH.

Not sure if that has anything to do with it, but it stands out.

What's the output of sys._MEIPASS?

[0xWOLAND]

What's the output of sys._MEIPASS?

/Users/USER/Applications/myapp/myapp Runtime.app/Contents/Frameworks

[0xWOLAND]

Looks like it was

tufup/src/tufup/utils/platform_specific.py

Line 259 in 4bb16ad

shutil.copytree(src_dir, dst_dir, dirs_exist_ok=True)

throws the problem because symlink aren't handled. It works if symlinks=True.

[0xWOLAND]

@dennisvang I am happy to close this issue

[dennisvang]

Looks like it was

tufup/src/tufup/utils/platform_specific.py

Line 259 in 4bb16ad

shutil.copytree(src_dir, dst_dir, dirs_exist_ok=True)

throws the problem because symlink aren't handled. It works if symlinks=True.

@0xWOLAND Good find! :)

[dennisvang]

@dennisvang I am happy to close this issue

@0xWOLAND It would probably be convenient to expose the symlinks argument in the tufup api.

I'll keep the issue open until that's resolved.

[dennisvang]

@0xWOLAND I've prepared #148 with the option to enable symlinks as follows:

client.download_and_apply_updates(..., symlinks=True)

Perhaps you could give that a try?

[dennisvang]

@0xWOLAND The symlinks option is now available in tufup 0.9.0 (via pypi)

[0xWOLAND]

@0xWOLAND I've prepared #148 with the option to enable symlinks as follows:

client.download_and_apply_updates(..., symlinks=True)

Perhaps you could give that a try?

Yep it works!

[dennisvang]

@0xWOLAND Thanks! That's good to know. :)