Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

configure Ghidrathon using absolute path of Python interpreter used to install Jep #85

Merged
merged 53 commits into from
Jan 27, 2024

Conversation

mike-hunhoff
Copy link
Collaborator

@mike-hunhoff mike-hunhoff commented Jan 24, 2024

This PR narrows Ghidrathon's installation steps to:

  1. Run pip install jep==4.2
  2. Run python ghidrathon_configure.py <absolute_path_to_ghidra_install> using the Python interpreter that you'd like to configure for Ghidrathon (ghidrathon_configure.py)
  3. Download Ghidrathon extension ZIP from releases (see
    ghidra_11.0_PUBLIC_20240125_Ghidrathon.zip for beta example) and install to Ghidra

This PR borrows concepts from #50 but takes a different path to configuring Jep for the following reasons:

We cannot rely on per session environment variables, e.g. VIRTUAL_ENV, to be available when Ghidrathon's code is executed. For example, VIRTUAL_ENV works fine in Linux but is lost on Windows making it difficult to detect when a virtual environment is being used. This also complicates our ability to dynamically resolve where the correct Python and Jep are installed.

The Python sys module must be specially configured because Jep uses an embedded Python interpreter, not the interpreter that was used to install it. Specifically, we must set the following correctly:

  • sys.prefix
  • sys.base_prefix
  • sys.exec_prefix
  • sys.base_exec_prefix
  • sys.executable
  • sys._base_executable

The site module sets sys.base_/prefix and sys.base_/exec_prefix differently if a virtual environment is being used. Specifically, the site module checks if the parent directory of sys.executable contains a file named pyenv.cfg w/out relying on VIRTUAL_ENV being set. Therefore, if we manually set sys._base_/executable to the absolute path of the Python interpreter used to install Jep then the site module can be invoked manually to properly configure the sys module with no further intervention on our part, including Python virtual environments. For this to work, we must follow the correct ordering to avoid a chicken and egg problem:

  1. Configure jep.MainInterpreter to use the correct Jep native library path, add the Jep Python package path to sys.path, and disable auto importing the site module
  2. Create our first jep.SubInterpreter
  3. Manually set sys._base_/executable to the absolute path of the Python interpreter used to install Jep
  4. Manually import the site module and execute site.main()
  5. Extend sys.path to include Ghidrathon-specific Python paths

We may be able to avoid these steps using Py_SetProgramName before Py_Initialize is called but Jep does not currently support this. I've opened a new issue upstream to request this.

Until Jep enables more fine-grained control over the embedded Python interpreter's configuration I believe our least path to resistance requires that Ghidrathon know the absolute path of the Python interpreter used to install Jep.

TODO:

  • Runtime verify Jep version is as expected (should match what we use to build release)
  • Configure CI to run tests across Windows, Linux, macOS

closes #1, #3, #8, #27, #34, #49, #62, #78, #81
supersedes #4, #46, #50, #63

@mike-hunhoff
Copy link
Collaborator Author

mike-hunhoff commented Jan 26, 2024

We passing CI tests across:

  • OS: Windows, Linux, macOS
  • Python 3.8, 3.12
  • Ghidra 10.3.2, 11.0.0
  • Java 17
  • Jep 4.2.0

🥳

Also, we need to squash and merge this 🤣

@mike-hunhoff mike-hunhoff merged commit 32f3969 into main Jan 27, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Build suggestions
1 participant