Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use installed JEP DLL rather than packaged DLL #50

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 0 additions & 31 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -22,37 +22,6 @@
// application.gradle.version property in <GHIDRA_INSTALL_DIR>/Ghidra/application.properties
// for the correction version of Gradle to use for the Ghidra installation you specify.

def javaHome
def pythonBin

if (project.hasProperty("PYTHON_BIN")) {
pythonBin = project.getProperty("PYTHON_BIN")
}
else {
pythonBin = "python"
}

// we need to install Jep; this requires C++ build tools on Windows (see README); we need to define
// the env variable "JAVA_HOME" containing absolute path to Java JDK configured for Ghidra
task installJep(type: Exec) {
environment "JAVA_HOME", System.getProperty("java.home")
commandLine pythonBin, '-m', 'pip', 'install', 'jep'
}

// we need to copy the Jep native binaries built in installJep to our extension directory; we use a small
// utility script written in Python
task copyJepNativeBinaries(type: Exec) {
dependsOn installJep
workingDir "${projectDir}"
commandLine pythonBin, "util${File.separator}configure_jep_native_binaries.py"
}

// make all tasks not matching copyJepNativeBinaries or installJep depend on copyJepNativeBinaries; mostly
// used to ensure our tasks run before Ghidra buildExtension task
tasks.matching { it.name != 'copyJepNativeBinaries' && it.name != 'installJep' }.all { Task task ->
task.dependsOn copyJepNativeBinaries
}

// from here we use the standard Gradle build provided by Ghidra framework

//----------------------START "DO NOT MODIFY" SECTION------------------------------
Expand Down
199 changes: 180 additions & 19 deletions src/main/java/ghidrathon/interpreter/GhidrathonInterpreter.java
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,34 @@
import ghidrathon.GhidrathonConfig;
import ghidrathon.GhidrathonScript;
import ghidrathon.GhidrathonUtils;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.IOException;
import java.lang.reflect.*;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.PathMatcher;
import java.util.List;
import java.util.Locale;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import jep.Jep;
import jep.JepConfig;
import jep.JepException;
import jep.MainInterpreter;
import jep.PyConfig;
import org.apache.commons.io.output.WriterOutputStream;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.LogManager;
williballenthin marked this conversation as resolved.
Show resolved Hide resolved


/** Utility class used to configure a Jep instance to access Ghidra */
public class GhidrathonInterpreter {
static final Logger log = LogManager.getLogger(GhidrathonInterpreter.class);

private Jep jep = null;
private GhidrathonConfig ghidrathonConfig = null;
Expand Down Expand Up @@ -118,7 +134,7 @@ public void write(byte[] b, int off, int len) throws IOException {
}

// we must set the native Jep library before creating a Jep instance
setJepNativeBinaryPath();
setJepPaths();

// create a new Jep interpreter instance
jep = new jep.SubInterpreter(jepConfig);
Expand All @@ -129,6 +145,146 @@ public void write(byte[] b, int off, int len) throws IOException {
setJepRunScript();
}

private PathMatcher getJepDllPathMatcher() throws Exception {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • docstring comments needed for all new routines

String os = System.getProperty("os.name", "generic").toLowerCase(Locale.ENGLISH);
if ((os.indexOf("mac") >= 0) || (os.indexOf("darwin") >= 0)) {
String arch = System.getProperty("os.arch");
if (arch == "amd64") {
// x86
return FileSystems.getDefault().getPathMatcher("glob:**libjep.so");
} else if (arch == "arm64") {
// arm m1
// TODO: just guessing this arch name arm64
return FileSystems.getDefault().getPathMatcher("glob:**libjep.jnilib");
}
Comment on lines +152 to +159
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • testing needed on m1 mac
  • testing needed on x86 mac

} else if (os.indexOf("win") >= 0) {
return FileSystems.getDefault().getPathMatcher("glob:**jep.dll");
} else if (os.indexOf("nux") >= 0) {
return FileSystems.getDefault().getPathMatcher("glob:**libjep.so");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • testing needed on linux

} else {
throw new Exception("OS not implemented: " + os);
}

throw new Exception("OS not implemented: " + os);
}

private Path searchJepDll(Path path) {
PathMatcher matcher;
try {
matcher = getJepDllPathMatcher();
} catch (Exception e) {
return null;
}

List<Path> dllPaths;
try (Stream<Path> walk = Files.walk(path)) {
dllPaths = walk
.filter(Files::isRegularFile)
.filter(x -> matcher.matches(x))
.collect(Collectors.toList());

} catch (IOException e) {
return null;
}

if (dllPaths.isEmpty()) {
return null;
}

if (dllPaths.size() > 1) {
// not sure which to pick
log.error("too many results in directory: " + path.toString());
return null;
}

return dllPaths.stream().findFirst().get();
}

// DANGER: DO NOT PASS DYNAMIC COMMANDS HERE!
private String execCmd(String ... commands) {
Runtime runtime = Runtime.getRuntime();
Process process = null;
try {
process = runtime.exec(commands);
} catch (IOException e) {
log.error("error: " + e.toString());
return "";
}

BufferedReader lineReader = new BufferedReader(new java.io.InputStreamReader(process.getInputStream()));
String output = String.join("\n", lineReader.lines().collect(Collectors.toList()));

BufferedReader errorReader = new BufferedReader(new java.io.InputStreamReader(process.getErrorStream()));
String error = String.join("\n", errorReader.lines().collect(Collectors.toList()));

if (error.length() > 0) {
log.error(error);
}

return output;
}

private Path findPythonPathJep() {
String var = "PYTHONPATH";
String env = System.getenv(var);
if (env != null) {
for (String envv : env.split(";")) {
Path path = java.nio.file.FileSystems.getDefault().getPath(envv);
Path location = searchJepDll(path);
if (location != null) {
// return first matching DLL
return location;
}
}
}
return null;
}

private Path findVirtualEnvJep() {
String var = "VIRTUAL_ENV";
String env = System.getenv(var);
if (env != null) {
Path path = java.nio.file.FileSystems.getDefault().getPath(env);
Path location = searchJepDll(path);
if (location != null) {
// return only matching DLL
return location;
}
}
return null;
}

private Path findSystemJep() {
String output = execCmd("python3", "-c", "import sys; import base64; print((b' '.join(map(lambda s: base64.b64encode(s.encode('utf-8')), sys.path))).decode('ascii'))");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we assuming python3 is available on all systems?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. can you give a scenario in which we expect Ghidrathon to work but python3 is not available? i can't immediately think of one, but i dont feel reliable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably don't need this with LibraryLocator, so lets not spend a lot of time answering this question unless that other strategy fails.


Charset UTF8_CHARSET = Charset.forName("UTF-8");

for (String base64 : output.split(" ")) {
byte[] bytes = java.util.Base64.getDecoder().decode(base64);
String s = new String(bytes, UTF8_CHARSET);

Path path1 = java.nio.file.FileSystems.getDefault().getPath(s);
Path location = searchJepDll(path1);

if (location != null) {
return location;
}
}

return null;
}

private void setJepDll(Path path) {
log.info("set JEP DLL: " + path.toString());
try {
MainInterpreter.setJepLibraryPath(path.toAbsolutePath().toString());
} catch (IllegalStateException e) {
// library path has already been set elsewhere,
// we expect this to happen as Jep Maininterpreter
// thread exists forever once it's created
}
}

/**
* Configure native Jep library.
*
Expand All @@ -139,28 +295,33 @@ public void write(byte[] b, int off, int len) throws IOException {
* @throws JepException
* @throws FileNotFoundException
*/
private void setJepNativeBinaryPath() throws JepException, FileNotFoundException {

File nativeJep;

try {

nativeJep = Application.getOSFile(GhidrathonUtils.THIS_EXTENSION_NAME, "libjep.so");

} catch (FileNotFoundException e) {

// whoops try Windows
nativeJep = Application.getOSFile(GhidrathonUtils.THIS_EXTENSION_NAME, "jep.dll");
private void setJepPaths() throws JepException, FileNotFoundException {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider using Jep's LibraryLocator? LibraryLocator should cover PYTHON_HOME and VIRTUAL_ENV lending less code for us to maintain. It won't, however, cover your third solution for resolving the shared Jep library.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i didn't consider it because i didn't notice it, thank you! i think i can replace most of this PR with a call to that routine 🥳

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dug into this some more and realized that MainInterpreter is coded to use the LibraryLocator if the Jep library path isn't manually set...https://github.com/ninia/jep/blob/056ce9907f5ecbf2364df1ec55755404b2e8a947/src/main/java/jep/MainInterpreter.java#L124-L135. Essentially, if we don't do anything then Jep attempts to find the Jep native library.

I tested this locally in a Linux environment and it worked great with the caveat that the LibraryLocator is naive in that it chooses the first Jep native library that it finds e.g. if you have two Python installs, both with Jep installed, then we don't have a choice (that I can find) as to which is selected. I don't think this is a major roadblock as it, so far, "just works" without additional code on our end and the multiple Python installs can be addressed by the user either by:

  • ensuring only 1 Python install also has Jep installed
  • using virtual environments

I'm going to test this on Windows next.

// if this is set, it take precedence over VIRTUAL_ENV.
Path pythonPathJep = findPythonPathJep();
if (pythonPathJep != null) {
log.info("found JEP dll via PYTHONPATH: " + pythonPathJep);
}

try {
// if this is set, it takes precedence over system python
Path virtualenvJep = findVirtualEnvJep();
if (virtualenvJep != null) {
log.info("found JEP dll via VIRTUAL_ENV: " + virtualenvJep);
}

MainInterpreter.setJepLibraryPath(nativeJep.getAbsolutePath());
// fall back to whatever python3 references
Path systemJep = findSystemJep();
if (systemJep != null) {
log.info("found JEP dll via python3: " + systemJep);
}
Comment on lines +300 to +315
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we collect all the paths first so that we can log out the status fully, then we actually set the config second.


} catch (IllegalStateException e) {
// library path has already been set elsewhere, we expect this to happen as Jep
// Maininterpreter
// thread exists forever once it's created
if (pythonPathJep != null) {
setJepDll(pythonPathJep);
} else if (virtualenvJep != null) {
setJepDll(virtualenvJep);
} else if (systemJep != null) {
setJepDll(systemJep);
} else {
log.error("unable to find jep");
}
}

Expand Down