[GR-48384] Update GDB debugging support for Native Image with gdb-debughelpers.py.

docs/reference-manual/native-image/guides/debug-native-executables-with-gdb.md

@@ -564,4 +564,5 @@ void svm_dbg_print_locationInfo(graal_isolatethread_t* thread, size_t mem);
 
 ### Related Documentation
 
-* [Debug Info Feature](../DebugInfo.md)
+* [Debug Info Feature](../DebugInfo.md)
+* [Debug Native Executables with a Python Helper Script](debug-native-executables-with-python-helper.md)

docs/reference-manual/native-image/guides/debug-native-executables-with-python-helper.md

@@ -0,0 +1,166 @@
+---
+layout: ni-docs
+toc_group: how-to-guides
+link_title: Debug Native Executables with a Python Helper Script
+permalink: /reference-manual/native-image/guides/debug-native-image-process-with-python-helper-script/
+---
+# Debug Native Executables with a Python Helper Script
+
+Additionally to the [GDB debugging](debug-native-executables-with-gdb.md), you can debug a `native-image` process using a Python helper script, _gdb-debughelpers.py_.
+The [GDB Python API](https://sourceware.org/gdb/current/onlinedocs/gdb/Python.html) is used to provide a reasonably good experience for debugging native executables or shared libraries.
+It requires GDB with Python support.
+The debugging extension is tested against GDB 13.2 and supports the new debuginfo generation introduced in GraalVM for JDK 17 and later.
+
+> Note: The _gdb-debughelpers.py_ file does not work with versions older than version 13.2 of `gdb` or versions older than GraalVM for JDK 17.
+
+The Python script _gdb-debughelpers.py_ can be found in the _<GRAALVM\_HOME>/lib/svm/debug_ directory.
+If debuginfo generation is enabled (see [Build a Native Executable with Debug Information](debug-native-executables-with-gdb.md#build-a-native-executable-with-debug-information)), the script is copied to the build directory.
+The `native-image` tool adds the debugging section `.debug_gdb_scripts` to the debug info file, which causes GDB to automatically load _gdb-debughelpers.py_ from the current working directory.
+
+For [security reasons](https://sourceware.org/gdb/current/onlinedocs/gdb/Auto_002dloading-safe-path.html)
+the first time GDB encounters a native executable or shared library that requests a specific Python file to be loaded it will print a warning:
+
+>     warning: File "<CWD>/gdb-debughelpers.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
+>
+>     To enable execution of this file add
+>             add-auto-load-safe-path <CWD>/gdb-debughelpers.py
+>     line to your configuration file "<HOME>/.gdbinit".
+>     To completely disable this security protection add
+>             add-auto-load-safe-path /
+>     line to your configuration file "<HOME>/.gdbinit".
+>     For more information about this security protection see the
+>     "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
+>             info "(gdb)Auto-loading safe path"
+
+To solve this, either add the current working directory to _~/.gdbinit_ as follows:
+
+    echo "add-auto-load-safe-path <CWD>/gdb-debughelpers.py" >> ~/.gdbinit
+
+or pass the path as a command line argument to `gdb`:
+
+    gdb -iex "set auto-load safe-path <CWD>/gdb-debughelpers.py" <binary-name>
+
+Both enable GDB to auto-load _gdb-debughelpers.py_ from the current working directory.
+
+Auto-loading is the recommended way to provide the script to GDB.
+However, it is possible to manually load the script from GDB explicitly with:
+
+    (gdb) source gdb-debughelpers.py
+
+## Pretty Printing Support
+
+Loading _gdb-debughelpers.py_ registers a new pretty printer to GDB, which adds an extra level of convenience for debugging native executables or shared libraries.
+This pretty printer handles the printing of Java Objects, Arrays, Strings, and Enums for debugging native executables or shared libraries.
+If the Java application uses `@CStruct` and `@CPointer` annotations to access C data structures, the pretty printer will also try to print them as if they were Java data structures.
+If the C data structures cannot be printed by the pretty printer, printing is performed by GDB.
+
+The pretty printer also prints of the primitive value of a boxed primitive (instead of a Java Object).
+
+Whenever printing is done via the `p` alias of the `print` command the pretty printer intercepts that call to perform type casts to the respective runtime types of Java Objects.
+This also applies for auto-completion when using the `p` alias.
+This means that if the static type is different to the runtime type, the `print` command uses the static type, which leaves the user to discover the runtime type and typecast it.
+Additionally, the `p` alias understands Java field and array access and function calls for Java Objects.
+
+#### Limitations
+
+The `print` command still uses its default implementation, as there is no way to overwrite it, while still keeping the capability of the default `print` command.
+Overriding would cause printing non-Java Objects to not work properly.
+Therefore, only the `p` alias for the `print` command is overwritten by the pretty printer, such that the user can still make use of the default GDB `print` command.
+
+### Options to Control the Pretty Printer Behavior
+
+In addition to the enhanced `p` alias, _gdb-debughelpers.py_ introduces some GDB parameters to customize the behavior of the pretty printer.
+Parameters in GDB can be controlled with `set <param> <value>` and `show <param>` commands, and thus integrate with GDB's customization options.
+
+* #### svm-print on/off
+
+Use this command to enable/disable the pretty printer.
+This also resets the `print` command alias `p` to its default behavior.
+Alternatively pretty printing can be suppressed with the
+[`raw` printing option of GDB's `print` command](https://sourceware.org/gdb/current/onlinedocs/gdb/Output-Formats.html):
+
+    (gdb) show svm-print
+    The current value of 'svm-print' is "on".
+
+    (gdb) print str
+    $1 = "string"
+
+    (gdb) print/r str
+    $2 = (java.lang.String *) 0x7ffff689d2d0
+
+    (gdb) set svm-print off
+    1 printer disabled
+    1 of 2 printers enabled
+
+    (gdb) print str
+    $3 = (java.lang.String *) 0x7ffff689d2d0
+
+* #### svm-print-string-limit &lt;int&gt;
+
+Customizes the maximum length for pretty printing a Java String.
+The default value is `200`.
+Set to `-1` or `unlimited` for unlimited printing of a Java String.
+This does not change the limit for a C String, which can be controlled with GDB's `set print characters` command.
+
+* #### svm-print-element-limit &lt;int&gt;
+
+Customizes the maximum number of elements for pretty printing a Java Array, ArrayList, and HashMap.
+The default value is `10`.
+Set to `-1` or `unlimited` to print an unlimited number of elements.
+This does not change the limit for a C array, which can be controlled with GDB's `set print elements` command.
+However, GDB's parameter `print elements` is the upper bound for `svm-print-element-limit`.
+
+* #### svm-print-field-limit &lt;int&gt;
+
+Customizes the maximum number of elements for pretty printing fields of a Java Object.
+The default value is `50`.
+Set to `-1` or `unlimited` to print an unlimited number of fields.
+GDB's parameter `print elements` is the upper bound for `svm-print-field-limit`.
+
+* #### svm-print-depth-limit &lt;int&gt;
+
+Customizes the maximum depth of recursive pretty printing.
+The default value is `1`.
+The children of direct children are printed (a sane default to make contents of boxed values visible).
+Set to `-1` or `unlimited` to print unlimited depth.
+GDB's parameter `print max-depth` is the upper bound for `svm-print-depth-limit`.
+
+* #### svm-use-hlrep on/off
+
+Enables/disables pretty printing for higher level representations.
+It provides a more data-oriented view on some Java data structures with a known internal structure such as Lists or Maps.
+Currently supports ArrayList and HashMap.
+
+* #### svm-infer-generics &lt;int&gt;
+
+Customizes the number of elements taken into account to infer the generic type of higher level representations.
+The default value is `10`.
+Set to `0` to not infer generic types and `-1` or `unlimited` to infer the generic type of all elements.
+
+* #### svm-print-address absolute/on/off
+
+Enables/disables printing of addresses in addition to regular pretty printing.
+When `absolute` mode is used even compressed references are shown as absolute addresses.
+Printing addresses is disabled by default.
+
+* #### svm-print-static-fields on/off
+
+Enables/disables printing of static fields for a Java Object.
+Printing static fields is disabled by default.
+
+* #### svm-complete-static-variables on/off
+
+Enables/disables auto-completion of static field members for the enhanced `p` alias.
+Auto-completion of static fields is enabled by default.
+
+* #### svm-selfref-check on/off
+
+Enables/disables self-reference checks for data structures.
+The pretty printer detects a self-referential data structure and prevents further expansion to avoid endless recursion.
+Self-reference checks are enabled by default.
+For testing, this feature can be temporary disabled (usually you wouldn't want to do this).
+
+### Related Documentation
+
+* [Debug Info Feature](../DebugInfo.md)
+* [Debug Native Executables with GDB](debug-native-executables-with-gdb.md)

docs/reference-manual/native-image/guides/guides.md

@@ -25,6 +25,7 @@ Here you will learn how to:
 - [Containerize a Native Executable and Run in a Docker Container](containerise-native-executable-with-docker.md)
 - [Create a Heap Dump from a Native Executable](create-heap-dump-from-native-executable.md)
 - [Debug Native Executables with GDB](debug-native-executables-with-gdb.md)
+- [Debug Native Executables with a Python Helper Script](debug-native-executables-with-python-helper.md)
 - [Include Reachability Metadata Using the Native Image Gradle Plugin](include-reachability-metadata-gradle.md)
 - [Include Reachability Metadata Using the Native Image Maven Plugin](include-reachability-metadata-maven.md)
 - [Include Resources in a Native Executable](include-resources.md)

substratevm/CHANGELOG.md

@@ -2,6 +2,9 @@
 
 This changelog summarizes major changes to GraalVM Native Image.
 
+## GraalVM for JDK 24 (Internal Version 24.2.0)
+* (GR-48384) Added a GDB Python script (`gdb-debughelpers.py`) to improve the Native Image debugging experience.
+
 ## GraalVM for JDK 23 (Internal Version 24.1.0)
 * (GR-51520) The old class initialization strategy, which was deprecated in GraalVM for JDK 22, is removed. The option `StrictImageHeap` no longer has any effect.
 * (GR-51106) Fields that are accessed via a `VarHandle` or `MethodHandle` are no longer marked as "unsafe accessed" when the `VarHandle`/`MethodHandle` can be fully intrinsified.

substratevm/mx.substratevm/mx_substratevm.py

@@ -986,6 +986,125 @@ def build_debug_test(variant_name, image_name, extra_args):
         os.environ.update({'debuginfotest_isolates' : 'yes'})
         mx.run([os.environ.get('GDB_BIN', 'gdb'), '-ex', 'python "ISOLATES=True"', '-x', gdb_utils_py, '-x', testhello_py, hello_binary])
 
+
+def _gdbdebughelperstest(native_image, path, with_isolates_only, build_only, test_only, args):
+    test_proj = mx.dependency('com.oracle.svm.test')
+    test_source_path = test_proj.source_dirs()[0]
+    tutorial_proj = mx.dependency('com.oracle.svm.tutorial')
+    tutorial_c_source_dir = join(tutorial_proj.dir, 'native')
+    tutorial_source_path = tutorial_proj.source_dirs()[0]
+
+    gdbdebughelpers_py = join(mx.dependency('com.oracle.svm.hosted.image.debug').output_dir(), 'gdb-debughelpers.py')
+
+    test_python_source_dir = join(test_source_path, 'com', 'oracle', 'svm', 'test', 'debug', 'helper')
+    test_pretty_printer_py = join(test_python_source_dir, 'test_pretty_printer.py')
+    test_cinterface_py = join(test_python_source_dir, 'test_cinterface.py')
+    test_class_loader_py = join(test_python_source_dir, 'test_class_loader.py')
+    test_settings_py = join(test_python_source_dir, 'test_settings.py')
+    test_svm_util_py = join(test_python_source_dir, 'test_svm_util.py')
+
+    test_pretty_printer_args = [
+        '-cp', classpath('com.oracle.svm.test'),
+        # We do not want to step into class initializer, so initialize everything at build time.
+        '--initialize-at-build-time=com.oracle.svm.test.debug.helper',
+        'com.oracle.svm.test.debug.helper.PrettyPrinterTest'
+    ]
+    test_cinterface_args = [
+        '--shared',
+        '-Dcom.oracle.svm.tutorial.headerfile=' + join(tutorial_c_source_dir, 'mydata.h'),
+        '-cp', tutorial_proj.output_dir()
+    ]
+    test_class_loader_args = [
+        '-cp', classpath('com.oracle.svm.test'),
+        '-Dsvm.test.missing.classes=' + classpath('com.oracle.svm.test.missing.classes'),
+        '--initialize-at-build-time=com.oracle.svm.test.debug.helper',
+        # We need the static initializer of the ClassLoaderTest to run at image build time
+        '--initialize-at-build-time=com.oracle.svm.test.missing.classes',
+        'com.oracle.svm.test.debug.helper.ClassLoaderTest'
+    ]
+
+    gdb_args = [
+        os.environ.get('GDB_BIN', 'gdb'),
+        '--nx',
+        '-q',  # do not print the introductory and copyright messages
+        '-iex', 'set logging overwrite on',
+        '-iex', 'set logging redirect on',
+        '-iex', 'set logging enabled on',
+    ]
+
+    def run_debug_test(image_name: str, testfile: str, source_path: str, with_isolates: bool = True,
+                       build_cinterfacetutorial: bool = False, extra_args: list[str] = None, skip_build: bool = False):
+        extra_args = [] if extra_args is None else extra_args
+        build_dir = join(path, image_name + ("" if with_isolates else "_no_isolates"))
+
+        if not test_only and not skip_build:
+            # clean / create output directory
+            if exists(build_dir):
+                mx.rmtree(build_dir)
+            mx.ensure_dir_exists(build_dir)
+
+            build_args = args + [
+                '-H:CLibraryPath=' + source_path,
+                '--native-image-info',
+                '-Djdk.graal.LogFile=graal.log',
+                '-g', '-O0',
+            ] + svm_experimental_options([
+                '-H:+VerifyNamingConventions',
+                '-H:+SourceLevelDebug',
+                '-H:+IncludeDebugHelperMethods',
+                '-H:DebugInfoSourceSearchPath=' + source_path,
+            ]) + extra_args
+
+            if not with_isolates:
+                build_args += svm_experimental_options(['-H:-SpawnIsolates'])
+
+            if build_cinterfacetutorial:
+                build_args += ['-o', join(build_dir, 'lib' + image_name)]
+            else:
+                build_args += ['-o', join(build_dir, image_name)]
+
+            mx.log(f"native_image {' '.join(build_args)}")
+            native_image(build_args)
+
+            if build_cinterfacetutorial:
+                if mx.get_os() != 'windows':
+                    c_command = ['cc', '-g', join(tutorial_c_source_dir, 'cinterfacetutorial.c'),
+                                 '-I.', '-L.', '-lcinterfacetutorial',
+                                 '-ldl', '-Wl,-rpath,' + build_dir,
+                                 '-o', 'cinterfacetutorial']
+
+                else:
+                    c_command = ['cl', '-MD', join(tutorial_c_source_dir, 'cinterfacetutorial.c'), '-I.',
+                                 'libcinterfacetutorial.lib']
+                mx.log(' '.join(c_command))
+                mx.run(c_command, cwd=build_dir)
+        if not build_only and mx.get_os() == 'linux':
+            # copying the most recent version of gdb-debughelpers.py (even if the native image was not built)
+            mx.log(f"Copying {gdbdebughelpers_py} to {build_dir}")
+            mx.copyfile(gdbdebughelpers_py, join(build_dir, 'gdb-debughelpers.py'))
+
+            gdb_command = gdb_args + [
+                '-iex', f"set auto-load safe-path {join(build_dir, 'gdb-debughelpers.py')}",
+                '-x', testfile, join(build_dir, image_name)
+            ]
+            mx.log(' '.join(gdb_command))
+            # unittest may result in different exit code, nonZeroIsFatal ensures that we can go on with other test
+            mx.run(gdb_command, cwd=build_dir, nonZeroIsFatal=False)
+
+    if not with_isolates_only:
+        run_debug_test('prettyPrinterTest', test_pretty_printer_py, test_source_path, False,
+                       extra_args=test_pretty_printer_args)
+    run_debug_test('prettyPrinterTest', test_pretty_printer_py, test_source_path, extra_args=test_pretty_printer_args)
+    run_debug_test('prettyPrinterTest', test_settings_py, test_source_path, extra_args=test_pretty_printer_args, skip_build=True)
+    run_debug_test('prettyPrinterTest', test_svm_util_py, test_source_path, extra_args=test_pretty_printer_args, skip_build=True)
+
+    run_debug_test('cinterfacetutorial', test_cinterface_py, tutorial_source_path, build_cinterfacetutorial=True,
+                   extra_args=test_cinterface_args)
+
+    run_debug_test('classLoaderTest', test_class_loader_py, test_source_path, extra_args=test_class_loader_args)
+
+
+
 def _javac_image(native_image, path, args=None):
     args = [] if args is None else args
     mx_util.ensure_dir_exists(path)
@@ -1520,6 +1639,30 @@ def debuginfotestshared(args, config=None):
     # ideally we ought to script a gdb run
     native_image_context_run(_cinterfacetutorial, all_args)
 
+@mx.command(suite_name=suite.name, command_name='gdbdebughelperstest', usage_msg='[options]')
+def gdbdebughelperstest(args, config=None):
+    """
+    builds and tests gdb-debughelpers.py with multiple native images with debuginfo
+    """
+    parser = ArgumentParser(prog='mx gdbdebughelperstest')
+    all_args = ['--output-path', '--with-isolates-only', '--build-only', '--test-only']
+    masked_args = [_mask(arg, all_args) for arg in args]
+    parser.add_argument(all_args[0], metavar='<output-path>', nargs=1, help='Path of the generated image', default=[join(svmbuild_dir(), "gdbdebughelperstest")])
+    parser.add_argument(all_args[1], action='store_true', help='Only build and test the native image with isolates')
+    parser.add_argument(all_args[2], action='store_true', help='Only build the native image')
+    parser.add_argument(all_args[3], action='store_true', help='Only run the tests')
+    parser.add_argument('image_args', nargs='*', default=[])
+    parsed = parser.parse_args(masked_args)
+    output_path = unmask(parsed.output_path)[0]
+    with_isolates_only = parsed.with_isolates_only
+    build_only = parsed.build_only
+    test_only = parsed.test_only
+    native_image_context_run(
+        lambda native_image, a:
+            _gdbdebughelperstest(native_image, output_path, with_isolates_only, build_only, test_only, a), unmask(parsed.image_args),
+        config=config
+    )
+
 @mx.command(suite_name=suite.name, command_name='helloworld', usage_msg='[options]')
 def helloworld(args, config=None):
     """
@@ -1878,6 +2021,17 @@ def isJDKDependent(self):
         return True
 
 
+class GDBDebugHelpers(mx.ArchivableProject):
+    def output_dir(self):
+        return os.path.join(self.dir, 'src', self.name, 'gdbpy')
+
+    def archive_prefix(self):
+        return ''
+
+    def getResults(self):
+        return [os.path.join(self.output_dir(), 'gdb-debughelpers.py')]
+
+
 class SubstrateCompilerFlagsBuilder(mx.ArchivableProject):
 
     flags_build_dependencies = [