`Channel` connecting two `JVM` instances #13206

JaroslavTulach · 2025-05-30T12:51:32Z

Pull Request Description

Enhances the JVM class with support for extensible messages to be transferred between the two JVMs
Addresses part of Dual NI + JVM Mode for Loading Libraries #13172
builds upon Persistable.Pool to group related instances into pool #13201

Usage

@Persistable(id = 323432)
record PowerOf(int value) implements Function<Channel,  Integer> {
  public Integer apply(Channel otherJVM) {
     return value * value;
  }
}

final class PoolOfMessages implements Supplier<Persistable.Pool> {
  public Persistable.Pool get() {
    return Persistables.POOL;
  }
}

JVM jvm = JVM.create(....);
var channel = Channel.create(jvm, PoolOfMessages.class);
var four = channel.execute(new PowerOf(2))
assert 4 == four.intValue();

PoolOfMessages must be available for Class.forName in the jvm - either on classpath or module path.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
All code follows the
Scala,
Java,
Unit tests have been written where possible.
- Call from SVM to HotSpot JVM via a Message - back and forth factorial
- Call from HotSpot back to SVM via Message - done
- Propagation of exceptions from one side to another
- Extensibility of the Pool with other messages
- Select compatible Pool on both sides of the Channel

github-actions · 2025-05-30T12:52:33Z

✨ GUI Checks Results

Summary

🧹 Prettier: ✅ Passed
🧰 GUI Checks: ❌ Failed
📚 Storybook: ❌ Failed

⚠️ Action Required: Please review the failed checks and fix them before merging.

See individual check results for more details.

…th reflection

JaroslavTulach · 2025-06-03T05:43:46Z

build.sbt

    ),
-    Compile / moduleDependencies ++= slf4jApi ++ Seq(
-      "org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion


NetBeans Lookup shouldn't be used during runtime: 879a092

JaroslavTulach · 2025-06-03T05:45:04Z

build.sbt

@@ -4313,6 +4311,7 @@ lazy val `os-environment` =
      ),
      Compile / internalModuleDependencies ++= Seq(
        (`engine-common` / Compile / exportedModule).value,
+        (`persistance` / Compile / exportedModule).value,


os-environment now contains support for passing Messages via Channel. Messages serde is handled via persistance - hence the dependency.

…MSUtils warnings

JaroslavTulach · 2025-06-03T13:38:51Z

lib/java/os-environment/src/main/java/org/enso/os/environment/jni/Channel.java

+import org.graalvm.nativeimage.c.type.CCharPointer;
+import org.graalvm.nativeimage.c.type.CTypeConversion;
+
+/** Channel connects two {@link JVM} instances. */


Since e28e6ea the channels are allocated in pairs. There is always an SubstrateVM side and HotSpot JVM side of the channel. Right now these two sides are going to have the same ID stored in ID_TO_CHANNEL map.

JaroslavTulach · 2025-06-03T17:51:16Z

lib/java/os-environment/src/main/java/org/enso/os/environment/jni/Channel.java

+      Function<MemorySegment, Long> send) {
+    try (var arena = Arena.ofConfined()) {
+      var bytes = pool.write(msg, null);
+      var memory = arena.allocate(Math.max(bytes.length, 4096));


Current limitation: the size of reply cannot be bigger than size of request or 4096.

I guess we can live with it and integrate

then proceed with other parts of Dual NI + JVM Mode for Loading Libraries #13172 to execute something in the other JVM

then go back and fix this limitation and also do some performance work

JaroslavTulach · 2025-06-04T05:51:12Z

Debugging the HotSpot code

One can influence HotSpot virtual machine options created by the JVM class by _JAVA_OPTIONS environment variable. Thus one can be debug with:

enso$ sbt os-environment/test
enso$ _JAVA_OPTIONS=-agentlib:jdwp=transport=dt_socket,address=5005 ./test-os-env
ERROR: transport error 202: connect failed: Connection refused
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)

E.g. instruct your IDE to listen on 5005 port and debug the code running in HotSpot JVM.

Btw:

Replace JAVA_OPTS variable with JAVA_TOOL_OPTIONS #13256 investigates whether we could use such a style for all JVMs

Akirathan

Looks good. Few minor comments inline.

lib/java/os-environment/src/main/java/module-info.java

lib/java/os-environment/src/main/java/org/enso/os/environment/jni/Channel.java

lib/java/os-environment/src/test/java/org/enso/os/environment/jni/LoadClassTest.java

hubertp · 2025-06-05T08:11:45Z

Our CI is reporting problems with this change on MacOS ARM:

 INFO ide_ci::program::command: sbt ℹ️ [1/8] Initializing...                                                                                    (0,0s @ 0,95GB)
 INFO ide_ci::program::command: sbt ℹ️ Error: Support for the Foreign Function and Memory API is currently available only on the AMD64 architecture.
 INFO ide_ci::program::command: sbt ℹ️ Internal exception: com.oracle.svm.core.util.UserError$UserException: Support for the Foreign Function and Memory API is currently available only on the AMD64 architecture.

https://github.com/enso-org/enso/actions/runs/15457103830/job/43511280601#step:10:3634

GitHub
Nightly Release · enso-org/enso@8931518
Enso Analytics is a self-service data prep and analysis platform designed for data teams. - Nightly Release · 8931518

JaroslavTulach · 2025-06-05T08:32:41Z

Our CI is reporting problems with this change on MacOS ARM:

 INFO ide_ci::program::command: sbt ℹ️ [1/8] Initializing...                                                                                    (0,0s @ 0,95GB)
 INFO ide_ci::program::command: sbt ℹ️ Error: Support for the Foreign Function and Memory API is currently available only on the AMD64 architecture.
 INFO ide_ci::program::command: sbt ℹ️ Internal exception: com.oracle.svm.core.util.UserError$UserException: Support for the Foreign Function and Memory API is currently available only on the AMD64 architecture.

https://github.com/enso-org/enso/actions/runs/15457103830/job/43511280601#step:10:3634

I guess I should have read this GraalVM document sooner:

foreign calls are supported on the x64 architecture. Specifically, downcalls are supported on x64 Linux, Windows and MacOS, while upcalls are supported only on x64 Linux.

I'll think about alternative approaches today:

@fangerer points us to [GR-58659] [GR-58660] Add FFM Support for {darwin,linux}-aarch64 on SubstrateVM oracle/graal#10856 (and not [GR-63074] Support FFM API on macOS/x64. oracle/graal#11311) - which will be available in GraalVM25
@tzezula recommends native bridge
there also is jniutils module which might come handy

Sending messages between two JVMs with the help of Persistance.Pool

64fd1ac

JaroslavTulach requested review from 4e6, hubertp and Akirathan as code owners May 30, 2025 12:51

JaroslavTulach marked this pull request as draft May 30, 2025 12:51

JaroslavTulach mentioned this pull request May 30, 2025

Dual NI + JVM Mode for Loading Libraries #13172

Closed

3 tasks

JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label May 30, 2025

JaroslavTulach self-assigned this May 30, 2025

ExecuteMainClass message doesn't have a return value

2e3911c

enso-bot bot mentioned this pull request May 31, 2025

Deliver expression updates via Ydoc AST elements #11159

Open

JaroslavTulach added 8 commits June 2, 2025 08:00

Exchange messages between JVMs. Send ReportResult back.

5b42e1d

Introducing channel between two JVMs

9dc5646

Using RequestFactorial message

4d69228

Message with a result

1e753eb

Return BigInteger result

d7d547b

Remove runtime dependency on Lookup

879a092

Back and forth computation of factorial

a23cdd5

Enabling Foreign API support

eeab6a9

JaroslavTulach added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label Jun 3, 2025

JaroslavTulach added 2 commits June 3, 2025 07:14

Moving Channel and Message into their own class

cf02db1

Handling executeMain the old way to avoid access checks associated wi…

32aa5b0

…th reflection

JaroslavTulach commented Jun 3, 2025

View reviewed changes

JaroslavTulach added 6 commits June 3, 2025 09:17

Removing usage of NetBeans Lookup from benchmarks

22de1c4

Removing org-openide-util-lookup from componentModulesPaths to fix JP…

5e1a8e7

…MSUtils warnings

Allocate both peers of the Channel at once

e28e6ea

Look a Channel peer up by its ID

7133830

Add debugging symbols when ENSO_LAUNCHER=debug

3f16786

Multiple nested callbacks are possible

b04963d

JaroslavTulach linked an issue Jun 3, 2025 that may be closed by this pull request

Dual NI + JVM Mode for Loading Libraries #13172

Closed

3 tasks

JaroslavTulach commented Jun 3, 2025

View reviewed changes

JaroslavTulach added 4 commits June 3, 2025 15:53

Allow implementation of Channel messages as records

d11a0cf

Avoid global pool reference when local one is available

42d24f4

Load pool from a specified Class supplying it

21fb777

Public factory method to create a Channel

bc9aab7

JaroslavTulach marked this pull request as ready for review June 3, 2025 17:42

JaroslavTulach commented Jun 3, 2025

View reviewed changes

JaroslavTulach changed the title ~~Sending messages between two JVMs with the help of Persistance.Pool~~ Sending messages between two JVMs with the help of Persistance.Pool Jun 3, 2025

JaroslavTulach added 4 commits June 4, 2025 05:13

RuntimeVisualizationsTest uses HandlerFactory

4531025

Basic handling of exceptions in the other JVM

fa46a7c

Propagate exceptions back and forth among JVMs

0445962

Allocate the method handle at Channel creation time

93c6fd5

JaroslavTulach changed the title ~~Sending messages between two JVMs with the help of Persistance.Pool~~ Channel connecting two JVM instances Jun 4, 2025

Akirathan approved these changes Jun 4, 2025

View reviewed changes

JaroslavTulach added the CI: Ready to merge This PR is eligible for automatic merge label Jun 4, 2025

mergify bot merged commit 62acca9 into develop Jun 4, 2025
116 of 121 checks passed

mergify bot deleted the wip/jtulach/JvmMessage13172 branch June 4, 2025 12:27

This was referenced Jun 5, 2025

Avoid the need for -H:+ForeignAPISupport #13230

Merged

Using Channel to load Java classes #13238

Merged

Replace JAVA_OPTS variable with JAVA_TOOL_OPTIONS #13256

Merged

unfurl-links bot mentioned this pull request Jun 11, 2025

Remove embedded native libraries from JARs #13150

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

`Channel` connecting two `JVM` instances #13206

`Channel` connecting two `JVM` instances #13206

Uh oh!

JaroslavTulach commented May 30, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 30, 2025 •

edited

Loading

Uh oh!

JaroslavTulach Jun 3, 2025

Uh oh!

JaroslavTulach Jun 3, 2025 •

edited

Loading

Uh oh!

JaroslavTulach Jun 3, 2025

Uh oh!

JaroslavTulach Jun 3, 2025

Uh oh!

JaroslavTulach commented Jun 4, 2025 •

edited

Loading

Uh oh!

Akirathan left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hubertp commented Jun 5, 2025 •

edited

Loading

Uh oh!

JaroslavTulach commented Jun 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Channel connecting two JVM instances #13206

Channel connecting two JVM instances #13206

Uh oh!

Conversation

JaroslavTulach commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Usage

Checklist

Uh oh!

github-actions bot commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✨ GUI Checks Results

Summary

Uh oh!

JaroslavTulach Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugging the HotSpot code

Uh oh!

Akirathan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hubertp commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JaroslavTulach commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

`Channel` connecting two `JVM` instances #13206

`Channel` connecting two `JVM` instances #13206

JaroslavTulach commented May 30, 2025 •

edited

Loading

github-actions bot commented May 30, 2025 •

edited

Loading

JaroslavTulach Jun 3, 2025 •

edited

Loading

JaroslavTulach commented Jun 4, 2025 •

edited

Loading

hubertp commented Jun 5, 2025 •

edited

Loading

JaroslavTulach commented Jun 5, 2025 •

edited

Loading