Replace language server WebSocket connection with Ydoc #14438

4e6 · 2025-12-08T10:02:53Z

Pull Request Description

close Avoid networking layer of Akka by ydoc-server talking directly to the engine+ls #14142

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

JaroslavTulach · 2025-12-09T09:28:25Z

lib/java/ydoc-api/src/main/java/module-info.java

@@ -0,0 +1,3 @@
+module org.enso.ydoc.api {
+  exports org.enso.ydoc.api;


There is no need for a new API module.

there already is YdocServerApi

please put the new interfaces there as inner classes of YdocServerApi.

Possibly you can move YdocServerApi to the new org.enso.ydoc.api module.

but the whole API needs to be co-located

JaroslavTulach · 2025-12-09T09:30:54Z

lib/java/ydoc-api/src/main/java/org/enso/ydoc/api/NoOpMessageCallbacks.java

+package org.enso.ydoc.api;
+
+public class NoOpMessageCallbacks implements MessageCallbacks {
+  public static final NoOpMessageCallbacks INSTANCE = new NoOpMessageCallbacks();


Please don't put NoOpMessagesCallbacks implementation class into the API

remove it

if you have to have it, then encapsulate it

put public static final MessageCallbacks NO_OP into MessageCallbacks and make the implementation class package private

JaroslavTulach

ideally HostAccess is configured internally
and both tests and production code just work with the default configuration
possible trick to achieve that is to crate a proxy around interfaces passed to JavaScript

JaroslavTulach · 2025-12-10T06:30:19Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Ydoc.java

      return this;
    }

+    public Builder hostAccessBuilder(HostAccess.Builder hostAccessBuilder) {


Why is this a builder?

Isn't just HostAccess instance good enough?

Suggested change

public Builder hostAccessBuilder(HostAccess.Builder hostAccessBuilder) {

public Builder hostAccess(HostAccess hostAccessBuilder) {

without WebEnvironment.defaultHostAccess being public, one might need a builder

but having both builder and WebEnvironment.defaultHostAccess seems unnecessary

JaroslavTulach · 2025-12-10T06:35:46Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Main.java

+          WebEnvironment.defaultHostAccess
+              // allowImplementations is required to call methods on JS objects from Java, i.e. to
+              // call methods on `YjsChannel` object returned from JS
+              .allowImplementations(YjsChannel.class)


allowing implementation of a well known interface is OK

but opening up allowAccess to a random callbacks.getClass() class is a security flaw

please create a proxy around callbacks and expose its methods

preferrably via @HostAccess.Explicit

or via allowAccess(ProxyMessageCallbacks.getDeclararedMethod("")) but while hardcoding ProxyMessageCallbacks class

The current state with:

.allowImplementations(YjsChannel.class) .allowPublicAccess(true);

is certainly better than the original one. Thanks.

Still, it be good to remove .allowPublicAccess(true) and use @HostAccess.Explicit annotation instead.

JaroslavTulach · 2025-12-10T06:36:49Z

lib/java/ydoc-api/src/main/java/module-info.java

@@ -0,0 +1,3 @@
+module org.enso.ydoc.api {
+  exports org.enso.ydoc.api;


Possibly you can move YdocServerApi to the new org.enso.ydoc.api module.

but the whole API needs to be co-located

lib/scala/json-rpc-server/src/main/scala/org/enso/jsonrpc/YdocJsonRpcServer.scala

JaroslavTulach · 2025-12-15T06:05:10Z

app/ydoc-channel/src/YjsChannel.ts

+          for (const item of items) {
+            // Only notify handlers if the message is from another sender
+            if (item.senderId !== this.senderId) {
+              this.notifyHandlers(item.payload)


As of 8713e6f we are probably dealing with ever growing Y.Array

somehow we need to find out that the "message has been delivered" to all handlers

that's kind of hard in a distributed framework

where handlers can appear and diappear

one solution would be to add receiverId to item

deliver the message only when senderId matches receiverId

e.g. if (item.receiverId === this.senderId)`

if there was only one receiver of a message

then it'd be the receiver's responsibility to remove the item from array

JaroslavTulach

It all looks quite decent. Few comments inline.

JaroslavTulach · 2025-12-29T12:45:28Z

app/ydoc-channel/src/YjsChannel.ts

+/**
+ * A bidirectional communication channel backed by Y.Array.
+ *
+ * This class allows multiple parties to send and receive messages through a shared


really "multiple parties"?

shouldn't the YjsChannel be just bidirectional?

app/ydoc-shared/src/util/net/__tests__/YjsTransport.test.ts

JaroslavTulach · 2025-12-29T13:46:20Z

engine/runner-common/src/main/java/org/enso/runner/common/YdocServerApi.java

 import java.util.ServiceLoader;
+import org.enso.ydoc.api.MessageCallbacks;

 public abstract class YdocServerApi {


Why don't we put YdocServerApi into org.enso.ydoc.api module as well? As this comment suggests?

lib/java/ydoc-polyfill/src/test/java/org/enso/ydoc/api/CallbacksTest.java

JaroslavTulach · 2025-12-29T14:00:52Z

lib/java/ydoc-polyfill/src/test/java/org/enso/ydoc/api/CallbacksTest.java

+
+  @Test
+  public void onConnectSend() throws Exception {
+    var res = new AtomicReference<>();


I'd slightly prefer you to create CountingTest class and register its instance instead

annotate its set method with @HostAccess.Explicit

JaroslavTulach · 2025-12-29T14:02:47Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Main.java

@@ -25,21 +28,22 @@ public static void main(String[] args) throws Exception {
      var then = System.currentTimeMillis();


Is this public static void main still useful for anything?

Shouldn't it be removed?

JaroslavTulach · 2025-12-29T14:04:30Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Main.java

+          WebEnvironment.defaultHostAccess
+              // allowImplementations is required to call methods on JS objects from Java, i.e. to
+              // call methods on `YjsChannel` object returned from JS
+              .allowImplementations(YjsChannel.class)


The current state with:

.allowImplementations(YjsChannel.class) .allowPublicAccess(true);

is certainly better than the original one. Thanks.

Still, it be good to remove .allowPublicAccess(true) and use @HostAccess.Explicit annotation instead.

JaroslavTulach · 2025-12-29T14:11:18Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Ydoc.java

                  var bindings = ctx.getBindings("js");
                  bindings.putMember("YDOC_HOST", hostname);
                  bindings.putMember("YDOC_PORT", port);
+                  bindings.putMember("YDOC_MESSAGE_CALLBACKS", callbacks);


registering non-primitive type as global symbol seems strange

with YDOC_HOST one could post it as environment variable, but

one cannot register callbacks via environment variable

why don't create a initializeYdocServer function

and call it here directly with proper arguments?

JaroslavTulach · 2025-12-29T14:14:40Z

app/ydoc-server-polyglot/src/main.ts


 configureAllDebugLogs(debug)

+if (YDOC_MESSAGE_CALLBACKS == undefined) {


this check makes main.ts unusable from command line

at least I believe usage from CLI was the motivation for these YDOC_ (environment) variables

why don't we define:

function initializeYdocServer(YDOC_HOST, YDOC_PORT, YDOC_LS_DEBUG, YDOC_MESSAGE_CALLBACKS) {

and then just call this method with proper arguments from Java?

JaroslavTulach · 2026-01-12T15:15:01Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Main.java

-      }
-    }
+  public static void main(String[] args) {
+    // main method declaration is required to build the native library


GraalVM native-image doesn't need main method

probably the invocation logic in NativeImage.scala is wrong

enso/project/NativeImage.scala

Line 260 in a282b3a

args ++= Seq("-jar", pathToJAR.toString)

when there is no mainClass then we shouldn't pass in any option

especially when shared=true

JaroslavTulach · 2026-01-14T05:29:28Z

...ydoc-server-registration/src/main/java/org/enso/ydoc/server/registration/YdocServerImpl.java

    assert impl != null;
-    var arr = ProxyArray.fromArray(hostname, "" + port);
-    impl.invokeMember("main", arr);
+    impl.invokeMember("launch", hostname, port + "", jsonChannelCallbacks, binaryChannelCallbacks);


the launch method is starting new Thread to form a JavaScript event loop executor

due to restrictions spelled out at Allow multiple threads to access a Channel simultaneously #14264

"only master can initialize connection on a new thread"

"then slave can reply on such a thread or any other thread which was initiated previously by the master"

thus we must start the executor here and pass it to the other JVM

btw. previously it wasn't a problem as there were no callbacks from JavaScript to Java

@4e6

) - @4e6 has troubles to obtain exceptions from _dual JVM mode_ - in his #14438 - this PR intends to make stacktraces of exceptions more useful

…nfig. No need for YjsCallbacksSynchronized constructor to be public.

4e6 self-assigned this Dec 8, 2025

4e6 added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 8, 2025

4e6 force-pushed the wip/db/14142-ydoc-websocket branch 2 times, most recently from 7356274 to 4371781 Compare December 8, 2025 18:03

JaroslavTulach reviewed Dec 9, 2025

View reviewed changes

JaroslavTulach reviewed Dec 10, 2025

View reviewed changes

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from 71e9b86 to 8713e6f Compare December 13, 2025 16:18

JaroslavTulach reviewed Dec 15, 2025

View reviewed changes

lib/scala/json-rpc-server/src/main/scala/org/enso/jsonrpc/YdocJsonRpcServer.scala Show resolved Hide resolved

JaroslavTulach reviewed Dec 15, 2025

View reviewed changes

4e6 force-pushed the wip/db/14142-ydoc-websocket branch 2 times, most recently from f122b95 to 9625b1f Compare December 29, 2025 11:49

JaroslavTulach reviewed Dec 29, 2025

View reviewed changes

enso-bot bot mentioned this pull request Dec 30, 2025

Run Standard.Snowflake in dual JVM mode #14474

Merged

4 tasks

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from 27df1af to b5f7639 Compare January 1, 2026 20:50

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from 149e994 to 9a35c00 Compare January 12, 2026 09:09

JaroslavTulach reviewed Jan 12, 2026

View reviewed changes

JaroslavTulach reviewed Jan 14, 2026

View reviewed changes

unfurl-links bot mentioned this pull request Jan 15, 2026

Using Arrow Date32 format to transfer LocalDate across JVMs #14632

Merged

2 tasks

JaroslavTulach mentioned this pull request Jan 20, 2026

Transfer stack trace elements from the other JVM when rethrowing #14666

Merged

2 tasks

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from 89fd5f5 to 5a30daa Compare January 21, 2026 14:05

4e6 added 8 commits January 22, 2026 20:39

add: ydoc-api

6e027a4

add: ydoc-channel

913ca09

update: simplify JsonRpcServer

7ab9d00

add: YdocJsonRpcServer

7511423

add: YjsWSTransport

f26a88a

misc: format

b1a33b5

add: JsonRpcServer callbacks

65ec4cb

add: callbacks test

461a208

4e6 and others added 23 commits January 22, 2026 20:39

update: yjs callbacks handle Uint8Arrays

d4a2329

update: receive ArrayBuffer

ae7aedc

update: receive ByteBuffer

7ceb98c

refactor: YjsChannelCallbacks

a319ad5

refactor: yjs binary channel

5fa9a1d

fix: native image building

b3fa26c

fix: reflection config

fc07e4e

fix: pnpm build

8a71fd2

add: YdocScheduledExecutionService

29b8814

DEBUG: channel.subscribe

922dcba

Log the process and its argument to be executed

bd9e5d9

DEBUG: reflection config

c4474c8

YjsChannelSynchronized.subscribe has to be mentioned in reflection co…

d4120e8

…nfig. No need for YjsCallbacksSynchronized constructor to be public.

Applying javafmtAll

11e0b2d

Applying javafmtAll

c3877c4

Need ability to proxy to Consumer

382d892

Update: run ydoc as daemon

dc726f7

Update: pnpm i

e5b19f7

misc: format

9e360e8

misc: pnpm lint

2db1a70

misc: cleanup YjsChannel

035928b

update: YdocTest

06dfe34

fix: ls tests

75734a9

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from dead671 to 75734a9 Compare January 22, 2026 20:42

4e6 added 6 commits January 22, 2026 20:52

fix: format

f42ecbf

fix: typecheck

e30b40a

misc: cleanup

954a3cf

fix: bazel build

acf3afe

update: lintAll on CI

6e106b0

misc: tests

c021f3c

		@@ -0,0 +1,3 @@
		module org.enso.ydoc.api {
		exports org.enso.ydoc.api;

	public Builder hostAccessBuilder(HostAccess.Builder hostAccessBuilder) {
	public Builder hostAccess(HostAccess hostAccessBuilder) {

		@@ -25,21 +28,22 @@ public static void main(String[] args) throws Exception {
		var then = System.currentTimeMillis();


		configureAllDebugLogs(debug)

		if (YDOC_MESSAGE_CALLBACKS == undefined) {

Replace language server WebSocket connection with Ydoc #14438

Are you sure you want to change the base?

Replace language server WebSocket connection with Ydoc #14438

Uh oh!

Conversation

4e6 commented Dec 8, 2025

Pull Request Description

Important Notes

Checklist

Uh oh!

JaroslavTulach Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JaroslavTulach Dec 9, 2025 •

edited

Loading

JaroslavTulach Dec 29, 2025 •

edited

Loading