Understanding the Slynk protocol?

[HiPhish]

Hello,

I'm a Neovim user who wants something akin to Sly. Obviously I will have to do the client side myself, but I want to use Slynk as the server component. The first step is understanding how the protocol works so that I can connect the editor to Slynk. I know how to start a standalone Slynk process, but I don't know what format the messages need to be in.

(slynk:create-server :port 0)

This will cause Slynk to listen for TCP messages on some port. Next I would like to feed messages to that port using netcat. So here are my questions:

• What format need the messages be in?
• How are data types serialized? If an expression like (+ 2 3) is sent to the server for evaluation, is it simple encoded as a string?
• What about security? How does Sly authenticate itself towards Slynk?
• Is TCP the only communication channel? No stdio?

I think once I know how to exchange messages I can start chipping away at the actual Neovim plugin. At the moment I am using Vlime, which is nice, but it is very clunky. There are so many nice Neovim improvements that could be made and I would like if both communities could share the same backend tooling. I don't know Emacs Lisp, but if someone can point me to specific functions I might be able to make sense of them.

[joaotavora]

Hello Alejandro,

• What format need the messages be in?
• How are data types serialized? If an expression like (+ 2 3) is sent to the server for evaluation, is it simple encoded as a string?

To answer most of your questions, the quickest way is to see the protocol happening live in front of you. If you can start up Sly (for emacs), pull up the *sly-events* buffer and follow along as you do things. Here's an example bit of a transcript:

(:emacs-channel-send 1
                     (:process "(+ 2 3)"))
(:channel-send 1
               (:write-values
                (("5 (3 bits, #x5, #o5, #b101)" 5 "5"))))
(:channel-send 1
               (:prompt "COMMON-LISP-USER" "CL-USER" 0 6))
(:emacs-channel-send 1
                     (:process "(princ 'hello)"))
(:channel-send 1
               (:write-values
                (("HELLO" 6 "'hello"))))
(:channel-send 1
               (:prompt "COMMON-LISP-USER" "CL-USER" 0 7))

This is plain UTF-8 encoded text over TCP. Whitespace not very relevant.

• What about security? How does Sly authenticate itself towards Slynk?

Sly authenticates with a sly-secret thing. I don't know the details, to be honest, as I never use it. SLY is meant to connect to locally established servers on single user systems. It also allows you to connect through the network to other systems, in which case you have to sort out security using some other program (for example, only allowing connections on a port that's gated through an SSH tunnel). There was some talk about this some years ago in the SLIME mailing list.

• Is TCP the only communication channel? No stdio?

The way M-x sly (and M-x slime this part hasn't changed) works is it starts up an inferior process and briefly talks to it via stdio to convince it to start a Slynk server (over TCP). Then SLY connects to this server. You can also M-x sly-connect if the server has been started already. This approach is very flexible as it allows many clients per server, persistent servers, and also multiple communication channels between the same client and server (i.e. as in the REPL "dedicated stream"). It also allows you to snoop traffic with a tool like Wireshark.

[HiPhish]

That is not quite what I wanted to know. Let's say you wanted to test Slynk without Emacs, you would need to build a mock client that lets you send raw bytes. So what format would the messages need to be at a byte level? Through experimentation and poking in the Slynk code I have found that I can enter the following into netcat

2c    (:emacs-rex (slynk:connection-info) nil t 1)

The message consists of a header which is always 6 bytes and gives the length of the message in hexadecimal notation, followed by the actual message. Slynk then gave me back a giant s-expression which I would have to process client-side. It looks like this format of size followed by body is used by all messages. The next question is what messages exist. What is the deal with the arguments nil, t and 1 above? Some messages require a channel, what is a channel? A worker thread?

It also allows you to connect through the network to other systems, in which case you have to sort out security using some other program (for example, only allowing connections on a port that's gated through an SSH tunnel). There was some talk about this some years ago in the SLIME mailing list.

Makes sense, if my system is already compromised to the point where one process can just write to arbitrary ports I am already done for. And for networking it makes sense to rely on proven tools like SSH.

[joaotavora]

That is not quite what I wanted to know.

I'm sorry, I did my best. If I was in your position, I would still open Emacs and see how it works, because Emacs contains the reference implementation of a Slynk client.

If you want to build a Slynk client for Neovim, it's going to be much harder if not downright impossible if you don't want to look at this reference implementation, because there is really no better description, no real documentation besides the code. I can give you a pointer or two once in a while, of course.

Besides if you're interested in doing this, you must surely be interested in Common Lisp, and if you're interested in Common Lisp, then Emacs Lisp isn't that far off a dialect.

The message consists of a header which is always 6 bytes and gives the length of the message in hexadecimal notation, followed by the actual message.

Oh right, that's indeed 6 UTF-8 chars of indicating the length in hex. That's slightly silly but it was an inheritance from SLIME. And then the corresponding number of UTF-8 characters.

I had forgotten about this detail. How did I look it up? I looked at the Elisp functions sly-net-send and sly-net-encode-length.

What is the deal with the arguments nil, t and 1 above? Some messages require a channel, what is a channel? A worker thread?

Read CONTRIBUTING.md for some hints. But most of all read the Elisp in SLY to understand what it means. Threads only exist Lisp side, by the way, Emacs has no threads. It's close to JavaScript (but not horrible) in that regard.

[HiPhish]

I'm sorry, I did my best. If I was in your position, I would still open Emacs and see how it works, because Emacs contains the reference implementation of a Slynk client.

So I need to first get a hang of Emacs. That's precisely the thing I was trying to avoid, but I guess I'll have to bite the bullet on this one. My hope was that looking at the server code would be enough to understand it. I am not interested in the UI side of the client, I will have to do all of that completely differently in Neovim anyway. My plan was as follows:

• Understand the protocol and be able to feed messages to the server by hand
• Be able to connect from Neovim to Slynk so I can feed prepared messages
• Implement in Lua all the s-expression parsing that Emacs gives you for free (not looking forward to this one)
• Implement very primitive UI that's mostly manual
• Implement the actual nice UI I would like to use

Besides if you're interested in doing this, you must surely be interested in Common Lisp, and if you're interested in Common Lisp, then Emacs Lisp isn't that far off a dialect.

I am quite good at Common Lisp, and Emacs Lisp is close enough that I can get the gist of what it says, but I fear that when it comes to low-level details the gist is not good enough.

[joaotavora]

Yes I understand your idea. I'm talking how I would do it. I might have been exaggerating when I said it's impossible without looking at Emacs, but it's not. It's just harder.

You might want to contact the author of https://github.com/s-clerc/slyblime to see how he/she proceeded.

When I re-wrote YASnippet for Emacs, I pulled up the TextMate editor a lot to see how it worked.

[grpineiro]

First, sorry for intruding, but I found the discussion interesting.

For a while, I was looking at a neovim plugin project called "conjure". A swank client was already made for it, but a few months ago, a guy is rewriting the client to support slynk instead of swank.

Olical/conjure#348

It maybe help you to understand, there's only one thing, one part of the plugin is not written in lua, but written in "fennel", a lisp that translates to lua.

[HiPhish]

That's really cool, I did not know Conjure has Common Lisp support as well. Fennel is not an issue, I can read it (it's more or less Lua with a different syntax). This is definitely going to be easier than reading the Sublime extension code.

[joaotavora]

Nice, good input @grpineiro and so @HiPhish can be happy and stay away from dreaded emacs ☠️ 🤣

Now seriously, converting from swank to slynk is relatively easy and I didn't gratuitously change the protocol, so it's very useful indeed to have such a starting point.

But for sly's REPL, which is considerably more sophisticated and featureful than slime's, it's better to either read the Elisp code (and experiment!) or go to slyblime (and, again, experiment a lot).