simplify remote plugins, massively #27949

justinmk · 2024-03-20T14:47:06Z

Problem

The "remote plugin" model currently is too complex, and 99% of the complexity provides very little benefit.

Lifecycle is complicated, unnecessarily hard to test, troubleshoot, and explain to plugin authors.
Every "remote plugin host" must implement 1000s of lines of code to solve the same problem in every API client. This is a pointless waste of time.
- Example: all of the code below is just for the "remote plugin" implementation in:
  - node-client:
  - go-client:
    - nvim/plugin/
- The benefits of the extra code are
  1. "multi-tenancy" (one node process for all node rplugins).
  2. "decorators" can be used in the remote module to define Nvim commands/autocmds.
Too many moving parts, too many concepts.
- why is a "plugin host" needed?
- why can't I just call vim.rplugin('node', 'path/to/index.js'), then call functions defined in the remote process?
- why do I need "decorators" (@pynvim.plugin) ? why can't I just use Lua to define commands/events in plugin/foo.lua?
  - @pynvim.command('MyCommand', …) vs Lua vim.api.nvim_create_user_command('MyCommand', function(...) vim.rpcrequest('MyCommand', ...))
:UpdateRemotePlugins and the "manifest" are extra state that must be resolved, refreshed, and healthchecked.

Solution

Reposition "remote plugins" as "remote modules". For example, a Node.js "remote plugin" is just a Node.js module that (1) imports "neovim", (2) can be started as a Nvim RPC client, and (3) handles requests from Nvim.
- Users no longer need to globally install neovim-node-host.
- Client defines RPC method handlers via setHandler().
- Client calls setup() (placeholder name) which attaches and calls nvim_set_client_info() with the methods defined by setHandler().
Eliminate the "host" / "client" distinction. The client is the host (it can handle requests from Nvim).
Drop "multi tenancy". Each plugin starts its own client ("host"). Plugins do not share a "host".
Drop host-specific "sugar" for registering Nvim-side commands/autocmds (aka "bindings"). Plugins define commands/autocmds in Lua, like any other Nvim plugin. The commands call methods in the remote module.

Implementation

Implementation details

From doc/remote_plugin.txt:

Plugin hosts ... take care of most boilerplate involved in defining commands, autocmds, and functions implemented over |RPC| connections. Hosts are loaded only when one of their registered plugins require it.

We can address the above use-cases as follows:

Remote-plugins are just plain Lua plugins (plugin/foo.lua) that start API clients and call functions on the client.
- Eliminates all the unnecessary complexity of trying to find foo.js/foo.py/… plugin "mains".

Make it easy from Lua to start any API client and treat it as a "remote host".

One (API client) process per plugin.
- Future: Consider "sharing" client processes in the future. Out of scope for now.
Eliminates all the redundant, per-platform (js/py/…) impls that implement a "remote host".
Drop the concept of a "plugin host". We only need plain old API clients, not a "plugin host".
Nvim Lua stdlib will provide (once—not in every API client!) an ergonomic interface to:
1. start a "known" API client
2. allow the caller to specify the path to the remote-plugin "module" or "main.
3. define commands/autocmds/functions that call remote functions.
  - "Decorators" are not need for this! If it's painful to do this in Lua, fix that once, in Nvim instead of "fixing" it N times in every API client!

Examples (compare to :help remote-plugin-example):

go-client example

pynvim example:
"Remote" Python code:

import pynvim

def main():
    pynvim.setHandler('command_handler', command_handler)
    pynvim.setHandler('autocmd_handler', autocmd_handler)
    pynvim.setHandler('function_handler', function_handler)
    # (placeholder name, TBD) 
    # Attaches and calls `nvim_set_client_info()` with the `methods` defined by `setHandler()`.
    pynvim.setup()
main()

"Local" Lua code:

local rplugin = vim.rplugin('python', 'path/to/init.py')

vim.api.nvim_create_user_command('Cmd',
  function(args)
    vim.rpcrequest(rplugin.chan_id, 'command_handler', args.args, args.range)
  end,
  { nargs = '*' })
vim.api.nvim_create_autocmd({'BufEnter'}, {
  pattern = {"*.c", "*.h"},
  callback = function(ev)
    vim.rpcrequest(rplugin.chan_id, 'autocmd_handler', ev.file)
  end
})
function my_func(...)
  vim.rpcrequest(rplugin.chan_id, 'function_handler', {...})
end

With the above design...
- ELIMINATES:
  - a fuckton of redundant documentation explaining crap like decorators, the "sync" flag, Remote plugin manifest, remote#host#RegisterPlugin, "bootstrapping" details.
  - a fuckton of code: neovim/src/plugin decorators/src/plugin neovim/src/host
  - rplugin/node/ runtimepath directory
  - "registration", :UpdateRemotePlugins, rplugin "manifest"
  - remote#host#PluginsForHost
  - NvimPlugin.registerFunction, NvimPlugin.registerAutocmd, NvimPlugin.registerCommand
- STILL NEEDED:
  - instead of NvimPlugin, the remote plugin uses the exact same NvimClient type that is returned by attach().
    A remote plugin is just a library that operates on the same old NvimClient that any other API client operates on.
  - provider#Poll()
  - detect()
  - require()
- GAINS:
  - The "bootstrapping" is now extremely obvioius and the plugin implementor fully controls when to call vim.rplugin() from Lua.
  - vim.rplugin() loads the plugin "main" and returns a channel:
    1. find the platform interpreter (node, python, …)
    2. start the interpreter with a stdio RPC channel, targeting the plugin "main" file.
      - The plugin "main" file is expected to attach() to stdio, and use the resulting NvimClient to serve requests.
    3. calls provider#Poll() until success.
    4. returns a channel id.
  - NO sugar for creating commands/functions/autocmds that connect to the remote plugin.
    If creating commands/functions/autocmds is cumbersome we should fix that IN GENERAL, not only for remote plugins.
```
vim.api.nvim_create_autocmd({'BufEnter'}, {
  pattern = {"*.c", "*.h"},
  callback = function(ev)
    vim.rpcrequest(rplugin.chan_id, 'autocmd_handler', ev.file)
  end
})
```

FAQ

How does a plugin avoid loading the interpreter (slow) on startup?
- To "share" an API client we could add an optional ns (namespace) parameter to vim.rplugin(), then it could be called on-demand and re-uses the existing channel if found.
  - This doesn't need to be solved right now, can just use jobstart().

Work plan

The text was updated successfully, but these errors were encountered:

tarruda · 2024-03-20T15:55:13Z

Hi @justinmk. Following up from your comment in a issue I was subscribed to.

I will try to answer some of your questions, but please take it with a grain of salt as it has been 8 years since I worked on Neovim:

Lifecycle is complicated, unnecessarily hard to test, troubleshoot, and explain to plugin authors.

Can you elaborate on this one? I don't remember and couldn't find anything in the docs.

Every "remote plugin host" must implement a 1000s of lines of code to solve the same problem in every API client. This is a pointless waste of time.
why is a "plugin host" needed?

The idea of a plugin host is to implement the necessary boilerplate code to communicate with Neovim. There should not be any need to write more than one plugin host per language (at least that was my goal at the time).

why can't I just call vim.rplugin('node', 'path/to/index.js'), then call functions defined in the remote process?

How would path/to/index.js expose functions? Neovim uses msgpack-rpc to do it, so at very least there should be some infrastructure that allows it to expose functions via RPC, which should be done by the plugin host.

why do I need "decorators" (@pynvim.plugin) ?

IIRC, this decorator would be part of the python plugin host and an easy way to export functions to Neovim via RPC.

why can't I just use Lua to define commands/events in plugin/foo.lua?

Not sure if I understood this question, but if you are writing Lua plugins, then it is not a remote plugin.

In hindsight, the idea of remote plugins might not have been that great. While in theory the ability to write plugins in any programming language is cool, in practice what happened is that the community ended up embracing Lua as the standard language for plugins (At least I haven't seen any major plugins written in anything other than Lua and vimscript).

justinmk · 2024-03-20T16:43:00Z

The idea of a plugin host is to implement the necessary boilerplate code to communicate with Neovim.

I am distinguishing "API client" vs "plugin host".

The "API client" part of node-client is here:
- https://github.com/neovim/node-client/tree/master/packages/neovim/src/attach
- https://github.com/neovim/node-client/tree/master/packages/neovim/src/api
The "plugin host" part of node-client is here:
- https://github.com/neovim/node-client/tree/master/packages/neovim/src/host
- https://github.com/neovim/node-client/tree/master/packages/neovim/src/plugin

The "plugin host" is 99% just nice-to-have "sugar" that makes it possible to define Nvim commands/autocmds from the remote. None of that is really necessary, it's easy to do that from Lua (or we should improve that).

There should not be any need to write more than one plugin host per language

But implementing the "remote host" nice-to-have stuff in every language is a chore, and a source of extra bugs, and doubles the amount of code + docs needed in each API client.

How would path/to/index.js expose functions?

The remote host will need to handle rpcrequest() calls from Nvim and "dispatch" them. That will require some sort of initialization/declaration in the remote module. But that still avoids needing to register things in a manifest from the Nvim (client) side.

IIRC, this decorator would be part of the python plugin host and an easy way to export functions to Neovim via RPC.

Yes. Not worth the complexity and bugs.

Not sure if I understood this question, but if you are writing Lua plugins, then it is not a remote plugin.

I'm proposing to move the bindings logic to Lua:

start and manage an API client(s), from Lua
call remote functions, and setup remote handlers, from Lua

but the business logic stays in the remote module.

Currently, trying to set up bindings from the remote side is 99% of the complexity, but 1% of the value. The business logic (not bindings) is where remote plugins can be useful. Furthermore, with this simplified design, we get it nearly for free.

in practice what happened is that the community ended up embracing Lua

That's parially because creating a remote plugin is too complicated. This proposal fixes that. There are cases where I want to integrate with code that doesn't have a CLI but can be loaded as a node/python/etc module.

Problem: The "remote plugin" concept is too complicated. neovim/neovim#27949 Solution: - Let the "client" also be the "host". Eliminate the separate "host" concept and related modules. - Let any node module be a "host". Any node module that imports the "neovim" package and defines mappings from a method name => function, is a "remote module". It can be loaded by Nvim as a regular "node client" which also responds to requests. TEST CASE / DEMO: const found = findNvim({ orderBy: 'desc', minVersion: '0.9.0' }) const nvim_proc = child_process.spawn(found.matches[0].path, ['--clean', '--embed'], {}); const nvim = attach({ proc: nvim_proc }); nvim.setHandler('foo', (ev, args) => { nvim.logger.info('handled from remote module: "%s": args:%O', ev.name, args); }); nvim.callFunction('rpcrequest', [(await nvim.channelId), 'foo', [42, true, 'bar']]); 2024-03-26 16:47:35 INF handleRequest: foo 2024-03-26 16:47:35 DBG request received: foo 2024-03-26 16:47:35 INF handled from remote module: "foo": args:[ [ 42, true, 'bar' ] ]

Problem: The "remote plugin" concept is too complicated. neovim/neovim#27949 Solution: - Let the "client" also be the "host". Eliminate the separate "host" concept and related modules. - Let any node module be a "host". Any node module that imports the "neovim" package and defines method handler(s) is a "remote module". It is loaded by Nvim same as any "node client". TEST CASE / DEMO: const found = findNvim({ orderBy: 'desc', minVersion: '0.9.0' }) const nvim_proc = child_process.spawn(found.matches[0].path, ['--clean', '--embed'], {}); const nvim = attach({ proc: nvim_proc }); nvim.setHandler('foo', (ev, args) => { nvim.logger.info('handled from remote module: "%s": args:%O', ev.name, args); }); nvim.callFunction('rpcrequest', [(await nvim.channelId), 'foo', [42, true, 'bar']]); 2024-03-26 16:47:35 INF handleRequest: foo 2024-03-26 16:47:35 DBG request received: foo 2024-03-26 16:47:35 INF handled from remote module: "foo": args:[ [ 42, true, 'bar' ] ]

justinmk · 2024-03-26T16:05:26Z

Skeleton / proof of concept: neovim/node-client#344

Problem: The "remote plugin" concept is too complicated. neovim/neovim#27949 Solution: - Let the "client" also be the "host". Eliminate the separate "host" concept and related modules. - Let any node module be a "host". Any node module that imports the "neovim" package and defines method handler(s) is a "remote module". It is loaded by Nvim same as any "node client". Old architecture: nvim rplugin framework -> node: cli.js -> starts the "plugin Host" attaches itself to current node process searches for plugins and tries to load them in the node process (MULTI-TENANCY) -> plugin1 -> plugin2 -> ... New architecture: nvim vim.rplugin('node', '…/plugin1.js') -> node: neovim.cli() nvim vim.rplugin('node', '…/plugin2.js') -> node: neovim.cli() 1. A Lua plugin calls `vim.rplugin('node', '/path/to/plugin.js')`. 2. Each call to `vim.rplugin()` starts a new node process (no "multi-tenancy"). 3. plugin.js is just a normal javascript file that imports the `neovim` package. 4. plugin.js provides a "main" function. It can simply import the `neovim.cli()` util function, which handles attaching/setup. TEST CASE / DEMO: const found = findNvim({ orderBy: 'desc', minVersion: '0.9.0' }) const nvim_proc = child_process.spawn(found.matches[0].path, ['--clean', '--embed'], {}); const nvim = attach({ proc: nvim_proc }); nvim.setHandler('foo', (ev, args) => { nvim.logger.info('handled from remote module: "%s": args:%O', ev.name, args); }); nvim.callFunction('rpcrequest', [(await nvim.channelId), 'foo', [42, true, 'bar']]); 2024-03-26 16:47:35 INF handleRequest: foo 2024-03-26 16:47:35 DBG request received: foo 2024-03-26 16:47:35 INF handled from remote module: "foo": args:[ [ 42, true, 'bar' ] ]

Problem: The "remote plugin" concept is too complicated. neovim/neovim#27949 Solution: - Let the "client" also be the "host". Eliminate the separate "host" concept and related modules. - Let any node module be a "host". Any node module that imports the "neovim" package and defines method handler(s) is a "remote module". It is loaded by Nvim same as any "node client". Story: - The value in rplugins is: 1. it finds the interpreter on the system 2. it figures out how to invoke the main script with the interpreter Old architecture: nvim rplugin framework -> node: cli.js -> starts the "plugin Host" attaches itself to current node process searches for plugins and tries to load them in the node process (MULTI-TENANCY) -> plugin1 -> plugin2 -> ... New architecture: nvim vim.rplugin('node', '…/plugin1.js') -> node: neovim.cli() nvim vim.rplugin('node', '…/plugin2.js') -> node: neovim.cli() 1. A Lua plugin calls `vim.rplugin('node', '/path/to/plugin.js')`. 2. Each call to `vim.rplugin()` starts a new node process (no "multi-tenancy"). 3. plugin.js is just a normal javascript file that imports the `neovim` package. 4. plugin.js provides a "main" function. It can simply import the `neovim.cli()` util function, which handles attaching/setup. TEST CASE / DEMO: const found = findNvim({ orderBy: 'desc', minVersion: '0.9.0' }) const nvim_proc = child_process.spawn(found.matches[0].path, ['--clean', '--embed'], {}); const nvim = attach({ proc: nvim_proc }); nvim.setHandler('foo', (ev, args) => { nvim.logger.info('handled from remote module: "%s": args:%O', ev.name, args); }); nvim.callFunction('rpcrequest', [(await nvim.channelId), 'foo', [42, true, 'bar']]); 2024-03-26 16:47:35 INF handleRequest: foo 2024-03-26 16:47:35 DBG request received: foo 2024-03-26 16:47:35 INF handled from remote module: "foo": args:[ [ 42, true, 'bar' ] ]

See: neovim/neovim#27949 This includes a helper module for defining remote modules as well as an acceptance spec to demonstrate their usage. I chose to implement a new DSL class just for remote modules because the existing plugin DSL is far too complicated for simple RPC handling. As remote plugins are phased out, I expect to phase out and eventually deprecate the existing plugin DSL.

See: neovim/neovim#27949 This includes a helper method for defining remote modules as well as an acceptance spec to demonstrate their usage. I chose to implement a new DSL class just for remote modules because the existing plugin DSL is far too complicated for simple RPC handling. As remote plugins are phased out, I expect to phase out and eventually deprecate the existing plugin DSL.

Problem: The "remote plugin" concept is too complicated. neovim/neovim#27949 Solution: - Let the "client" also be the "host". Eliminate the separate "host" concept and related modules. - Let any node module be a "host". Any node module that imports the "neovim" package and defines method handler(s) is a "remote module". It is loaded by Nvim same as any "node client". Story: - The value in rplugins is: 1. it finds the interpreter on the system 2. it figures out how to invoke the main script with the interpreter Old architecture: nvim rplugin framework -> node: cli.js -> starts the "plugin Host" attaches itself to current node process searches for plugins and tries to load them in the node process (MULTI-TENANCY) -> plugin1 -> plugin2 -> ... New architecture: nvim vim.rplugin('node', '…/plugin1.js') -> node: neovim.cli() nvim vim.rplugin('node', '…/plugin2.js') -> node: neovim.cli() 1. A Lua plugin calls `vim.rplugin('node', '/path/to/plugin.js')`. 2. Each call to `vim.rplugin()` starts a new node process (no "multi-tenancy"). 3. plugin.js is just a normal javascript file that imports the `neovim` package. 4. plugin.js provides a "main" function. It can simply import the `neovim.cli()` util function, which handles attaching/setup. TEST CASE / DEMO: const found = findNvim({ orderBy: 'desc', minVersion: '0.9.0' }) const nvim_proc = child_process.spawn(found.matches[0].path, ['--clean', '--embed'], {}); const nvim = attach({ proc: nvim_proc }); nvim.setHandler('foo', (ev, args) => { nvim.logger.info('handled from remote module: "%s": args:%O', ev.name, args); }); nvim.callFunction('rpcrequest', [(await nvim.channelId), 'foo', [42, true, 'bar']]); 2024-03-26 16:47:35 INF handleRequest: foo 2024-03-26 16:47:35 DBG request received: foo 2024-03-26 16:47:35 INF handled from remote module: "foo": args:[ [ 42, true, 'bar' ] ]

brianhuster · 2024-12-06T07:47:13Z

So vim.rplugin hasn't been implemented yet, right? I can't find it in the doc

justinmk · 2024-12-06T11:55:49Z

So vim.rplugin hasn't been implemented yet, right?

Correct, and it may not be needed for the first phase (if ever), it's just pseudocode.

The concepts here are already possible with the go-client (see neovim/go-client#167) and the neovim-ruby client (since neovim/neovim-ruby#107). I need to finish up the work on node-client and then update Nvim core to help with "migration" of legacy remote-plugins.

dlants · 2024-12-08T15:19:18Z

Hey, I ran into this when starting my node-based plugin.

I want to chime in and say that there's another benefit of implementing things this way, which is that it opens up the possibility of using alternate js runtimes - like deno and bun. Bun in particular claims much faster startup times and less memory overhead (and is more batteries-included with package management, testing and typescript compilation baked in).

I did try to set things up in a way similar to this by setting up an init.lua entrypoint on the lua side with:

vim.fn.jobstart({
    'bun', 'your-script.ts'
  }, {
    env = {
      NVIM_LISTEN_ADDRESS = socket
    }
  })

and then using node client's "attach" mechanism on the node side.

  const nvim = attach({ socket: process.env.NVIM_LISTEN_ADDRESS });

This got me the ability to manipulate nvim from bun. However, I got stuck on how to trigger bun code from nvim. So how to define a command for example.

I think there is an rpc channel from nvim to node, so it should be possible to send things along it from the lua side and receive it on the node end.

Overall this felt like a yak-shaving exercise for the actual plugin development so I didn't dig into it beyond this. But it would be nice to know how to set this up in a "future proof" way. Thanks, I appreciate all your work on nvim!

justinmk · 2024-12-10T00:32:39Z

However, I got stuck on how to trigger bun code from nvim. So how to define a command for example.

I think there is an rpc channel from nvim to node, so it should be possible to send things along it from the lua side and receive it on the node end.

@dlants yup, the only thing missing is neovim/node-client#344 which would allow your js code to define a handler.

justinmk added enhancement feature request api libnvim, Nvim RPC API plugin plugins and Vim "pack" remote-plugin plugins as RPC coprocesses (node.js, python, etc) labels Mar 20, 2024

This was referenced Mar 20, 2024

System remote plugin manifest #9413

Closed

rplugin: cache registrations in 'shada' file #4439

Closed

let remote hosts update their specs #6742

Closed

justinmk changed the title ~~simplify remote plugins~~ massively simplify remote plugins Mar 20, 2024

justinmk changed the title ~~massively simplify remote plugins~~ simplify remote plugins, massively Mar 20, 2024

This was referenced Mar 20, 2024

Prepare for version 1.0 #20451

Open

remote plugins: deprecate :UpdateRemotePlugins #5532

Closed

justinmk added this to the 0.11 milestone Mar 20, 2024

justinmk mentioned this issue Mar 26, 2024

WIP: redesign "remote plugins", eliminate "host" concept neovim/node-client#344

Draft

4 tasks

This was referenced Apr 7, 2024

Refactoring and tests for plugin Host and decorators neovim/pynvim#559

Draft

[WIP] Use function name for decorators neovim/pynvim#335

Closed

justinmk mentioned this issue Apr 17, 2024

export RPC functions neovim/node-client#108

Open

justinmk mentioned this issue Apr 23, 2024

simplify remote plugins neovim/pynvim#567

Open

justinmk mentioned this issue May 6, 2024

incorrect detection of ruby neovim in checkhealth #28642

Closed

alexgenco mentioned this issue May 30, 2024

Add basic remote module support neovim/neovim-ruby#107

Merged

This was referenced Jun 10, 2024

simplify remote plugins neovim/neovim-ruby#108

Open

Which plugins use :UpdateRemotePlugins ? #29270

Open

fidgetingbits mentioned this issue Jun 11, 2024

neovim: migrate to neovim's new remote plugin design cursorless-dev/cursorless#2397

Open

mrcjkb mentioned this issue Aug 5, 2024

feat: update remote plugins after install/update nvim-neorocks/rocks.nvim#485

Merged

justinmk mentioned this issue Aug 29, 2024

Setting g:node_host_prog in init.vim generates an error. #15308

Closed

justinmk mentioned this issue Sep 15, 2024

test: spawn_wait() starts a non-RPC Nvim process #30384

Merged

This was referenced Sep 30, 2024

legacy python provider is slow #7063

Closed

bindeval / funcrefs / if_py, if_lua #1898

Closed

justinmk mentioned this issue Oct 13, 2024

UpdateRemotePlugins does not work #30470

Open

bon-ami mentioned this issue Oct 19, 2024

Nvim 0.9.5 still can not register wilder.nvim with UpdateRemotePlugins on Windows gelguy/wilder.nvim#205

Open

This was referenced Nov 2, 2024

an API to get client capabilities #31050

Closed

API: get UI client-capabilities/implemented-features #31052

Open

dlants mentioned this issue Dec 23, 2024

feature: switch to bun after the new rplugin system is built. dlants/magenta.nvim#1

Closed

1 task

This was referenced Jan 2, 2025

pyfile fails on unittest.main() and breaks python provider #31834

Open

merge lua-client into main repo #11474

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simplify remote plugins, massively #27949

simplify remote plugins, massively #27949

justinmk commented Mar 20, 2024 •

edited

Loading

tarruda commented Mar 20, 2024

justinmk commented Mar 20, 2024 •

edited

Loading

justinmk commented Mar 26, 2024

brianhuster commented Dec 6, 2024

justinmk commented Dec 6, 2024 •

edited

Loading

dlants commented Dec 8, 2024 •

edited

Loading

justinmk commented Dec 10, 2024

simplify remote plugins, massively #27949

simplify remote plugins, massively #27949

Comments

justinmk commented Mar 20, 2024 • edited Loading

Problem

Solution

Implementation

FAQ

Related

Work plan

tarruda commented Mar 20, 2024

justinmk commented Mar 20, 2024 • edited Loading

justinmk commented Mar 26, 2024

brianhuster commented Dec 6, 2024

justinmk commented Dec 6, 2024 • edited Loading

dlants commented Dec 8, 2024 • edited Loading

justinmk commented Dec 10, 2024

justinmk commented Mar 20, 2024 •

edited

Loading

justinmk commented Mar 20, 2024 •

edited

Loading

justinmk commented Dec 6, 2024 •

edited

Loading

dlants commented Dec 8, 2024 •

edited

Loading