Native LSP support in micro

[glupi-borna]

Since micro is getting a lot of activity lately, I figured now is a good time to bring back this discussion.

Specifically, I'd like to see what the maintainers think about this topic, because I currently have a fork of micro based on the old LSP branch, and I managed to get a good amount of LSP features working (although in a fairly hacky way, with some stuff only implemented as a plugin):

Now, I don't think it's a tenable idea to merge this stuff back into micro -- on top of being kinda hacky, it's diverged a lot over the years and is mixed in with a handful of my own mods and experiments. However, in getting this working well enough to be my daily driver (for quite a while now too), I believe that I have a much better idea of how an LSP client fits into micro, and what needs to be done to get it working and looking good.

What I'm trying to ask is:

• Is there even any interest for getting this implemented? I realize that some people might see this as bloat, and I could understand the argument that micro is simply not that kind of editor. However, as far as I'm aware, there is currently more than one LSP client plugin implemented in micro, which to me says that there is at least some interest from the community.
• If there is interest, would you be open to a relative outsider working on this? I would be willing to volunteer some of my own time to get this implemented right, but there is a hefty amount of stuff that needs to be added, and we would have to discuss how a lot of the stuff would be implemented (specifically concerning the overlay tooltips, etc).
• Is this even a good time to implement this? With the current ongoing work on the highlighting code, and the eventual rewrite of the handling of the backing text buffer that zyedidia was talking about, there is a possible worst-case scenario where some amount of the LSP client code would have to be rewritten too, if it were implemented before those other changes. Although this could be mitigated somewhat by agreeing on some sort of minimal safe API that the LSP client could depend on.

To start, here's a rough overview of things that might need to be done, to get the conversation started (please feel free to point out stuff that is missing, in the likely case that I'm forgetting/don't know about something):

• Some sort of generic UI overlay system that we could use to implement tooltips/autocomplete boxes and other widgets that hover over the buffers.
  • Depending on how this is implemented, it could also be used to power fzf-like menus, command-palettes, and other small editor widgets, as well as dedicated screens for showing stuff like an integrated git client, or a filetree. If it was exposed to plugins, most of this stuff could also be implemented in lua-land, so micro's main codebase could stay relatively lean.
• Conversion of buffer Locs to and from UTF16 offsets, and possibly applying UTF16 edits directly to buffers.
  • LSP uses UTF16 and getting it to play nice with micro in my current fork was (and still is, sometimes) a major pain.
  • On the other hand, this maybe is not necessary since PositionEncodingKind now exists in the spec, but I'm not sure how well supported uft8 is in the wild.
• (Maybe) some way to display inlay hints?
• Some sort of integration between the LSP client and the highlighting engine would be nice to have, and it would probably be a good idea to think about it ahead of time.
• Plugins should likely also get access to LSP functionality, but since all of LSP is asynchronous, it would be good if a bit of thought was put into the API to make it ergonomic to use from lua.

I'd love to hear everyone's thoughts and get some input on all of this :)

[JoeKar]

• Is there even any interest for getting this implemented?

Yes and the community seems to have the same opinion.

• If there is interest, would you be open to a relative outsider working on this?

Yes, because the current maintainer have...at least AFAIK...an other focus and especially limited time. 😉

• Is this even a good time to implement this?

Depends.

Just my thoughts:
As you already mentioned, we've at least two LSP plugins already...

• micro-plugin-lsp @AndCake
• mlsp @Andriamanitra

...so why not bundling the efforts together?

If possible let us bring this into a very good shape as external plugin first and afterwards we can decide, if it is fundamental enough to include it as an default plugin. To be honest, since I don't use one of those plugins so far, I've no clue if it really can be implemented straight forward as an plugin, but including it natively will add a lot more complexity to micro, where we've already a lot to do to maintain the current code base.

[glupi-borna]

I see what you mean. In that case, do you think it would be a good idea to add small features piecewise to micro that would help support these plugins? That way, we could probably slowly build up to being able to implement a native LSP client sometime in the future (if that was even necessary, at that point), without hampering other progress in the process.

One of the first things that jumped out at me as I checked out both of these plugins is that micro currently does not provide a nice way for plugins to register a custom completer. Both plugins deal with this, with micro-plugin-lsp rolling it's own completions, separate from micro's, while mlsp directly sets micro's Buffer.Completions and Buffer.Suggestions arrays, manually calls Buffer.CycleAutocomplete, etc.

It might be a good first step to officially support registering custom completers in some way, then. But that's just from taking a cursory glance at the plugins - I'd love to hear what the plugin authors think.

[dmaluka]

do you think it would be a good idea to add small features piecewise to micro that would help support these plugins? That way, we could probably slowly build up to being able to implement a native LSP client sometime in the future (if that was even necessary, at that point), without hampering other progress in the process.

IMO a very good idea. Contribute relatively small and independent changes, so that maintainers can get familiar with those changes and review them relatively easily.

I, for instance, know almost nothing about LSP (and absolutely nothing about those two plugins), and I'm not sure I wish or have time to learn any more about that in the nearest future. But things like customizable autocompletion may be done generically enough to be useful even regardless of LSP?

[JoeKar]

But things like customizable autocompletion may be done generically enough to be useful even regardless of LSP?

Good point, definitely.
Currently we've the autocompletion in place, while scrolling the available options in the status bar. I can remember someone (issue/feature request?) requesting to have a list or something like that next to the position where the completion takes place (similar to Vim).

This brings us to two points:

• your mentioned customization (or at least one configurable option)
• the requested capability to easily draw a custom list, buffer, tooltip, popup (call it as you like 😉) on top of the current buffer (with w/h from (current) x/y?)

So nothing directly related to LSP in the first place, but with benefits for plugins needing such features.
Sounds like something to plan and do. 👍

[glupi-borna]

I looked into custom completers today, and as it turns out, they already work fine by just supplying a lua function as the completer argument:

local config = import("micro/config")

function command_complete(buf)
	-- completions are the literal text that
	-- will be inserted into the buffer
	local completions = {"1", "2", "3"}
	-- labels are the text that will be shown in
	-- the bottom bar, can be used for showing
	-- annotated completions
	local labels = {"one", "two", "three"}
	return completions, labels
end

function buffer_complete(buf)
	-- an example of what I mean by "annotated completions"
	local completions = {"foo", "bar", "baz"}
	local labels = {"foo: int", "bar: str", "baz: func"}
	return completions, labels
end

function init()
	config.MakeCommand(
		"custom_complete",

		function(bp, args)
			bp.Buf:Autocomplete(buffer_complete)
		end,

		command_complete
	)

	config.TryBindKey("F1", "command:custom_complete", false)
end

So, that's pretty good news! I suppose this is just another area where the docs could be significantly improved. I'll see if I can get on that this week.

[Andriamanitra]

I'm sort of indifferent about whether or not a LSP client should be included in the editor itself. I'm the author of the µlsp plugin but I still keep it disabled most of the time because I enjoy the minimal editor experience. I implemented my own even though micro-plugin-lsp already existed because I wanted something that is off by default but easy to fire up when needed, and allows running multiple language servers at the same time.

It might be a good first step to officially support registering custom completers in some way, then. But that's just from taking a cursory glance at the plugins - I'd love to hear what the plugin authors think.

I hadn't even thought about that but yeah that would be helpful. Auto-completion in both plugins is currently kind of buggy, just in different ways in each.

If we want to enable plugins to do a better job here's my wishlist as a plugin author:

• Some way to temporarily show tooltip or dialog on top of bufpane (for things like documentation on hover)
• Some way to select actions from a menu. My plugin currently uses read-only buffer with some custom event handlers for this (for go-to-references etc.) and there is a number of other plugins that rely on shelling out to fzf for similar purposes. Both ways are kind of clunky.
• onEdited(buf, startPos, endPos, newText) event handler. Currently both plugins register a ton of event handlers for all kinds of events that could possibly have edited a buffer, and then just send the full document (again) to the language server. Language servers support a TextEdit message that specifies a range that was modified, but figuring out that range would be quite difficult in Lua (especially with multi-cursors). Because of this both plugins are currently pretty much unusable with very long files (µlsp is little bit better because it waits for user to stop typing before sending the modified document, but still bad).
• Better ways for plugins to have complex configuration. Micro's settings.json doesn't allow nesting so micro-plugin-lsp crams its server configuration into a single string, and µlsp opts to use a Lua file for configuration instead. Neither is a great way to configure a plugin.

[Andriamanitra]

BTW looking at your examples of those "initialization settings", an obvious question comes to my mind: we don't really want to require the user to manually specify all those crazy expandWorkspaceToModule or diagnosticsDelay settings in the normal case, do we?

Correct. Language servers that follow the standard shouldn't require specific support or configuration at all, and the default for most (currently all) language servers is to omit the (optional) initializationOptions altogether. Users can add various bits of language server specific configuration if they wish to tweak their behavior. There are some exceptions though like the generic diagnostic-languageserver that can be configured to wrap any linter(s) and/or formatter(s) in a language server.

Yes, sane defaults should be chosen by the plugin already, but at least we should leave the door open for further user defined adjustments of them.

Well, I know it's very dirty and most probably insane or stupid again, but...

{
    [...]
    "lsp.lang1.myAdditionalOptions": "option1=1\noption2=2"
    [...]
}

...the plugin only needs to split the string by line breaks. Currently I'm not 100% sure, if it's json5 compliant, but we started to hack the parser already.

This would only make it harder for the plugin to handle because the options can have (arbitrarily nested) JSON objects as the values. This means we would need a JSON parser anyway, at which point we might as well put the entire configuration as one big JSON string under a single key instead of bothering with splitting on newlines.

[JoeKar]

This means we would need a JSON parser anyway, at which point we might as well put the entire configuration as one big JSON string under a single key instead of bothering with splitting on newlines.

But then we could directly return to the beginning, leave the set use case behind and store an dedicated, clean and readable config to one (the global one) or more locations (workspace-specific) and the plugin takes care of what it needs.

[glupi-borna]

Maybe I should have put more emphasis when I said this, but:

This might seem unreasonably large, but it is a worst-case scenario where the user wants to configure a lot of stuff - none of these options are mandatory.

Like @Andriamanitra already pointed out, the most common pattern of usage for language servers is to use them while providing 0 (or very few) settings, because the servers usually already have sane defaults, so (IMO) just using e.g. "lsp.[server].settings.[option].[suboption]": [value] should be good enough for a good 90% of use cases.

I'm going to say this next part against my better judgement, because I'm pretty convinced that we don't actually need nested maps in the settings to get things rolling right now:

Maybe an option would be to make set and show work with nested maps too, set lsp.gopls.settings.show_foo true could set the key in a nested map, and autocomplete properly. show could similarly be made to work properly with nested maps.

On top of that, the average user would not even need to know that nested maps in settings are something that exists, even if they use LSP. They simply wouldn't be exposed to them as long as none of the plugins that they want to configure need complicated settings.

[dmaluka]

Maybe an option would be to make set and show work with nested maps too, set lsp.gopls.settings.show_foo true could set the key in a nested map, and autocomplete properly. show could similarly be made to work properly with nested maps.

Then it would conflict with the actual lsp.gopls.settings.show_foo option, if there is one.

[glupi-borna]

Well, yeah, in the hypothetical situation where something like this is implemented, lsp.gopls.settings.show_foo would either not be a valid setting, or be an alias for the nested map thing. In the second case, "lsp.gopls.settings.show_foo": true would be shorthand for:

{
   "lsp": {
      "gopls": {
        "settings": {
            "show_foo": true
        }
      }
   }
}

However, I'm totally on the side of not introducing complexity where it is not needed, and so, unless somebody shows up with a really good argument for allowing this, I don't think the time it would take to get this implemented would be worth it.

[berkeleynerd]

The hex helix editor written in rust might provide a source of inspiration when considering how to implement various LSP features ergonomically in a CLi-based text editor. They've put a lot of work into this feature and it shows.

[glupi-borna]

I looked it up, but was unable to find an editor called "hex". Did you mean helix? I haven't checked it out yet. Does it do things significantly differently from neovim?

[berkeleynerd]

Darn, yes, Helix. Sorry about that. It has really great contextual on-boarding guidance and zero additional setup required to work with treesitter, lsp, and other components which provides a really great on-boarding experience. I'm not sure their interface to LSP and other features can't be replicated with enough elbow grease in vim or neovim but at least with Helix the various overlays seemed well implemented and performant. If it had a CUA mode I would probably be using it but I've just never felt like, for me personally, editing speed has ever been the blocker I needed to work around.