Completion being inserted at random place

[Hi-Angel]

Output of the command M-x company-diag

Details

Emacs 30.0.50 (x86_64-pc-linux-gnu) of 2024-02-21 on dell-g15
Company 0.10.2

company-backends: (company-capf company-etags company-dabbrev)

Used backend: company-capf

Value of c-a-p-f: (lsp-completion-at-point tags-completion-at-point-function)
Major mode: go-mode
Prefix: "Fprin"

Completions:
#("Fprint" 0 1 (face (completions-first-difference) lsp-completion-item #s(hash-table test equal data ("label" "Fprint" "kind" 3 "detail" "func(w io.Writer, a ...any) (n int, err error)" "documentation" #s(hash-table test equal data ("kind" "markdown" "value" "Fprint formats using the default formats for its operands and writes to w.
")) "preselect" t "sortText" "00000" "filterText" "Fprint" "insertTextFormat" 2 "textEdit" #s(hash-table test equal data ("range" #s(hash-table test equal data ("start" #s(hash-table test equal data ("line" 74 "character" 13)) "end" #s(hash-table test equal data ("line" 74 "character" 18)))) "newText" "Fprint")) "_emacsStartPoint" 2563)) lsp-sort-text "00000" lsp-completion-start-point 2563 lsp-completion-markers (2516 #<marker (moves after insertion) at 2521 in step_reattach_cdrom.go>) lsp-completion-prefix "Fprin") 1 6 (lsp-completion-item #s(hash-table test equal data ("label" "Fprint" "kind" 3 "detail" "func(w io.Writer, a ...any) (n int, err error)" "documentation" #s(hash-table test equal data ("kind" "markdown" "value" "Fprint formats using the default formats for its operands and writes to w.
")) "preselect" t "sortText" "00000" "filterText" "Fprint" "insertTextFormat" 2 "textEdit" #s(hash-table test equal data ("range" #s(hash-table test equal data ("start" #s(hash-table test equal data ("line" 74 "character" 13)) "end" #s(hash-table test equal data ("line" 74 "character" 18)))) "newText" "Fprint")) "_emacsStartPoint" 2563)) lsp-sort-text "00000" lsp-completion-start-point 2563 lsp-completion-markers (2516 #<marker (moves after insertion) at 2521 in step_reattach_cdrom.go>) lsp-completion-prefix "Fprin")) " func(w io.Writer, a ...any) (n int, err error) (Function)"
#("Fprintf" 0 1 (face (completions-first-difference) lsp-completion-item #s(hash-table test equal data ("label" "Fprintf" "kind" 3 "detail" "func(w io.Writer, format string, a ...any) (n int, err error)" "documentation" #s(hash-table test equal data ("kind" "markdown" "value" "Fprintf formats according to a format specifier and writes to w.
")) "sortText" "00001" "filterText" "Fprintf" "insertTextFormat" 2 "textEdit" #s(hash-table test equal data ("range" #s(hash-table test equal data ("start" #s(hash-table test equal data ("line" 74 "character" 13)) "end" #s(hash-table test equal data ("line" 74 "character" 18)))) "newText" "Fprintf")) "_emacsStartPoint" 2563)) lsp-sort-text "00001" lsp-completion-start-point 2563 lsp-completion-markers (2516 #<marker (moves after insertion) at 2521 in step_reattach_cdrom.go>) lsp-completion-prefix "Fprin") 1 7 (lsp-completion-item #s(hash-table test equal data ("label" "Fprintf" "kind" 3 "detail" "func(w io.Writer, format string, a ...any) (n int, err error)" "documentation" #s(hash-table test equal data ("kind" "markdown" "value" "Fprintf formats according to a format specifier and writes to w.
")) "sortText" "00001" "filterText" "Fprintf" "insertTextFormat" 2 "textEdit" #s(hash-table test equal data ("range" #s(hash-table test equal data ("start" #s(hash-table test equal data ("line" 74 "character" 13)) "end" #s(hash-table test equal data ("line" 74 "character" 18)))) "newText" "Fprintf")) "_emacsStartPoint" 2563)) lsp-sort-text "00001" lsp-completion-start-point 2563 lsp-completion-markers (2516 #<marker (moves after insertion) at 2521 in step_reattach_cdrom.go>) lsp-completion-prefix "Fprin")) " func(w io.Writer, format string, a ...any) (n int, err error) (Function)"
#("Fprintln" 0 1 (face (completions-first-difference) lsp-completion-item #s(hash-table test equal data ("label" "Fprintln" "kind" 3 "detail" "func(w io.Writer, a ...any) (n int, err error)" "documentation" #s(hash-table test equal data ("kind" "markdown" "value" "Fprintln formats using the default formats for its operands and writes to w.
")) "sortText" "00002" "filterText" "Fprintln" "insertTextFormat" 2 "textEdit" #s(hash-table test equal data ("range" #s(hash-table test equal data ("start" #s(hash-table test equal data ("line" 74 "character" 13)) "end" #s(hash-table test equal data ("line" 74 "character" 18)))) "newText" "Fprintln")) "_emacsStartPoint" 2563)) lsp-sort-text "00002" lsp-completion-start-point 2563 lsp-completion-markers (2516 #<marker (moves after insertion) at 2521 in step_reattach_cdrom.go>) lsp-completion-prefix "Fprin") 1 8 (lsp-completion-item #s(hash-table test equal data ("label" "Fprintln" "kind" 3 "detail" "func(w io.Writer, a ...any) (n int, err error)" "documentation" #s(hash-table test equal data ("kind" "markdown" "value" "Fprintln formats using the default formats for its operands and writes to w.
")) "sortText" "00002" "filterText" "Fprintln" "insertTextFormat" 2 "textEdit" #s(hash-table test equal data ("range" #s(hash-table test equal data ("start" #s(hash-table test equal data ("line" 74 "character" 13)) "end" #s(hash-table test equal data ("line" 74 "character" 18)))) "newText" "Fprintln")) "_emacsStartPoint" 2563)) lsp-sort-text "00002" lsp-completion-start-point 2563 lsp-completion-markers (2516 #<marker (moves after insertion) at 2521 in step_reattach_cdrom.go>) lsp-completion-prefix "Fprin")) " func(w io.Writer, a ...any) (n int, err error) (Function)"

Describe the issue

After company provides a completions list, sometimes choosing a candidate makes it paste the chosen text at random location of a buffer.

I'm sorry I'm unable to provide the exact steps-to-reproduce, but I'm open to try suggestions to dig into it when it happens. The issue seems to heavily rely on window and text sizes and the caret location.

It's been randomly happening in different modes (e.g. markdown-mode, go-mode, etc) over the years as far as I can remember, but I've never reported it due to not being able to find reliable way of reproducing. At some point I started using a workaround (see at the bottom) and forgot about it. But recently I disabled company-box and the problem re-appeared and I figured it needs to be somehow dealt with.

Steps to reproduce

(Note: as described above that most of the time it does not happen)

1. Type enough text to make company pop up a completion list
2. Select a candidate and press Enter

Expected behavior

It's inserted where your caret at

Actual behavior

It's inserted at a random location different from the caret placement

Video

2024-03-22.12-23-34.mp4

Workaround

company-box makes the problem disappear completely.

[Hi-Angel]

Workaround

company-box makes the problem disappear completely.

Ugh, actually… It seems I do have company-box enabled and it no longer helps.

Either way, I re-tested by disabling company-box and the problem reproduces, so I assure it has nothing to do with that mode.

[dgutov]

Hi!

Okay, obvious question: do you see this behavior with any other backends except for lsp-completion-at-point?

E.g. during Emacs Lisp completion. Or if you try the same scenario for Go using Eglot?

[dgutov]

That's not to say that lsp-mode is definitely a problem (I've been doing some coding in Go using it just fine), but this seems the most natural direction to investigate. Could be that this is caused by some combination of customizations.

[Hi-Angel]

I think I've seen this e.g. with markdown-mode (with no lsp-server). However now that I read my to-be-reported description from an year ago when I was trying to get this to reproduce with a Markdown document, I think it may have been a bit different. What was happening then is that once completion popup appears, the caret gets moved to some other place. What happens then with the popup I didn't document, maybe it disappears. I'm not sure if it's the same as what I've seen today with go-mode or not.

[Hi-Angel]

Anyway, let me then wait for it to reappear with another major mode and I'll report it back.

[dgutov]

We did indeed have a point jumping issue in certain situations (having to do with how popup is rendered and line overflow), but that should be fixed and it looked different in behavior.

[dgutov]

Note that it would be sensible to try to reproduce it with both Elisp and Eglot completion backends.

The first one would be easier to troubleshoot in case of the bug, but the latter might be necessary, given that some bugs only surface in the presence of subprocess output.

[Hi-Angel]

@dgutov so, I reproduced the bug which Idk if this one or a different one. But it's the one that I've been seeing for years and couldn't find exact steps to reproduce.

It does not require any lsp-server, i.e. it's just company in text-mode. Company version: company-20240311.1943

I shot a video of it happening, attaching below. You can see how completion appears, and then caret gets displaced elsewhere. I tested it with company-box mode as well and I still confirm that company-box still fixes the problem.

2024-04-12.11-19-06.mp4

[Hi-Angel]

…perhaps I could get somehow a stracktrace from the moment the completion appears…?

[dgutov]

You could get a stack trace printed to messages by adding a (backtrace) call somewhere in company-pseudo-tooltip-show, for example. But I'm not sure how that would help.

What could help in theory, is step-by-step debugging, but the problem is that it's unlikely to work with this feature.

[dgutov]

it's just company in text-mode.

It does look nasty. Is the problem consistent? Could you reduce it to a smaller file, a specific window configuration, or etc?

Anything that would help me to reproduce it on my machine.

[dgutov]

With a scenario starting with emacs -Q, of course.

[dgutov]

The essence of the problem is that the popup likely gets misrendered with that specific window configuration in a way that the point gets into an unexpected position (somewhere outside of the visible area). I have seen and fixed certain instances of this in the past.

[Hi-Angel]

You won't believe what just happened… I spent last 1.5 hour reducing the problem to the minimum testcase and just after I wrote on this report steps-to-reproduce that covered most of the problem except the geometry of the frame, I tried to attach the 1.txt file that is part of the steps, and then my browser got closed due to qutebrowser/qutebrowser#8143 and then subsequently my system crashed and I lost everything: the steps, the file… Wow…

Anyway. I have at least some code still left, so from memory:

The problem is caused by some weird interaction between flyspell + 'visual line-numbers and the latter being set with setq-default as opposed with setq. If I used setq the problem was disappearing.

This is the minimum init code necessary:

(setq-default display-line-numbers 'visual)
(package-initialize)
(use-package ispell)
(use-package company)

(setq ispell-program-name "hunspell")
(ispell-set-spellchecker-params) ;; ispell initialization, a mandatory call

With that if I did the necessary windows configuration

(not that it really matters, but for posterity)

(split-window-right)
(split-window-right)
(split-window-vertically)
(balance-windows)
(other-window 2)

…and then enabled company-mode, flyspell-mode and went to certain point in text and started typing, the displacement have been happening.

[dgutov]

I tried to attach the 1.txt file that is part of the steps, and then my browser got closed due to qutebrowser/qutebrowser#8143 and then subsequently my system crashed and I lost everything: the steps, the file… Wow…

Aw man, that's a nasty combination. Could it be that whatever bug is in the kernel, also triggered the browser crash? If there is some correlation between their occurrences, I'd look in that direction.

Thanks for the steps. Without a test file, it seems hard to reproduce anyway, right?

[Hi-Angel]

Could it be that whatever bug is in the kernel, also triggered the browser crash? If there is some correlation between their occurrences, I'd look in that direction.

Nah, I've seen both happen independently of each other, it's just I got bad luck… On the bright side though, I finally managed to create the report to the kernel. I couldn't before because it's a laptop with NVidia and sometimes I need to load the module which would make kernel "tainted" aka unsuitable for reporting. It's the first time I had it without any 3rd-party modules, so finally reported.

Thanks for the steps. Without a test file, it seems hard to reproduce anyway, right?

Yeah, sure. Just kind of hoped it might ring some bell, but yeah…

[dgutov]

But I've noticed something upon re-watching the video: there is a frame with the popup rendered.

It seems to indicate that for some reason the popup has been placed one line higher than it should be. Possibly because of the special characters inside the file..?

You could try debugging company-pseudo-tooltip-show (either with edebug, or just with message calls), to see which row it's called to render at, and which (possibly incorrect) position move-to-window-line ends up moving to.

[dgutov]

And the initial calculation of the row to use is in company-pseudo-tooltip-show-at-point, which you could also try to step through (it's a smaller function).

[Hi-Angel]

Cool, thank you! I'll check it out the next time that happens

[dgutov]

It's hard to say how this could be caused by the visual line numbers, or flyspell, but if there is some other bug, I suppose flyspell could have made the situation worse (where the wrong positioning might not have led to such outcome otherwise).