Treesitter not updating or refreshing automatically as text changes

[wookayin]

Describe the bug

Treesitter syntax tree does not update or refresh automatically as I update the buffer. Please see the video below.

To Reproduce

This is how I did when generating the video above:

$ nvim -u minimal.vim a.py

minimal.vim:

packadd nvim-treesitter
packadd nvim-treesitter-playground
packadd nvim-gps

source ~/.config/nvim/lua/config/treesitter.lua
set statusline=%{%v:lua.require'nvim-gps'.get_location()%}

where treesitter.lua contains a minimal setup config, nothing special there.

Expected behavior

Treesitter syntax tree (shown from TSPlayground or nvim-gps) should update automatically. It seems that playground does update automatically when I insert new blank lines, but the syntax tree gets out of sync. nvim-gps does not recognize the new context (nested functions) that was typed after opening the buffer.

Output of :checkhealth nvim-treesitter

nvim-treesitter: require("nvim-treesitter.health").check()
========================================================================
## Installation
  - OK: `tree-sitter` found 0.20.4 (parser generator, only needed for :TSInstallFromGrammar)
  - OK: `node` found v17.3.0 (only needed for :TSInstallFromGrammar)
  - OK: `git` executable found.
  - OK: `cc` executable found. Selected from { vim.NIL, "cc", "gcc", "clang", "cl", "zig" }
    Version: Apple clang version 13.0.0 (clang-1300.0.29.30)
  - OK: Neovim was compiled with tree-sitter runtime ABI version 14 (required >=13). Parsers must be compatible with runtime ABI.

## Parser/Features H L F I J
  - julia          ✓ ✓ ✓ ✓ ✓ 
  - java           ✓ ✓ . ✓ ✓ 
  - pioasm         ✓ . . . ✓ 
  - vala           ✓ . . . . 
  - hocon          ✓ . . . ✓ 
  - cmake          ✓ . ✓ . . 
  (..omitted..)
  - python         ✓ ✓ ✓ ✓ ✓ 
  (...)

Output of nvim --version

Version information:

• NVIM v0.7.0-dev+1032-g01e58fb14 (built today)
• tree-sitter -V: tree-sitter 0.20.4 (installed through homebrew)
• MacOS 12.0 (arm64)
• I've run :TSUpdate and all parsers are up-to-date

Additional context

I thought this was a bug of nvim-gps, but it seems that it's the problem of my treesitter itself (ref: SmiteshP/nvim-gps#69) because playground is also not working.

I searched existing issues but haven't found anything relevant (e.g., nvim-treesitter/nvim-treesitter#78, nvim-treesitter/nvim-treesitter#235 are about highlights.)

This happens for all filetypes, not only python.

[theHamsta]

At the moment the updates rely on a tree-sitter highlighter to be active (typically by an active highlight module). Can you check whether you have tree-sitter highlighting enabled?

[wookayin]

Oh... wow, interesting. Enabling the highlight module fixes the problem. Not only playground but also nvim-gps works as expected, doing dynamic updates.

But I can't understand why highlight modules is mandatory. I didn't want to have it turned on because it conflicts with built-in (and custom) vim syntax files and I don't have much fine-grained control over it. Neither this was properly documented somewhere. Is this by design or just a practical limitation? I expect treesitter-based plugins that doesn't necessarily have something to do with highlight would still work nicely with dynamically-updated syntax tree, even if the treesitter-highlight module is disabled.

[clason]

Practical and current limitation: nvim-treesitter evolved from a syntax highlighting alternative. More fine-grained control is definitely a desideratum but requires somewhat extensive changes, not just to nvim-treesitter but also to tree-sitter integration in Neovim core and the legacy syntax engine.

I will add the usual reminder here that tree-sitter support is still very much experimental and won't be stable for quite a while yet!

[wookayin]

So it looks like this is limitation of neovim's lua runtime API vim.treesitter.highlighter (or TSHighlighter) rather than this nvim-treesitter plugin itself as all the core implementation lies there. Am I correct?

As a workaround, would be there any good recommended way for me to disable all the treesitter highlight groups (i.e., I don't want it overriding existing vim syntaxes)? For me, treesitter syntax for some filetypes I'm working on has some issues so I would prefer using legacy syntax files for some of them.

UPDATE: A solution that I came up with to disable all TS- highlights is to simply "clear" the highlight map table:

local hlmap = vim.treesitter.highlighter.hl_map
for k in pairs(hlmap) do
    hlmap[k] = nil
end

UPDATE 2: The above hack will make TS-highlight information (e.g., :TSHighlightCapturesUnderCursor) broken. I just cleared and removed all the TS.. highlight groups (by having a bunch of hi link TSsomething NONE). Maybe we could add an option to disable all highlight default link directives in plugin/nvim-treesitter.vim for the time being.

[theHamsta]

vim.treesitter.highlighter is implemented in core. The parser will not re-parse by it self. Only when told to do so.

TSHighlighter uses nvim_set_decoration_provider to re-parse on buffer redraw.
https://github.com/theHamsta/neovim/blob/10b273a9ebdfb5fe93b601a90b27a1bcaa2a507c/runtime/lua/vim/treesitter/highlighter.lua#L285-L304

You could also trigger the re-parse by an auto-command/timer.

[theHamsta]

This is a bug of nvim-treesitter/playground. It should request a redraw when it does an update.