Fix: multiple tree-sitter highlighting bugs

The precedence order for neovim (and now the tree-sitter CLI as well it
seems?) is to prefer the last match found rather than the first. This
moves ad over to use the same behaviour but I still need to update the
existing highlight queries as there are now several places where the
ordering is incorrect if we want to maintain the existing highlighting.

This should make it easier to port over existing queries from neovim
but I'm currently guarding and rejecting queries that use unknown custom
predicates (such as neovim's #lua-match) as the resulting highlights are
all messed up if we don't correctly implement the same logic used by
whichever editor or tool the query was taken from.

The tree-sitter tests can now run in CI as I've updated the setup of
parsers and TS state to support using Rust crates rather than dynamic
linking.

Cargo.toml

@@ -54,3 +54,5 @@ toml = "0.8.19"
 [dev-dependencies]
 simple_test_case = "1.2.0"
 criterion = "0.5"
+tree-sitter-python = "0.23.6"
+tree-sitter-rust = "0.23.2"

data/config.toml

@@ -72,6 +72,7 @@ function = { fg = "#957FB8" }
 keyword = { fg = "#Bf616A" }
 module = { fg = "#2D4F67" }
 number = { fg = "#D27E99" }
+operator = { fg = "#E6C384" }
 punctuation = { fg = "#9CABCA" }
 string = { fg = "#61DCA5" }
 type = { fg = "#7E9CD8" }

src/buffer/internal.rs

@@ -414,7 +414,7 @@ impl GapBuffer {
     }
 
     /// Convert a byte index to a character index
-    pub fn byte_to_char(&self, byte_idx: usize) -> usize {
+    pub fn raw_byte_to_char(&self, byte_idx: usize) -> usize {
         self.chars_in_raw_range(0, byte_idx)
     }
 
@@ -739,7 +739,7 @@ impl GapBuffer {
     }
 
     #[inline]
-    fn byte_to_raw_byte(&self, byte: usize) -> usize {
+    pub fn byte_to_raw_byte(&self, byte: usize) -> usize {
         if byte > self.gap_start {
             byte + self.gap()
         } else {

src/lsp/capabilities.rs

@@ -86,7 +86,7 @@ impl PositionEncoding {
             Self::Utf8 => {
                 let line_start = b.txt.line_to_char(pos.line as usize);
                 let byte_idx = b.txt.char_to_byte(line_start + pos.character as usize);
-                let col = b.txt.byte_to_char(byte_idx);
+                let col = b.txt.raw_byte_to_char(byte_idx);
 
                 (pos.line as usize, col)
             }