Create cursors by matching text objects

[swarn]

Just an idea I'm kicking around.

Creating cursors based on matching/splitting with regex is great. What about creating cursors based on text objects?

Splitting the following into arguments by vi( and splitting on , is great:

foo(x, y, z)

But what about this?

foo(x, bar(y, z), (a, b))

Text objects are good. Treesitter text objects are super smart. A few uses:

• Selecting all (top-level) arguments in an argument list
• selecting all quoted strings in a text paragraph
• select all sentences in a text paragraph
• selecting all list items in a list

I've got an initial implementation here. It splits a single visual selection. For the example above, it creates three selections:

foo(x, bar(y, z), (a, b))
    -  ---------  ------

It is likely naive and fragile. It relies on the search capability of mini.ai so that e.g. vi" will find quoted text to the right.

Thoughts? Am I missing something obvious?

[jake-stewart]

seems useful. i think it would be best if iw, i" and some others were implemented by hand and then everything else automatically falls back to the idea you've come up with. because iw can select non alphabetic words which is very unlikely what the user wants. i" can select the stuff between pairs of quotes. these would both be fairly easy to add and would be the most used.

[jake-stewart]

playing around with it, i think it can be done with a filter of (char: string) => boolean.

for iw, we define the filter as iskeyword and while iterating over each char, we only query the iw text object once we have found an iskeyword char. this would avoid the whitespace etc.

then for i", we define the filter as char => char == '"'. once we find a text object, we can record its start/end position. trying to match a i" between two strings will end up having the start position the same as a previous end position and so it will not be added.

then for other text objects, the prefilter is just () => true.

does this make sense?

EDIT: to explain better, put your cursor at the start of the line. press l until you reach a keyword character (eg [a-zA-Z0-9_]). only once you reach one, press viw. this defines a match. continue pressing l from the end of the selection.

for i", do the same thing, but only press vi" when you are on a " character. it never selects a bad string region. it may reselect a previous selection but we can detect and skip it.

[jake-stewart]

small feedback, Cursor:feedkeys is fairly expensive: it writes the cursor state (all its registers, marks, position) to vim, feeds the keys & flushes, reads the new cursor state, and updates its extmarks. so it's a good idea to combine Cursor:feedkeys calls when possible.

search_cursor:feedkeys(TERM_CODES.ESC .. "`>")

[swarn]

small feedback, Cursor:feedkeys is fairly expensive

Ah, this is a good tip. I divided it into multiple calls just to visually separate "change to normal mode" vs "move to end," but with as much as I'm creating cursors (simply to act as placeholders), I should aim for efficiency.

As for the filtering, hmm. I agree that the built-in iw and i" text objects are not likely to be useful. For the latter, I use the i" implementation from mini.ai, which is smart about matched pairs. I'm a bit leery of the specialization here:

• I hate to over-anticipate. I can't think of a use for splitting on iw, but that doesn't mean no one wants to grab the words in *+*x*+*x*+*.
• I kind of like the separation of concerns of "defining text objects" from "multicursor." I might create an "identifier" text object outside of this. Or match on [%w_]+.
• I'm not sure about "it uses your text objects, except these two." Messes with the clean abstraction.

I don't know that these vague concerns stand up against "make the plugin more useful, with no dependencies," though. In any case, I mostly want to use it for treesitter-derived text objects — things where regex/patterns are a pain.

Your implementation concept for these two makes sense to me. We are both probably re-implementing echasnovski's work in mini.ai; I haven't looked closely, but he clearly put a lot of thought into searching for text objects not covered by the cursor.

[jake-stewart]

We can figure out if they have the key remapped using maparg('i"', "o"). Then, if it is mapped to iw, we apply the iw special case. If it is mapped to something random, then we use that text object as is.

In the end it should appear like plain text objects are being used to user, but a tiny bit of magic behind the scenes for builtin text objects so they work "correctly".

[jake-stewart]

i am working on this

[jake-stewart]

needs a lot more work but here is my prototype:

local regexes = {
    ["iw"] = "\\k",
    ['i"'] = '"',
    ['ip'] = '^.*\\S',
    -- ["i'"] = "'",
}

local function matchVisualObjects()
    local motion = vim.fn.input("> ")
    if not motion or motion == "" then
        return
    end

    mc.action(function(ctx)
        local mainCursor = ctx:mainCursor()
        local visualStartPos, visualEndPos = mainCursor:getVisual()
        local startPos = mainCursor:mode() == "V"
            and { visualStartPos[1], 1 }
            or visualStartPos
        local INT_MAX = 2147483647
        local endPos = mainCursor:mode() == "V"
            and { visualEndPos[1], INT_MAX }
            or visualEndPos
        local lastPos = { startPos[1], startPos[2] - 1 }
        local lines = mainCursor:getVisualLines()
        local origPos = mainCursor:getPos()
        local found = false

        local mapped = vim.fn.mapcheck(motion)
        if not mapped or mapped == "" then
            mapped = motion
        end
        local regex = regexes[mapped]

        if regex then
            for _, match in ipairs(util.matchlist(lines, regex)) do
                local newPos = {
                    startPos[1] + match.idx,
                    startPos[2] + match.byteidx
                }
                if comparePos(newPos, lastPos) then
                    goto continue
                end
                mainCursor:setPos(newPos)
                mainCursor:setMode("n")
                mainCursor:feedkeys("v" .. motion)
                local vs, ve = mainCursor:getVisual()
                if comparePos(vs, lastPos)
                    or comparePos(ve, lastPos)
                then
                    goto continue
                end
                found = true
                mainCursor:clone()
                lastPos = { ve[1], ve[2] + 1 }
                ::continue::
            end
        else
            local newPos = { startPos[1], startPos[2] }
            while newPos[1] - startPos[1] < #lines do
                mainCursor:setMode("v")
                mainCursor:setVisual(newPos, newPos)
                mainCursor:feedkeys(motion, { remap = true })
                if not mainCursor:inVisualMode() then
                    newPos[2] = newPos[2] + 1
                else
                    local vs, ve = mainCursor:getVisual()
                    if comparePos(endPos, vs) or comparePos(endPos, ve)
                    then
                        break
                    end
                    if comparePos(vs, lastPos) or comparePos(ve, lastPos)
                    then
                        newPos[2] = newPos[2] + 1
                    else
                        found = true
                        mainCursor:clone()
                        lastPos = { ve[1], ve[2] + 1 }
                    end
                end
                if newPos[2] > #lines[newPos[1] - startPos[1] + 1] then
                    newPos[2] = 1
                    newPos[1] = newPos[1] + 1
                end
            end
            if found then
                mainCursor:delete()
            else
                mainCursor
                    :setMode("n")
                    :setPos(origPos)
            end
        end
    end)
end

without mini.ai the motions i" and iw work well. with mini.ai the motions i" and ia work well, but iw isn't happy. i tried playing around with doing '"zy' + motion to set the change registers but mini.ai sets them regardless. i can't think of any other ways to improve it.

the snippet needs a lot of cleaning and needs to properly respect the visual bounds. if i do a visual block selection, i expect only the objects contained in that block to be matched.

play around without mini.ai and use for example ip and iw. they behave a lot better. i think it might be a good idea to always do it this way unless it is an unknown object (like ia), or if the user passes a boolean.

@swarn

[swarn]

Very nice. Thanks for working with me on this. I got very excited when I found that this plugin worked with non-standard objects and motions :).

In the next few days, I will mess around with and without mini.ai. Definitely a goal that this work as well as possible without additional plugins. However, my primary goal is that it work with any crazy user-defined text objects.

I see you have replaced a bunch of my uses of extra-cursor-and-g`< by directly using the positions. That makes a lot of sense to me; in general I prefer programming via api instead of scripting inputs, because it feels less fragile. Vim doesn't always make that easy.

Part of the reason I did it that way is that the neovim api is annoying with its variety of position indexing; then multicursor must necessarily add its own conventions. It'd be easier if multicursor used named indexing instead of tuples. Before:

local startPos, endPos = cursor:getVisual()
local newPos = vim.deepcopy(startPos)
newPos[2] = newPos[2] + 1

After:

local range = cursor:getVisual()
local newPos = vim.deepcopy(range.start)
newPos.line = newPos.line + 1

On the topic of bounds, three things. First, the extra-cursor-and-g`< trick, while not ideal, does handle both visual and visual-line modes without needing mainCursor:mode() == "V" checks. Perhaps the API could hide this so that start is start, regardless of mode?

Second, I hadn't thought at all about visual-block selections, that's a good point. That makes things trickier.

Third, I was pretty sloppy thinking about bounds in any case. Here's an example using vit:

           selection
           ---------------------
<foo>****</foo><bar>****</bar><baz>****</baz>
     ----           ----
     result         result

So that's:

1. get objects selectable from the range of the current selection
2. if the first object is outside the range, keep it
3. after the first object, exclude objects outside the range

Instead, it should probably be consistent. The three boundary conditions I can think of are "selectable from inside range," "overlaps range," and "is contained in range." I think I prefer the first, which would look like:

           selection
           ---------------------
<foo>****</foo><bar>****</bar><baz>****</baz>
     ----           ----           ----
     result         result         result

At the moment, that doesn't work with my implementation and default mini.ai behavior, which seeks ahead, leading to this:

           selection
           ---------------------
<foo>****</foo><bar>****</bar>     <baz>****</baz>
     ----           ----                ----
     result         result              result

In my opinion, that's a good reason to disable the default searching behavior of mini.ai, which has other methods to opt-in to search.

The reality is that for any i motion, you probably want objects where the corresponding a motion would overlap the initial selection. But that sounds ugly to implement. So I vote for "selectable from within the range." I think that also simplifies visual-block mode, later.

It's also the intention to select maximal objects, so it should select this for vi(

(  (  )  )((  )(  ))
 --------  --------

and not this:

(  (  )  )((  )(  ))
 --------   --  --

Right now, the maximal behavior is happy result of scanning left-to-right and skipping to the end of found objects. I can't think of any examples where you'd find a larger object on the right than on the left; can you?

[jake-stewart]

I got very excited when I found that this plugin worked with non-standard objects and motions :)

Thanks. I don't find multicursor plugins useful when they don't feel like real vim.

It'd be easier if multicursor used named indexing instead of tuples.

I think this can be done without breaking changes. I would return a { line: int, col: int, offset: int } object with a metatable with __index and __newindex metamethods to handle integer keys. I don't even think this would have a performance impact since metamethods are only called for keys which don't exist. This would need a bit of extra thought for cases like table unpacking but I think it's a good idea.

EDIT: It might be a better idea return the [int, int, int] and have the metamethods for line, col, offset keys to ensure everything works. I could imagine someone doing #pos for whatever reason and luajit does not support overloading the # operator. Why would they ever do this? Maybe something like map(pos, function(n) return n + 1 end). Alternatively we can possibly break their config or release a 2.0 which I don't like since majority of people will be left behind.

Perhaps the API could hide this so that start is start, regardless of mode?

Yes, I tried changing it to be that way but something else broke. I plan to have it that way but need enough time to test everything.

On the topic of bounds

I think only objects completely contained within the visual selection should be included. A crude example would be visually selecting the line } foo { bar } baz {, where the braces at the start/end are massive blocks. If we select with i}, we definitely don't want to select the top and bottom brace blocks.

You mentioned selecting maximal objects. I think we should match from top to bottom and matches contained within objects will be skipped since they come after their parent, as you pointed out.

[swarn]

EDIT: It might be a better idea return the [int, int, int]

This makes sense to me. I'm a frequent user of Python's namedtuple for coordinates in other domains. Having both named and indexed fields is handy. It sounds like luajit only supports that in one way, native tuples (more or less) with named metamethods. Works for me. I feel like the overhead of metamethods is insignificant, especially for this.

I think only objects completely contained within the visual selection should be included.

You're right, that makes the most sense. It prevents some astonishing behaviors.

[swarn]

To avoid duplication of effort: I'm thinking of starting a little testing framework for this. I'm tired of reloading and doing the same motions. I like to visually see the test conditions; maybe something where a test case looks like this (where <leader>j is mapped to matchCursors):

foo(a, b(c, d), (f, g))
<   v-----------------
>   v  v------  v-----
= <leader>jia<cr>

I'm not decided on that syntax or anything; just want to see if you're interested or already have something cooking.

[jake-stewart]

Progress is slow for this one since I keep trying different ideas to make it work seamlessly. I managed to use "zy from normal mode and measured the yanked text to tell whether it was successful. Works for text objects like i" but other objects just yank a single char which is not ideal.

I think we may just have to have an API to help define text objects. All it really needs is the motion (w for iw and aw) and a regex (\k).

My progress so far:

--- @param a SimplePos
--- @param b SimplePos
--- @return boolean
local function comparePos(a, b)
    if a[1] == b[1] then
        return a[2] < b[2]
    end
    return a[1] < b[1]
end

local function identity(x)
    return x
end

--- @class TextObjectInfo
--- @field regex string
--- @field motion fun(char: string): string
--- @field type "i" | "a"

--- @type {[string]: TextObjectInfo}
local TEXT_OBJECT_LOOKUP = {}

--- @param key string
--- @param regex string
--- @param motion? string | fun(match: string): string
local function addTextObject(key, regex, motion)
    local motionCallback
    if type(motion) == "string" then
        motionCallback = function() return motion end
    elseif motion then
        motionCallback = motion
    else
        motionCallback = function()
            return key
        end
    end
    for _, type in ipairs({"i", "a"}) do
        TEXT_OBJECT_LOOKUP[type .. key] = {
            type = type,
            regex = regex,
            motion = motionCallback
        }
    end
end

addTextObject("w", "\\k")
addTextObject("W", "\\S")
addTextObject("p", "\\S")
addTextObject("'", "'")
addTextObject('"', '"')
addTextObject('`', '`')
addTextObject("t", "<")
addTextObject("s", "\\S")
addTextObject("(", "(")
addTextObject(")", ")")
addTextObject("{", "{")
addTextObject("}", "}")
addTextObject("[", "[")
addTextObject("]", "]")
addTextObject("<", "<")
addTextObject(">", ">")
addTextObject("q", "[\"'`]", identity)
addTextObject("b", "(", "(")
addTextObject("B", "{", "{")

--- @param vs SimplePos
--- @param startPos SimplePos
--- @param fullLines string[]
--- @return boolean
local function skipLeadingWhitespace(vs, startPos, fullLines)
    while true do
        local line = fullLines[vs[1] - startPos[1] + 1]
        if not line then
            return false
        end
        while true do
            local char = line:sub(vs[2], vs[2])
            local isWhitespace = char == " " or char == "\t"
            if char == "" then
                break
            elseif not isWhitespace then
                return true
            else
                vs[2] = vs[2] + 1
            end
        end
        vs[1] = vs[1] + 1
        vs[2] = 1
    end
end

function examples.textObjects()
    --- @type string | nil
    local motion = vim.fn.input("> ")
    if motion == "" then
        motion = nil
    end
    if not motion then
        return
    end

    --- @type string | nil
    local mapped = vim.fn.mapcheck(motion, "o")
    if mapped == "" then
        mapped = nil
    end

    local info = (mapped and TEXT_OBJECT_LOOKUP[mapped]) or TEXT_OBJECT_LOOKUP[motion]

    if not info and not mapped then
        util.echoerr(
            "multicursor.nvim: unrecognized text object '"
            .. motion .. "'")
        return
    end

    mc.action(function(ctx)
        local mainCursor = ctx:mainCursor()
        local visualStartPos, visualEndPos = mainCursor:getVisual()
        local startPos = mainCursor:mode() == "V"
            and { visualStartPos[1], 1 }
            or visualStartPos
        local endPos = mainCursor:mode() == "V"
            and { visualEndPos[1], INT_MAX }
            or visualEndPos
        local lastPos = { startPos[1], startPos[2] - 1 }
        local lines = mainCursor:getVisualLines()
        local fullLines = mainCursor:getFullVisualLines()
        local origPos = mainCursor:getPos()
        local found = false

        if info then
            for _, match in ipairs(util.matchlist(lines, info.regex)) do
                local newPos = {
                    startPos[1] + match.idx,
                    startPos[2] + match.byteidx
                }
                if comparePos(newPos, lastPos) then
                    goto continue
                end
                mainCursor:setPos(newPos)
                mainCursor:setMode("n")
                mainCursor:feedkeys("v" .. info.type .. info.motion(match.text))
                local vs, ve = mainCursor:getVisual()

                if not mainCursor:inVisualMode()
                    or comparePos(ve, lastPos)
                    or comparePos(endPos, ve)
                    or not skipLeadingWhitespace(vs, startPos, fullLines)
                    or comparePos(vs, lastPos)
                    or comparePos(endPos, vs)
                then
                    goto continue
                end
                found = true
                mainCursor:clone()
                    :setVisual(vs, ve)
                lastPos = { ve[1], ve[2] + 1 }
                ::continue::
            end
        else
            local newPos = { startPos[1], startPos[2] }
            while newPos[1] - startPos[1] < #lines do
                mainCursor:setMode("v")
                mainCursor:setVisual(newPos, newPos)
                mainCursor:feedkeys(motion, { remap = true })
                local vs, ve = mainCursor:getVisual()

                if not mainCursor:inVisualMode()
                    or comparePos(ve, lastPos)
                    or comparePos(endPos, ve)
                    or not skipLeadingWhitespace(vs, startPos, fullLines)
                    or comparePos(vs, lastPos)
                    or comparePos(endPos, vs)
                then
                    newPos[2] = newPos[2] + 1
                else
                    found = true
                    mainCursor:clone()
                    lastPos = { ve[1], ve[2] + 1 }
                    newPos = lastPos
                end
                if newPos[2] > #fullLines[newPos[1] - startPos[1] + 1] then
                    newPos[2] = 1
                    newPos[1] = newPos[1] + 1
                end
            end
        end
        if found then
            mainCursor:delete()
        else
            mainCursor
                :setMode("n")
                :setPos(origPos)
        end
    end)
end

[swarn]

Don't rush on my account; I'm doing alright with my little prototype for now. Also, I'm both using your advice and stealing your ideas from here to improve it.

I will fork the repo, create a branch, and make a PR solely to make discussion easier. Maybe you make a PR too? Easier than pasting code in comments. We can circle around a few ideas for UI and implementation before you pick whatever you like best.

I wonder if you're going down a rabbit hole here. My feeling is that you want this to be 100% agnostic to objects, not be adding more object-related code. If that means that iw doesn't make a lot of sense by default with "match cursors to text objects," so be it.

You could provide some opt-in object mappings like iw that check if a multicursor operation is progress — maybe set/unset a buffer variable during the action — and change behavior accordingly.

[jake-stewart]

I wonder if you're going down a rabbit hole here. My feeling is that you want this to be 100% agnostic to objects, not be adding more object-related code. If that means that iw doesn't make a lot of sense by default with "match cursors to text objects," so be it.

The priority is that the feature works how the user expects. When a user
selects words with iw, they expect words to be selected, not the whitespace
and symbols between. I consider any other behaviour a bug.

Elegance and simplicity come second.

[swarn]

User experience first is a solid philosophy. 👍