Plugins, Busted and separate Neovim processes?

[HiPhish]

Hello fellow Neovimmers,

I am trying to use Busted to test my plugins and I want to run each test or at least each describe context, or even each test file in a separate Neovim instance. The reason is that I don't want side effects from one test bleeding into another and affecting the result. The level of separation is up for debate, but there should be isolation at least on some level.

I have set up Busted to use Neovim as its Lua interpreter. Here are all the pieces:

• First let's create a "shim" script which will act as the Lua interpreter and forward arguments to Neovim with the -l option. Save this as test/nvim-shim (or whatever, it doesn't really matter) and make the file executable.

  #!/bin/sh
  nvim -u NONE -l $@

• Now let's define our Busted configuration in .busted

  return {
  	_all = {
  		coverage = false,
  		lpath = 'lua/?.lua;lua/?/init.lua',
  		lua = './test/nvim-shim'
  	},
  	default = {
  		verbose = true,
  	},
  	unit = {
  		ROOT = {'./test/unit/'},
  	}
  }

  This defines the task unit which will run unit tests in test/unit.

• Now we can invoke Busted (installed from Luarocks). I use a makefile to automate the invocation.

  check:
  	eval $$(luarocks path --lua-version 5.1 --bin) && busted --run unit

Nothing surprising here. Let's take it for a spin. Create two test files:

• test/unit/first_spec.lua

  describe('A dumb test', function()
  	it('always works', function()
  		assert.truthy 'Yes'
  	end)
  	it('always fails', function()
  		assert.falsy(false)
  	end)
  	it('can define global Vim variables', function ()
  		vim.g.derp = 'xxx'
  		assert.are_equal('xxx', vim.g.derp)
  	end)
  end)
  
  describe('Another test', function()
  	it('can access global variables from previous test', function ()
  		assert.are_equal('xxx', vim.g.derp)
  	end)
  end)

  All these tests pass. This mean the g:derp defined in the third test persists across tests and contexts

• test/unit/second_spec.lua

  describe('The stack data structure', function()
  	describe('The empty stack', function()
  		it('Can be instantiated', function()
  		end)
  	end)
  end)

  This test will also succeed even though the g:derp variable was defined in a separate file.

This is bad because it means Busted cannot be used for anything except the simplest of unit tests. Is there some way to get isolation working?

[HiPhish]

I have found a solution, and it's actually quite obvious in hindsight. The key insight is that I do not need to run the tests in separate processes, but that I need to run the code I want to test in separate processes. Basically:

• Start an embedded headless Neovim process inside the test
• Manipulate and probe that process through MsgPack RPC from within the test
• If I want some value to compare I have to read it out of the process into a variable inside my test
• When done terminate the process

Here is an example:

local jobopts = {
	rpc = true,
	width = 80,
	height = 24,
}

describe('Process isolation', function()
	-- This will be the channel for each test
	local nvim

	before_each(function()
		-- Start the remote Neovim process.  The `--embed` flag lets us control
		-- Neovim through RPC, the `--headless` flag tells it not to wait for a
		-- UI to attach and start loading plugins and configuration immediately
		nvim = vim.fn.jobstart({'nvim', '--embed', '--headless'}, jobopts)
	end)

	after_each(function()
		vim.fn.jobstop(nvim)
	end)

	it('Defines a global variable', function()
		rpcrequest(nvim, 'nvim_set_var', 'foo', 'bar')
		local foo = rpcrequest(nvim, 'nvim_get_var', 'foo')
		assert.is.equal('bar', foo)
	end)

	it('Does not have access to previous editor state', function()
		assert.has_error(function() rpcrequest(nvim, 'nvim_get_var', 'foo') end)
	end)
end)

And that's it, we have process isolation. It isn't pretty, but it gets the job done. We can define some helper functions to reduce the boilerplate of rpcrequest, or we could come up with some sort of DSL, but that's a problem for another day. I should also mention that busted supports Moonscript out of the box, maybe scripts written in Moonscript would be easier on the eye. But then we lose the support of our awesome Lua language server.

[HiPhish]

To wrap up the topic, let's talk about configuration isolation. I do not want my own configuration and plugins to interfere with testing. On the other hand, I might want some plugins present during testing, such as nvim-treesitter for installing parsers. First we need to adjust the shim:

#!/bin/sh

# Set custom XDG base directory paths to isolate the test Neovim from the
# user's own configuration and data files.
export XDG_CONFIG_HOME='test/xdg/config/'
export XDG_STATE_HOME='test/xdg/local/state/'
export XDG_DATA_HOME='test/xdg/local/share/'

# We have to explicitly enable plugins, see ':h -l'
nvim --cmd 'set loadplugins' -l $@

This is yet another reason why the XDG Base Directory Specification is far superior to dotfiles (which are actually a bug). I have set aside the directory test/xdg/ for these paths; the environment variables then instruct Neovim to use them instead of the default directories. And if our sub-process inherits the parent's environment (which it does by default) it will use these directories as well.

Let's now talk about plugins. At minimum we need the current directory as a plugin, so let's add it to the standard path for plugins as a symlink.

ln -s ../../../../../../../../ test/xdg/local/share/nvim/site/pack/testing/my-plugin

Here testing and my-plugin are arbitrary names. We create a relative symlink that goes all the way up to the root of the plugin. This symlink needs to be commited to the repo and will stay in there. I had to go quite a number of levels up, but if you have a different directory structure you will have a different amount of ...

Next let's add nvim-treesitter as a plugin. I am going to use Git submodules, that way we can track exactly which version we use for testing.

git submodule add https://github.com/nvim-treesitter/nvim-treesitter test/xdg/local/share/nvim/site/pack/testing/start/nvim-treesitter

Note that we place the submodule next to the other plugin where it can be found automatically.

With this we have all pieces in place for reliable and reproducible plugin tests. Run them locally, run them remotely, run them in a CI pipeline, whatever. I hope this helps save time for whoever happens to come across it in the future.