Updating describe/it methods

[ktalebian]

I'd like to add describe.skip and describe.only (and to it) as well for my test runners. How can I update/register new executors?

[Tieske]

Looks interesting. Can you explain what the use case is?

[ktalebian]

Motivated by https://mochajs.org/#inclusive-tests. Specially when writing tests, I'd like to be able to have

describe('NewTest', function()
    it.only('run this one alone, and nothing else', function()

    end)
end)

Or if you want to for now skip a test, instead of commenting it out you'd just add it.skip/describe.skip

[Tieske]

I gave it some thought and I don't like it. I see the use case, but I think there is a risk when you use those, that you accidentally forget one of them and then unknowingly skip part of the tests on each test run.

The common way to do this is by using tags:

describe('NewTest', function()
    it('run this one alone, and nothing else #only', function()

    end)
end)

and execute it using busted --tags=only.

For skipping you'd use a #skip tag and do busted --exclude-tags=skip.

The existing approach using tags cannot accidentally be merged and then silently skip tests in CI for example. With your proposed approach you would run that risk.

Not my call in the end. Maybe @ajacksified or @DorianGray have different opinions.

[ktalebian]

I would still like to be able to modify/extend these methods. If I try to do

local busted = require('busted')

busted.describe =  ...

I get Attempt to modify busted from https://github.com/Olivine-Labs/busted/blob/master/busted/init.lua#L131

How can I extend these methods?

[Tieske]

I don't know, probably updating busted itself is easier than extending it at runtime. The code is heavily event based, I find it hard to reason about.

The error is probably generated by some meta table, so maybe inspect that?

[ktalebian]

I got at least half of the functionalities I wanted using

local busted  = require 'busted'

busted.subscribe({'test', 'start'}, function(md)
    if os.getenv('VERBOSE') ~= nil then
        print(string.format('### running test %s ###', md['trace']['message']))
    end
end) 

thanks for the info!

[ktalebian]

@Tieske I have another use case where I do need to extend it. I want to provide a "data provider" support for my tests. So for example, instead of doing this:

describe('myUtil', function()
    it('should test something', function() 
        assert.are.equal('foo1',  myUtil.evaluate('bar1'))
        assert.are.equal('foo2',  myUtil.evaluate('bar2'))
        assert.are.equal('foo3,  myUtil.evaluate('bar13))
    end)
end)

to something like:

describe('myUtil', function()
    local provider = function() 
        return {
            {'foo1', 'bar1'},
            {'foo2', 'bar2'},
            {'foo3', 'bar13'}
        }
    end)

    it('should test something', function(expected, input) 
        assert.are.equal(expected,  myUtil.evaluate(input))
    end, provider)
end)

(Of course, it should probably return a function(done) for async testing too)

[ktalebian]

This is what I have done so far:

_G.dataProvider = function(test, ...)
	local arg = {...}
	local data = {}
	local provider = arg[1]

	if #arg == 1 then
		if type(provider) == 'table' then
			data = provider
		elseif type(provider) == 'function' then
			data = provider()
		else
			error('Unsupported type ' .. type(provider) .. ' provided')
		end
	else
		data = arg
	end

	for _, value in pairs(data) do
		test(table.unpack(value))
	end
end

Then tests look like:

describe('util', function()
    -- Provider as a function
    local provider = function()
        return {
            {'expected-result1', 'arg1a', 'arg1b'},
            {'expected-result2', 'arg2a', 'arg2b'},
            ...
        }
    end
    
    -- Provider as a table
    local provider = {
        {'expected-result1', 'arg1a', 'arg1b'},
        {'expected-result2', 'arg2a', 'arg2b'},
        ...
    }
    
    dataProvider(function(expected, arg1, arg2)
        it('should run some test', function()
            assert.are.equal(expected, util.someMethod(arg1, arg2))
        end)
    end, provider)
    
    -- Or provider inline as multiple arguments
    dataProvider(function(expected, arg1, arg2)
        it('should run some test', function()
            assert.are.equal(expected, util.someMethod(arg1, arg2))
        end)
    end, {'expected', 'arg1', 'arg2'}, {'expected', 'arg3', 'arg4'})
end)

[ktalebian]

Closing this, in fav of #589