Unit Testing

wiki

Unit testing is about testing your code during development, not in production. Typically you start by preparing the testing environment, writing some code that calls production code and check expected results with actual results.

Lua has several frameworks to do that:

[lunit]
- Fork with Lua 5.2 support: [lunitx]
[Lunity] (similar to lunit, but can only run a single file and doesn't distinguish between errors and test assertions)
[lunatest] (upwardly compatible from lunit, but no dependencies, and also supports randomized testing)
[LuaUnit] (supports Lua 5.1 and 5.2, no dependencies, see below)
[Shake] - source preprocessing using basic assert-like syntax (see below)
[build-test-deploy (BTD)] - Unit testing inspired by jUnit and originally based on LuaUnit
[luaspec][1] - Behavior Driven Development (BDD) test framework
[PenLight pl.test] - PenLightLibraries? has a very small unit testing library, mostly for internal testing. See the Penlight tests [here]
[telescope] - A highly customizable test library for Lua that allows for declarative tests with nested contexts. Uses BDD-style spec names.
[lua-TestMore] - port of the Perl5 module Test::More (Test Anything Protocol).
[busted] - Behavior Driven Development (BDD) unit testing library, with various outputs formats including TAP and focus on elegance.
[gambiarra] - Ultimately minimal unit testing library for Lua 5.1 and Lua 5.2.
[Testy] - Lua script for minimal unit testing on Lua 5.1/5.2/5.3 that collects test functions from local variables.
[Minctest] - Tiny unit testing framework for Lua
[u-test] - complete unit-testing framework for lua 5.1/5.2/5.3, inspired by googletest. Has colourful command line interface.

They have mostly the same feature set, which is:

a special set of assert functions
facilities to group tests into test suites
facilities to detect where test fails

Commercial Lua testing tools are available from Software Verification[2].

Comparison: [3]

Tests of Lua itself (useful for testing LuaImplementations):

[Lua 5.2 test suite]
[Lua 5.1 test suite]
[Lua 5.0 test suite]
[lua-TestMore] contains Lua 5.1 and 5.2 test suites [announcement]

luaunit

I am going to provide an example for luaunit since I am more familiar with it. The example is pretty simple to understand.


-- Some super function to test
function my_super_function( arg1, arg2 ) return arg1 + arg2 end

-- Unit testing starts
require('luaunit')

TestMyStuff = {} --class
    function TestMyStuff:testWithNumbers()
        a = 1
        b = 2
        result = my_super_function( a, b )
        assertEquals( type(result), 'number' )
        assertEquals( result, 3 )
    end

    function TestMyStuff:testWithRealNumbers()
        a = 1.1
        b = 2.2
        result = my_super_function( a, b )
        assertEquals( type(result), 'number' )
        -- I would like the result to be always rounded to an integer
        -- but it won't work with my simple implementation
        -- thus, the test will fail
        assertEquals( result, 3 )
    end

-- class TestMyStuff

LuaUnit:run()

When you run it, you get:

shell $ lua use_luaunit1.lua
Started on 12/20/14 14:37:50
>>>>>>>>> TestMyStuff
>>> TestMyStuff.testWithNumbers
>>> TestMyStuff.testWithRealNumbers
test.lua:24: expected: 3, actual: 3.3
Failed

=========================================================
Failed tests:
-------------
>>> TestMyStuff.testWithRealNumbers failed
test.lua:24: expected: 3, actual: 3.3

Success: 50% - 1 / 2, executed in 0.015 seconds

The testing framework reports the file and line that caused the error, and some additional information.

When your program grows, you need more and more test cases. The next example is slightly more complicated.

-- Some super function to test
function my_super_function( arg1, arg2 ) return arg1 + arg2 end
function my_bad_function( arg1, arg2 ) return arg1 - arg2 end

-- Unit testing starts
require('luaunit')

-- now, we perform all the tests on int in one test class
-- and the tests on float in another one
-- when your test grows, you will have many test classes

TestWithInt = {} --class
    function TestWithInt:setUp() 
        -- this function is run before each test, so that multiple
        -- tests can share initialisations
        self.a = 1
        self.b = 2
    end

    function TestWithInt:tearDown() 
        -- this function is executed after each test
        -- here, we have nothing to do so we could have avoid
        -- declaring it
    end

    function TestWithInt:testSuperFunction()
        result = my_super_function( self.a, self.b )
        assertEquals( type(result), 'number' )
        assertEquals( result, 3 )
    end

    function TestWithInt:testBadFunction()
        result = my_bad_function( self.a, self.b )
        assertEquals( type(result), 'number' )
        assertEquals( result, -1 )
    end

    function TestWithInt:testThatFails()
        -- you can test anything with assertEquals
        assertEquals( self.a, 1 )
        assertEquals( type(self.a), 'number' )
        -- will fail
        assertEquals( 'hop', 'bof' )
    end
-- class TestWithInt

TestWithFloat = {} --class
    function TestWithFloat:setUp() 
        -- this function is run before each test, so that multiple
        -- tests can share initialisations
        self.a = 1.1
        self.b = 2.1
    end

    function TestWithFloat:tearDown() 
        -- this function is executed after each test
        -- here, we have nothing to do so we could have avoid
        -- declaring it
    end

    function TestWithFloat:testSuperFunction()
        result = my_super_function( self.a, self.b )
        assertEquals( type(result), 'number' )
        -- will fail
        assertEquals( result, 3 )
    end

    function TestWithFloat:testBadFunction()
        result = my_bad_function( self.a, self.b )
        assertEquals( type(result), 'number' )
        -- will work, but only by chance :-)
        assertEquals( result, -1 )
    end
-- class TestWithFloat

LuaUnit:run()

Run it:

shell $ lua use_luaunit2.lua
>>>>>> TestWithFloat
>>> TestWithFloat:testSuperFunction
use_luaunit2.lua:66: expected: 3.2, actual: 3
Failed
>>> TestWithFloat:testBadFunction
Ok

>>>>>> TestWithInt
>>> TestWithInt:testSuperFunction
Ok
>>> TestWithInt:testBadFunction
Ok
>>> TestWithInt:testThatFails
use_luaunit2.lua:44: expected: 'hop', actual: 'bof'
Failed

Success : 60% - 3 / 5

You can also run tests individually:

shell $ lua use_luaunit2.lua TestWithInt:testSuperFunction
>>>>>> TestWithInt
>>> TestWithInt:testSuperFunction
Ok

Success : 100% - 1 / 1


shell $ lua use_luaunit2.lua TestWithFloat
>>>>>> TestWithFloat
>>> TestWithFloat:testSuperFunction
use_luaunit2.lua:66: expected: 3.2, actual: 3
Failed
>>> TestWithFloat:testBadFunction
Ok

Success : 50% - 1 / 2

Shake

Shake is a simple and transparent test engine for Lua that assumes that tests only use standard assert and print calls. If you are looking for a xUnit style framework, check lunit and luaunit instead.

Many Lua modules and applications use a simple pattern for internal tests. They have a script (usually called test.lua) that exercises the module API using API calls and assert() calls to verify the results. It is also common practice in those modules to use print() to output information about the test progress and to use Lua comments to inform what is being tested to whoever reads the source. Although widespread this approach may be too crude for those wanting to test more than one module at a time or to have a more detailed view of the results.

Shake assumes that the tests have been implemented for the above scenario but offers a transparent test engine for those wanting the execution of the tests in batch mode or those wanting to get an overview of the various results. The Shake engine can inform not only the number of tests, failures and errors found in a group of test scripts, but it can infer information about the test context using the output and comments associated with the assert() calls.

The main characteristic of Shake is being transparent, which means that the module authors and test writers do not have to be aware that tests are going to be run with Shake. As long as the tests call assert() Shake can obtain quite a lot of information from the source and run time execution. This is done through the pre-processing of the tests source code using Leg (and therefore LPeg) to replace every call to assert() with one that can extract information about the expressions, values and operators involved in the assertion.

Assuming you have a module like LuaFileSystem installed and you go to its /tests directory and run Shake from there, the output would be:

>>>>~/workspace/luafilesystem/tests$ shake
->  test.lua OK!
_________________

Tests: 27
Failures: 0
Errors: 0

On the other hand, if you have a test script like the one below that includes two assertions that are supposed to fail (lines are numbered):

 1	items = 10
 2	-- checks the correct case
 3	assert (items == 10, "this should not fail")
 4
 5	items = 20
 6	-- checks an overflow case
 7	assert (items == 10, "wrong number of items")
 8
 9	print("Verifying the total")
10	items = 10
11	total = 30
12	assert (items == total, "wrong total")

Shake would register the failures but would run the whole test script, reporting at the end:

:~/workspace$ shake
----------------    test.lua failed!   ----------------

-- checks an overflow case
   #7 assert (items == 10, "wrong number of items")
   items -> 20

Verifying the total
   #12 assert (items == total, "wrong total")
   items -> 10
   total -> 30
_________________

Tests: 3
Failures: 2
Errors: 0

Note how much more informative this is when compared to the default output of running the test script with Lua:

:~/workspace$ lua5.1 test.lua
lua5.1: test.lua:7: wrong number of items
stack traceback:
        [C]: in function 'assert'
        test.lua:7: in main chunk
        [C]: ?

Unit Testing

luaunit

Shake

See Also