Object Benchmark Tests

wiki

The Code

-- Benchmarking support.
do
  local function runbenchmark(name, code, count, ob)
    local f = loadstring([[
        local count,ob = ...
        local clock = os.clock
        local start = clock()
        for i=1,count do ]] .. code .. [[ end
        return clock() - start
    ]])
    io.write(f(count, ob), "\t", name, "\n")    
  end

  local nameof = {}
  local codeof = {}
  local tests  = {}
  function addbenchmark(name, code, ob)
    nameof[ob] = name
    codeof[ob] = code
    tests[#tests+1] = ob
  end
  function runbenchmarks(count)
    for _,ob in ipairs(tests) do
      runbenchmark(nameof[ob], codeof[ob], count, ob)
    end
  end
end

function makeob1()
  local self = {data = 0}
  function self:test()  self.data = self.data + 1  end
  return self
end
addbenchmark("Standard (solid)", "ob:test()", makeob1())

local ob2mt = {}
ob2mt.__index = ob2mt
function ob2mt:test()  self.data = self.data + 1  end
function makeob2()
  return setmetatable({data = 0}, ob2mt)
end
addbenchmark("Standard (metatable)", "ob:test()", makeob2())

function makeob3() 
  local self = {data = 0};
  function self.test()  self.data = self.data + 1 end
  return self
end
addbenchmark("Object using closures (PiL 16.4)", "ob.test()", makeob3())

function makeob4()
  local public = {}
  local data = 0
  function public.test()  data = data + 1 end
  function public.getdata()  return data end
  function public.setdata(d)  data = d end
  return public
end
addbenchmark("Object using closures (noself)", "ob.test()", makeob4())

addbenchmark("Direct Access", "ob.data = ob.data + 1", makeob1())

addbenchmark("Local Variable", "ob = ob + 1", 0)


runbenchmarks(select(1,...) or 100000000)

The Results (current version)

| 2010-01-15 MikePall
| Intel Core2 Duo E8400 3.00GHz
| Linux x86, GCC 4.3.3 (-O2 -fomit-frame-pointer for both Lua and LuaJIT)
| Lua 5.1.4 (lua objbench.lua) vs.
| LuaJIT 1.1.5 (luajit -O objbench.lua) vs.
| LuaJIT 2.0.0-beta2 (lj2 objbench.lua)

Lua     LJ1    LJ2
-----------------------------------------------------
14.08	2.16   0.1  Standard (solid)
14.92	4.62   0.1  Standard (metatable)
14.28	2.66   0.1  Object using closures (PiL 16.4)
 9.14	1.68   0.1  Object using closures (noself)
 7.30   1.10   0.1  Direct Access
 1.22	0.34   0.1  Local Variable

| 2008-04-16 MikePall
| Intel Core2 Duo E6420 2.13GHz
| Linux x86, GCC 4.1.2 (-O3 -fomit-frame-pointer for both Lua and LuaJIT)
| Lua 5.1.3 (lua objbench.lua) vs. LuaJIT 1.1.4 (luajit -O objbench.lua)

17.93   3.11   Standard (solid)
20.36   6.25   Standard (metatable)
19.34   3.73   Object using closures (PiL 16.4)
12.76   2.23   Object using closures (noself)
 7.53   1.55   Direct Access
 2.59   0.47   Local Variable

| 2008-04-17 LeonardoMaciel
| Intel Core 2 Duo T7200 2.00 GHz
| WinXP, MSVC9 (VS 2008)
| Lua 5.1.3 (using luavs.bat) vs. LuaJIT 1.1.4 (using luavs.bat)
| [NOTE: this measurement probably didn't use luajit -O]

17.52  10.78  Standard (solid)
19.74  12.55  Standard (metatable)
18.31  10.88  Object using closures (PiL 16.4)
14.20   5.09  Object using closures (noself)
 7.99   5.94  Direct Access
 1.70   0.41  Local Variable

| 2008-04-19 DougCurrie
| Pentium M 2.00 GHz
| WinXP, GCC 3.4.5 (mingw special)
| Lua 5.1.3 (from wxLua build) vs. LuaJIT 1.1.4 (luajit -O objbench.lua)

28.68   4.76  Standard (solid)
31.23   9.49  Standard (metatable)
30.32   5.38  Object using closures (PiL 16.4)
19.60   3.27  Object using closures (noself)
12.47   2.26  Direct Access
 2.72   0.51  Local Variable

<--- Add your results here if you like.
<--- Please indicate the date and your name or wiki page.
<--- Add your CPU, OS, compiler and Lua and/or LuaJIT version.

The Results (old version)

Windows XP SP2  Intel P4 1.8a

Standard (solid)  Time: 34
Standard (metatable)  Time: 37
Object using closures (PiL 16.4)  Time: 40
Object using closures (noself)  Time: 29
Direct Access  Time: 19

Windows XP x64 SP1  AMD Athlon64 3500+ (64-bit Lua)

Standard (solid)  Time: 22
Standard (metatable)  Time: 23
Object using closures (PiL 16.4)  Time: 25
Object using closures (noself)  Time: 18
Direct Access  Time: 11

Windows Vista Ultimate(32bit), AMD Athlon X2 4200+ (Vanilla Lua 5.1.1 / LuaJIT 1.1.2)

Standard (solid)  Time: 26 / 11
Standard (metatable)  Time: 29 / 15
Object using closures (PiL 16.4)  Time: 30 / 12
Object using closures (noself)  Time: 20 / 6
Direct Access  Time: 13 / 8

Linux Xubuntu Feisty Fawn(32bit), Intel P4 Celeron 2.4ghz (Vanilla Lua 5.1.1)

Standard (solid)  Time: 34
Standard (metatable)  Time: 38
Object using closures (PiL 16.4)  Time: 40
Object using closures (noself)  Time: 25
Direct Access  Time: 20

Windows XP Prof. SP2, Intel PIII 500mhz (Vanilla Lua 5.1.1 / LuaJIT 1.1.2)

Standard (solid)  Time: 133 / 60
Standard (metatable)  Time: 146 / 76
Object using closures (PiL 16.4)  Time: 147 / 64
Object using closures (noself)  Time: 99 / 32
Direct Access  Time: 67 / 36

Conclusion

Direct Access to a local copy of a table is by far the fastest way to do things (as expected). This serves as a reference to the rest of them.

The noself method is the second fastest here. It relies strictly on closures and locals defined within that closure, returning nothing but a public interface. If the tests are modified to perform ten additions per method call, then it can exceed the speed of Direct Access since this reduces the overhead of the function call. The noself and PiL 16.4 methods are the only two that have any support for privately scoped variables.

Every other method is slower than standard direct access. The metatable method gums up the works even more with the extra lookup required but still has comparable speed to the PiL 16.4 method.

The method mentioned in PiL 16.4 adds a private scope advantage and uses a closure to store self, but this isn't nearly as fast as the optimization done in Lua for using a proper 'self'.

One last thing to note: An early version of the benchmarking code did not take a local reference to the objects (but indexed the global table). This affected everything by a few seconds, most of all the direct access--the time doubled.

Hope this helps -- AA

See Also

• OptimisationTips