LuaJIT vs MATLAB benchmark [Was: Re: help to (slightly) modify Lua interpreter]

[Leo Razoumov <slonik.az@...>]

On 2009-11-06, Mike Pall <mikelu-0911@mike.de> wrote:
> Please post current benchmark results vs. Matlab, if you have any.

For whatever it is worth below is a comparison of LuaJIT-1 versus
MATLAB-7.8.0 (February 2009 release) for a scalar tight loop
performance.
Comparison is done on 64-bit  4 way Quadcore machine (16 cores total)
running Linux

* 4 x QuadCore Intel(R) Xeon(R) CPU E5530  @ 2.40GHz
* Linux  2.6.18-128.1.14.el5 #1 SMP Wed Jun 17 06:38:05 EDT 2009
x86_64 GNU/Linux
* memory 64GB

* MATLAB Version 7.8.0.347 (R2009a) 64-bit (glnxa64) February 12, 2009

* Lua 5.1.4 (aka LuaJIT-1)

To double check MATLAB results I asked my colleague to run the test on
Windows XP version of MATLAB on Core2Duo laptop @ 2GHz.

The test programs that compute Euler constant are attached to this
message. The results are below. In the case of MATLAB the so called
"Elapsed time" as reported by tic/toc commands is more accurate metric
then output from "time" command. "time" counts Matlab startup time
that can be couple of seconds while MATLAB's "Elapsed time" is pure
script execution time.

* (1) MATLAB (Linux):
  N    = 1E+07
  euler= 0.577216
  Elapsed time is 16.714563 seconds.

  real    0m19.863s
  user    0m17.524s
  sys     0m0.173s

* (2) LuaJIT-1 (Linux)
  N    = 1E+07
  euler= 0.577216

  real    0m0.140s
  user    0m0.139s
  sys     0m0.001s

* (3) MATLAB (Windows XP, different hardware. Core2Duo @ 2GHz)
  N    = 1E+007
  euler= 0.577216
  Elapsed time is 12.677472 seconds.

It seems that MATLAB Windows port is better optimized but it is still
about 100 times slower than LuaJIT-1. The joke is that MATLAB has its
own JIT since version 6.5 and JIT was enabled for these tests.

Now you may ask how come that people put up with such a poor
performance of the interpreter. The answer is multi-fold. First they
buy hardware. 16 core machine can simultaneously run 16 independent
and slow simulations and I have an access to a dozen of those boxes.
Often one can convert loops into vector operations and offload them
from the interpreter to high performance numeric libraries included in
MATLAB and these libraries are good! However, if one has conditionals
inside a loop, vectorization becomes problematic. Also it is
difficult-to-impossible to vectorize event driven simulations.
Additionally, vectorization trades off memory for performance. MATLAB
cannot run on an embedded system but LuaJIT can. As the last resort
one can write critical sections in C/C++ using Matlab's MX interface.
There is also a MATLAB compiler but I have not much experience with
it.

LuaJIT performance together with its ability to nicely interface to
native libraries  can make it a viable contender in the fields of
numeric and scientific computing.

--Leo--

% Benchmark: Euler constant
% Run as: sh$ time matlab -nosplash -nojvm -r bmark_euler

tic
sum= 0;
N= 1E7;
for n=1:1:N
  sum = sum + 1/n;
end
euler= sum - log(N);
fprintf('N    = %G\n', N);
fprintf('euler= %G\n', euler);
toc
quit

%EoF

local N= 1E7
local sum= 0
for n=1,N do
    sum= sum + 1/n
end
local euler= sum - math.log(N)
print(("N    = %G"):format(N))
print(("euler= %G\n"):format(euler))