lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]

Benchmarking is tough.  One of the better groups is the EEMBC, which creates benchmarks for embedded applications.  They require reporting the configuration used.  For example, look at the benchmarks for the TI320C6416-720 DSP:
-- three sets of results are given (Out of the box, Optimized C (means using non-standard, DSP specific C extensions), and Optimized assembly
-- and look at the difference in the results (19.5 OTB, 379.1 optimzed C,  628.6 assembly).

So it's definitely necessary to compare apples to apples, not to optimized oranges.

Ideally, benchmarks shouldn't be used for bragging and marketing, but to determine if a solution you're considering (e.g. Lua versus a compiled language) is fast enough for your application.

--Tony

Benchmark results:
http://eembc.org/benchmark/reports/benchreport.php?benchmark_seq[]=417&benchmark_seq[]=418&benchmark_seq[]=416&suite=TLC&type=PRO&type_desc=Silicon&member_seq=&archive=&publish=&sort=

Jerome Vuarand wrote:
2009/6/5 Rob Kendrick <lua-l@nun.org.uk>:
  
On Fri, 5 Jun 2009 13:43:27 +0200
steve donovan <steve.j.donovan@gmail.com> wrote:

    
On Fri, Jun 5, 2009 at 1:23 PM, Jerome
Vuarand<jerome.vuarand@gmail.com>
      
Given how most of these comparative benchmarks implementation use
and abuse of C libraries (often for the reason that they are
shipped with the interpreter), I don't think the spirit of that
game is to use pure Lua.
        
So what are the rules? You have to use a stock distribution, and
whatever C libraries ship with that?  So (say) there was a Lua
distribution that came with TinyC (name of package escapes me), that
would be game as well, with inlined C?
      
The rules are not concrete, but are judged on spirit.  You can have
common libraries installed on request.
    
But then what is a common library ? Those shipped through LuaRocks?
LuaBinaries? LfW? LuaDist? LuaForge? most of these? all of these ?
What about those with unusual packaging (e.g. Roberto's struct library
which is a single C file without Makefile) ?

I think a fair spirit would be to use the same weapons as your
opponents. If they use fork, use a fork binding. If they use a super
optimized asm library (while not benchmarking asm), use it as well.
Etc.