Benchmarking is tough. One of the better groups is the EEMBC, which
creates benchmarks for embedded applications. They require reporting
the configuration used. For example, look at the benchmarks for the
-- three sets of results are given (Out of the box, Optimized C (means using non-standard, DSP specific C extensions), and Optimized assembly
-- and look at the difference in the results (19.5 OTB, 379.1 optimzed C, 628.6 assembly).
So it's definitely necessary to compare apples to apples, not to optimized oranges.
Ideally, benchmarks shouldn't be used for bragging and marketing, but to determine if a solution you're considering (e.g. Lua versus a compiled language) is fast enough for your application.
Jerome Vuarand wrote:
2009/6/5 Rob Kendrick <firstname.lastname@example.org>:On Fri, 5 Jun 2009 13:43:27 +0200 steve donovan <email@example.com> wrote:On Fri, Jun 5, 2009 at 1:23 PM, Jerome Vuarand<firstname.lastname@example.org>Given how most of these comparative benchmarks implementation use and abuse of C libraries (often for the reason that they are shipped with the interpreter), I don't think the spirit of that game is to use pure Lua.So what are the rules? You have to use a stock distribution, and whatever C libraries ship with that? So (say) there was a Lua distribution that came with TinyC (name of package escapes me), that would be game as well, with inlined C?The rules are not concrete, but are judged on spirit. You can have common libraries installed on request.But then what is a common library ? Those shipped through LuaRocks? LuaBinaries? LfW? LuaDist? LuaForge? most of these? all of these ? What about those with unusual packaging (e.g. Roberto's struct library which is a single C file without Makefile) ? I think a fair spirit would be to use the same weapons as your opponents. If they use fork, use a fork binding. If they use a super optimized asm library (while not benchmarking asm), use it as well. Etc.