Re: why benchmarks need salt

[Enrico Colombini <erix@...>]

On 20-Jul-15 21:12, Elias Barrionovo wrote:
> Soon:
>
> "Abstract: In this paper we present a new optimization technique,
> where we use an artificial neural network together with other
> heuristics to find a suitable name for (...)"

While you are busy writing that paper ;-) I ran a few tests under Linux 
with a freshly compiled 5.3.1, randomizing the executable name length 
from 1 to 10 characters.

I raised 'rep' to 10000 in Roberto's code, to minimize os.clock() 
granularity influence.

On 64-bit xubuntu (i7), len=1..10, 100 repetitions:
  min=1.074592, max=1.223682, avg=1.08894445

On 32-bit xubuntu (Pentium 4 HT), len=1..10, 10 repetitions:
  min=5.215278, max=9.036871 (!?!), avg=5.8223072

Strangely enough, the above pathological result (that happened in two 
instances with length=6, towards the end of the 10 runs) did not happen 
again with 100 repetitions. Now on the 32-bit machine I got:
  min=5.214088, max=5.517392, avg=5.2637858

(quick & dirty code attached - of course it could add its own problems)

Two notes:
- could swapping and/or other processes be involved?
  (e.g. something making heavy use of the system heap)
- my test program runs from the same intepreter that it calls
  to run 'temp'.

Now I *have* to turn off the old Pentium: 120 W in the middle of a heat 
wave...

--
  Enrico

local rep = string.rep
local rnd = math.random
local floor = math.floor

local exebase = 'a' -- Lua executable, no extension
local exeext = '' -- (Linux version)
local luaexe = exebase .. exeext

local nrep = tonumber(arg[1])
local maxlen = tonumber(arg[2])

if (nrep == nil) or (maxlen == nil)
   or (nrep <= 0) or (maxlen <= #luaexe) then
    print('usage: lua test.lua <nrep> <maxlen>')
    print('       (nrep > 0, maxlen > ' .. #luaexe .. ')')
    os.exit(1)
end

-- table of execution times
local t = {}

math.randomseed(os.clock())
for i = 1, nrep do
    -- make a copy of Lua executable with random name length
    local len = maxlen - #luaexe
    assert(maxlen >= 0)
    exename = exebase .. rep('x', floor(rnd(len))) .. exeext
    os.execute('cp ' .. luaexe .. ' ' .. exename)

    -- run the copy, get the result
    local f = io.popen('./' .. exename .. ' temp')
    local r = f:read()
    f:close()

    -- remove the copy
    os.remove(exename)

    -- add result to table (len includes extension, if any)
    t[#t + 1] = {len=#exename, time=r}
    
    -- show it is doing something
    io.write('time=', r, ', len=', #exename, '\n')
end
io.write('\n')

-- process results
local maxtime = t[1].time
local mintime = maxtime
local totaltime = 0

for i = 1, #t do
    local entry = t[i]
    local len = entry.len
    local time = entry.time
    if time > maxtime then 
        maxtime = time
    elseif time < mintime then 
        mintime = time
    end
    
    totaltime = totaltime + time
end

local avg = totaltime / #t
io.write('min=', mintime, ', max=', maxtime, ', avg=', avg, '\n')