Re: parallelizing thousands of tests

[Jerome Vuarand <jerome.vuarand@...>]

2011/4/16 Norman Ramsey <nr@cs.tufts.edu>:
> I've used Lua to create some infrastructure that runs unit tests.
> In a typical run we run a couple hundred tests against maybe 30
> programs, so a total of around 6000 tests.  The Lua code has almost no
> overhead; we're mostly launching a lot of processes.  Result: I can't
> keep even one CPU core busy.  We have an embarrassment of parallelism,
> but I can't figure out which of the many packages I ought to try.
>
> Here is the loop I wish to parallelize:
>
>  for tid, tests in pairs(tests) do  --- outer loop
>    local results = { }
>    matrix[tid] = results
>    for i, test in pairs(tests) do   --- middle loop
>      if test.valid then
>        results[i] = { }
>        local results = results[i]
>        for sid, bin in pairs(binaries) do -- inner loop
>          local outcome, witness = run_test(test, bin)
>          results[sid] = { outcome = outcome, witness = witness }
>        end
>      end
>    end
>  end
>
> I'd be perfectly happy to try running the outer and middle loops
> sequentially but to spawn a separate thread to run the inner loop in
> parallel.   That would result in 20 to 30 threads in flight at once,
> which is probably enough to keep 8 cores busy without an embarrassment
> of parallelism.
>
> Can anyone suggest which of the many parallel packages for Lua might
> be most suitable for parallelizing this loop?

Maybe you can simply try to start the programs with io.popen. That way
the Lua process won't be blocked, and you can run several in parallel.

Also in the past I've been successful at running such a pool of
parallel processes with a simple binding to POSIX APIs. I've only used
fork, execve, and wait. I can share the code if you want, but it's
veru specific to the task I had to perform.