Re: Load large amount of data fast

[Mike Pall <mikelu-1010@...>]

Sean Conner wrote:
> [spc]lucy:/tmp/foo>time lua data1
> 
> real    0m32.067s
> user    0m29.807s
> sys     0m1.747s
> [spc]lucy:/tmp/foo>time luajit data1
> 
> real    3m21.539s
> user    0m25.572s
> sys     0m18.276s
> 
>   Yes, LuaJIT did worse than Lua.  Some more data:
>
> Computer:	Intel(R) Pentium(R) D  CPU 2.66GHz (dual core)
> 		1G RAM (low I know)
> 		Linux 2.6.9

A large difference between the real time and the user time
generally indicates that your process is waiting for the kernel.
In this case your PC is running out of memory and probably
swapping to disk a lot.

Your test needs around 1.2GB of memory. LuaJIT may trigger the GC
at different points in time than Lua. If a GC happens to be
started after the point where you run out of physical memory,
you'll get a huge slowdown. So this is basically just an
unfortunate coincidence.

The test runs fine on my machine and is around 20% faster with
LuaJIT than with Lua (well, there's not much to optimize here).
Increase the test size and you'll run into problems with Lua, too.
Reduce the test size and it'll run fine on your machine with both
Lua and LuaJIT.

> file:write("table.insert(res,{ foo = 1 , bar = 2 , baz = 'text" .. i .. "'})\n")

BTW: file:write() takes multiple parameters. The concatenation is
unnecessary (and slow). Better replace the '..' with ','.

--Mike