Re: Load large amount of data fast

[Petite Abeille <petite_abeille@...>]

On Oct 17, 2010, at 3:15 AM, Alexander Gladysh wrote:

> I'm trying to load that 3M entries in to Lua table in memory faster
> than I do it now.

Ok, for what's worth, 10 million entries, ~500 MB, Lua 5.2.0 (work4), about 6 minutes and quite a bit of memory (and swapping):

--8<--

--[[
local aFile = io.open( 'TestLoad.txt', 'wb' )

aFile:write( 'return { ', '\n' )

local aCount = 10000000

for anIndex = 1, aCount do
    aFile:write( ( '{ foo = %d; bar = %d; baz = \'%s\' }, ' ):format( anIndex, aCount - anIndex, tostring( aCount + anIndex ) ), '\n' )
end

aFile:write( '}','\n'  )
aFile:close()
]]--

local aFile = io.open( 'TestLoad.txt', 'rb' )

Log( 'Started reading' )
local aContent = assert( aFile:read( '*a' ) )
Log( 'Finished reading' )

Log( 'Started load' )
local aChunk = assert( load( aContent ) )
Log( 'Finished load' )

Log( 'Started call' )
local aList = assert( aChunk() )
Log( 'Finish call' )

Log( 'Started count' )
print( #aList )
Log( 'Finish count' )

-->8--

% lua -v TestLoad.lua
Lua 5.2.0 (work4)  Copyright (C) 1994-2010 Lua.org, PUC-Rio
2010-10-17 03:32:35 Started reading
2010-10-17 03:32:38 Finished reading
2010-10-17 03:32:38 Started load
2010-10-17 03:36:07 Finished load
2010-10-17 03:36:10 Started call
2010-10-17 03:38:26 Finish call
2010-10-17 03:38:26 Started count
10000000
2010-10-17 03:38:26 Finish count


> Other ways of solving my original task are out of the scope of this
> question. :-)

Fair enough. Good luck though :)