Re: Load large amount of data fast

[Sean Conner <sean@...>]

It was thus said that the Great Alexander Gladysh once stated:
> Hi, list!
> 
> Apologies for a lazy question, I have not done my own homework.
> 
> I've got a large file  (3M entries, 250 MB) with data.
> Each entry is one line with a small Lua table:
> 
> { foo = 1; bar = 2; baz = 'text' };
> 
> (Actually, there are two different entry formats, but that does not matter.)
> 
> I need to load this data fast enough. (Faster than several hours that
> my original loader runs on LJ2, and it still had not stopped.)
> 
> So, if you know an implementation than ad-hoc unoptimized one below,
> please share.

  First off, if the file itself is in Lua, why not

	dofile "largedatafile.lua"

  Second, there are indeed limits to what Lua (and LuaJIT) can parse in a
single function.  When I did this (back in November of 2009, check the email
archives [1]) I found I had to break the data up a bit.  It wasn't that big
of a change when generating the data.  While I found that lua could handle
up to 16k constants per function (I think it's a bit higher), LuaJIT choked
on that, so I bumped the limit down to 4k.  I did that to avoid having to
parse the Lua code in Lua---I'm *already* running a Lua interpreter; why
have two?

  Third, you may need to apply a patch [2] to fix a bug I found when loading
up huge Lua scripts that caused excessive load times.

  -spc

[1]	I'd link to the email archive, but that's limited to members and I
	can't quite find my password for it.  Anyway, the thread starts Nov
	10, 2009 and it titled "The Lua interpreter and large scripts".

[2]	http://www.lua.org/bugs.html#5.1.4-6