Re: how to load a very large table with more than 2^18 literal strings?

[Jerome Vuarand <jerome.vuarand@...>]

2009/8/6 Peter Cawley <lua@corsix.org>:
> On Wed, Aug 5, 2009 at 2:36 AM, Norman Ramsey<nr@cs.tufts.edu> wrote:
>> I would like to write large Lua tables out to disk, possibly nested, but
>> with no sharing or cycles.  Does anybody have any suggestions how to
>> achieve this without running afoul of the dreaded 'constant table overflow'?
>
> 2^18 is the limit per function, not per file. Of course, if you file
> is a single function (i.e. the unnamed top-level chunk), then 2^18
> becomes the file limit.
>
> You can transform code like this:
>
> X = {
>  a = "b",
>  c = "d",
>  e = {
>    f = "g",
>    h = "i",
>  },
>  j = "k",
> }
>
> Into something like this:
>
> X = {
>  a = "b",
>  c = "d",
>  e = (function() return {
>    f = "g",
>    h = "i",
>  } end)(),
>  j = "k",
> }
>
> Thus moving some constants out of the top-level and into subfunctions.

This can be done easily with a couple gsub in the file content:

function dofile(filename, ...)
  return assert(loadstring(assert(assert(io.open(filename,
"rb")):read("*a")):gsub("{", "(function() return {"):gsub("}", "}
end)()")))(...)
end

Loading will be a bit slower, but this should remove the limit.