How to deal with massive tables

[Glenn <glenn@...>]

The short short version of my problem is that I have a massive table
with 100000+ entries (where each entry is a small table with five
elements), populated in C code, which has then been serialized from a
Lua script. When I deserialize it I get a 'constant table overflow'
error.  If I cut the number of entries down to around 55000, no error.

Here's the long version of my problem.  I have an embedded database that
I want to extract a bunch of information out of.  The information is
"key" data, used to look up the full record.  Specifically, the data
(scrubbed here to protect the guilty) is in a small table:

{ ["a"] = "blah", ["b"] = "yadda", ["c"] = "foo", ["d"] = 12345 }

There are multiple ways to look up the same account, which is why there
are so many elements to this table.  The keys are strings and the data
are all strings except for one integer.  The table is populated in C code.

Each small table is part of a larger table, which is structured as an array:

bigTable = {
  [1] = {
    { ["a"] = "blah",
      ["b"] = "yadda",
      ["c"] = "foo",
      ["d"] = 1234
  },
  [2] = {
    -- ...
  },
  -- ...
  [100000] = {
    -- ...
  }
}

The entire table is populated in memory, and works just fine.  I then
serialize it to disk using a Lua function similar to the one in PiL for
tables with no cycles.  I make no claims that its perfect (or even
pretty), just that it works for the most part.

function serialize(o, f)
    if type(o) == "number" then
        f:write(o)
    elseif type(o) == "string" then
        f:write(string.format("%q", o))
    elseif type(o) == "table" then
        f:write("{\n")
        for k,v in pairs(o) do
            f:write("  [")
            if type(k) == "number" then
                f:write(k)
            elseif type(k) == "string" then
                f:write(string.format("%q", k))
            else
                -- ? just write it out and see what happens
                f:write(k)
            end
            f:write("] = ")
            serialize(v, f)
            f:write(",\n")
        end
        f:write("}\n");
    else
        error("cannot serialize a " .. type(o))
    end
end

I tried adding another "level" to the structure to see if it was a
per-table problem, like so:

bigTable = {
  [1] = {
    [1] = {
      -- ... as before
    },
    -- ...
    [10000] = {
      -- ...
    }
  },
  -- ...
  [10] = {
    [1] = {
      -- ... as before
    },
    -- ...
    [10000] = {
      -- ...
    }
  }
}

As I'm sure is no surprise to Lua experts, that didn't work.  I get the
same error.  Interestingly, when I cut down the total number of entries
in the serialized file I can get to around 70000+ entries with the
tiered approach over the original "flat" array.

So the question is, is there anything I can do to get my 100000+ entries
into a serialized table that I can later suck into another script?  Or
am I just doomed?

Thanks,
--
Glenn McAllister     <glenn@somanetworks.com>      +1 416 348 1594
SOMA Networks, Inc.  http://www.somanetworks.com/  +1 416 977 1414

  Asking a writer what he thinks about criticism is like asking a
  lamppost what it feels about dogs.
                                                    - John Osborne