lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


I have a very large table that is split into two subtables, 
one with 3.9 million entries and one with 15 million entries.
Each entry has a string key and a numeric count; the total
process image is about 3.3GB.   I'm trying to write the data
to a file and am coming up with an 'out of memory' error:

  > wr('./trec-counts.lua', t)
  not enough memory

Here's the function

  function wr(f, t)
    local close = false
    if type(f) == 'string' then f = assert(io.open(f, 'w')); close = true end
    f:write('return {\n')
    for k, class in pairs(t) do
      f:write('  ', k, ' = {\n')
      for key, count in pairs(class) do
        f:write(string.format('    [%q] = %d,\n', key, count))
      end
      f:write('  },\n')
    end
    f:write('}\n')
    if close then f:close() end
  end

The function runs for some time and produces a 770MB output file
before stopping with the 'out of memory' error.  I'm not sure how to
proceed as I don't see a memory leak.

I've temporarily added 4GB of extra swap space and am trying again.

Does anyone have any other suggestions?  See a memory leak?


Norman