[Date Prev][Date Next][Thread Prev][Thread Next]
[Date Index]
[Thread Index]
- Subject: help wanted diagnosing out-of-memory error
- From: nr@... (Norman Ramsey)
- Date: Wed, 1 Apr 2009 19:00:25 -0400 (EDT)
I have a very large table that is split into two subtables,
one with 3.9 million entries and one with 15 million entries.
Each entry has a string key and a numeric count; the total
process image is about 3.3GB. I'm trying to write the data
to a file and am coming up with an 'out of memory' error:
> wr('./trec-counts.lua', t)
not enough memory
Here's the function
function wr(f, t)
local close = false
if type(f) == 'string' then f = assert(io.open(f, 'w')); close = true end
f:write('return {\n')
for k, class in pairs(t) do
f:write(' ', k, ' = {\n')
for key, count in pairs(class) do
f:write(string.format(' [%q] = %d,\n', key, count))
end
f:write(' },\n')
end
f:write('}\n')
if close then f:close() end
end
The function runs for some time and produces a 770MB output file
before stopping with the 'out of memory' error. I'm not sure how to
proceed as I don't see a memory leak.
I've temporarily added 4GB of extra swap space and am trying again.
Does anyone have any other suggestions? See a memory leak?
Norman