Re: help wanted diagnosing out-of-memory error

[Matthew Wild <mwild1@...>]

On Thu, Apr 2, 2009 at 8:08 PM, Robert G. Jakabosky
<bobby@sharedrealm.com> wrote:
> On Thursday 02, Norman Ramsey wrote:
>> For those interested I had several tables with 13M entries each, with
>> string keys totalling about 330MB of data.  Apparently the overheads
>> were enough to consume about 3.3GB of RAM, which pushes up against
>> fundamental limits on a 32-bit machine.
>
> These two patches could help you, since they improve the memory usage during
> resizing of the internal string hashtable and the hashpart of Lua tables:
> http://lua-users.org/files/wiki_insecure/users/RobertGabrielJakabosky/stringtable_resize-5.1.3.patch
> http://lua-users.org/files/wiki_insecure/users/RobertGabrielJakabosky/hashpart_resize-5.1.3.patch
>
> Internalized strings are keepted in a global hashtable, when the Lua core
> needs to grow that hashtable it doesn't just resize the memory block used by
> the old hashtable, it allocates a new hashtable and them copies the strings
> from the old hashtable.  The first patch above changes the Lua core to resize
> that hashtable in-place.  The second patch does that same for the hashpart of
> all Lua tables.
>

Out of curiosity, how often would this happen? Each time a new string
is encountered? or less frequently? If I don't have 100 million
strings (!) then would this patch still measurably benefit me?

> Those patches are included in the Emergency Garbage Collector (EGC) patch I
> wrote:
> http://lua-users.org/files/wiki_insecure/power_patches/5.1/emergency_gc-5.1.4-r2.patch
>

I believe 5.2 has the emergency garbage collector... does it also have
the 2 patches above?

Matthew.