Re: help wanted diagnosing out-of-memory error

["Robert G. Jakabosky" <bobby@...>]

On Thursday 02, Matthew Wild wrote:
> On Thu, Apr 2, 2009 at 8:08 PM, Robert G. Jakabosky
>
> <bobby@sharedrealm.com> wrote:
> > On Thursday 02, Norman Ramsey wrote:
> >> For those interested I had several tables with 13M entries each, with
> >> string keys totalling about 330MB of data.  Apparently the overheads
> >> were enough to consume about 3.3GB of RAM, which pushes up against
> >> fundamental limits on a 32-bit machine.
> >
> > These two patches could help you, since they improve the memory usage
> > during resizing of the internal string hashtable and the hashpart of Lua
> > tables:
> > http://lua-users.org/files/wiki_insecure/users/RobertGabrielJakabosky/str
> >ingtable_resize-5.1.3.patch
> > http://lua-users.org/files/wiki_insecure/users/RobertGabrielJakabosky/has
> >hpart_resize-5.1.3.patch
> >
> > Internalized strings are keepted in a global hashtable, when the Lua core
> > needs to grow that hashtable it doesn't just resize the memory block used
> > by the old hashtable, it allocates a new hashtable and them copies the
> > strings from the old hashtable.  The first patch above changes the Lua
> > core to resize that hashtable in-place.  The second patch does that same
> > for the hashpart of all Lua tables.
>
> Out of curiosity, how often would this happen? Each time a new string
> is encountered? or less frequently? If I don't have 100 million
> strings (!) then would this patch still measurably benefit me?

The internalized string hashtable is always a power of 2 in size with a 
minimal size of MINSTRTABSIZE (which is set to 32 in llimits.h).  When the 
unique string count reaches the current size it is resized to the next power 
of 2.  The GC will shrink the string table when the string count is lower 
then (current size) /4.  That /4 should maybe be added to luaconf.h so people 
who need to run Lua on very low memory devices can get better memory usage 
(at the cost of more table resizes).

If you don't have a large number of unique strings, then the only benefit 
would be if you need to restrict Lua to a low memory footprint.

> > Those patches are included in the Emergency Garbage Collector (EGC) patch
> > I wrote:
> > http://lua-users.org/files/wiki_insecure/power_patches/5.1/emergency_gc-5
> >.1.4-r2.patch
>
> I believe 5.2 has the emergency garbage collector... does it also have
> the 2 patches above?

It looks like the 5.2 will include an EGC:
http://lua-users.org/lists/lua-l/2008-02/msg00720.html

I don't know if the Lua core developers will include the 2 in-place resize 
patches.  It would be nice if they did.  I keepted them separate since the 
string table in-place resize patch is very small compared to the in-place 
resize patch for the hashpart of Lua tables.

Also the EGC idea was not mine, I have only been maintaining the EGC patch for 
Lua 5.1.x.  The above e-mail pre-dates my work on the EGC patch, so 5.2 will 
mostlikely not be using my EGC.

-- 
Robert G. Jakabosky