Re: Change in GC behaviour -- Ever-increasing memory usage

[Roberto Ierusalimschy <roberto@...>]

> I came across a supposed leak[1,2] with some C code. I think I distilled
> the behaviour that causes problems into the attached Lua program and
> would now like to get some input on "which code is to blame".
> 
> I noticed that the problem occurs with Lua 5.3, but does not occur with
> Lua 5.1 or 5.2. Thus, I could do a Git bisect based on the GitHub mirror
> of the Lua source code[3]. The result is that commit [4] introduced the
> problem.
> 
> My main question now is: What to do about this?
> 
> 
> Some more information about the program: The library in question
> provides bindings to GObject-based C libraries. To support callbacks, it
> has to luaL_ref() the callback function and it will then luaL_unref()
> later from the __gc metamethod of some of the involved object.
> 
> This is just what the attached program does: In a tight loop, it creates
> a table, luaL_ref()s it and arranges for a call to luaL_unref() later.
> Apparently the code can now create references faster than they are released.

Lua controls de pace of its GC like this: during each cycle, it tries to
estimate how much memory it is actually using. Then, after it completes
the cycle, it waits for its memory use to grow by some percentage
(typically 100%) before starting a new cycle.

Because the program uses finalizers to release most of its memory, it
needs two GC cycles to collect that memory. For objects with finalizers,
Lua knows about that, and therefore these objects do not enter in
that previous computation (how much memory is actually in use).
However, Lua knows nothing about references. For Lua, all objects
in the 'register' table are non garbage. So, by the end of a GC cycle,
it looks like it is using all that memory, so it wait a little more
for the next cycle. With this waiting, more objects will be in the
registry by the end of the next cycle, and there is a vicious cycle.

In my tests, Lua 5.1 presented the same problem. I am not sure why
Lua 5.2 does not.

-- Roberto