Re: Change in GC behaviour -- Ever-increasing memory usage

[Pedro Tammela <pctammela@...>]

> > I came across a supposed leak[1,2] with some C code. I think I distilled
> > the behaviour that causes problems into the attached Lua program and
> > would now like to get some input on "which code is to blame".
> >
> > I noticed that the problem occurs with Lua 5.3, but does not occur with
> > Lua 5.1 or 5.2. Thus, I could do a Git bisect based on the GitHub mirror
> > of the Lua source code[3]. The result is that commit [4] introduced the
> > problem.
> >
> > My main question now is: What to do about this?
> >
> >
> > Some more information about the program: The library in question
> > provides bindings to GObject-based C libraries. To support callbacks, it
> > has to luaL_ref() the callback function and it will then luaL_unref()
> > later from the __gc metamethod of some of the involved object.
> >
> > This is just what the attached program does: In a tight loop, it creates
> > a table, luaL_ref()s it and arranges for a call to luaL_unref() later.
> > Apparently the code can now create references faster than they are released.
>
> Lua controls de pace of its GC like this: during each cycle, it tries to
> estimate how much memory it is actually using. Then, after it completes
> the cycle, it waits for its memory use to grow by some percentage
> (typically 100%) before starting a new cycle.
>
> Because the program uses finalizers to release most of its memory, it
> needs two GC cycles to collect that memory. For objects with finalizers,
> Lua knows about that, and therefore these objects do not enter in
> that previous computation (how much memory is actually in use).
> However, Lua knows nothing about references. For Lua, all objects
> in the 'register' table are non garbage. So, by the end of a GC cycle,
> it looks like it is using all that memory, so it wait a little more
> for the next cycle. With this waiting, more objects will be in the
> registry by the end of the next cycle, and there is a vicious cycle.
>
> In my tests, Lua 5.1 presented the same problem. I am not sure why
> Lua 5.2 does not.
>
> -- Roberto

I believe this issue happens in 'normal' tables as well.

I've managed to reproduce this issue in both Lua 5.3 and Lua 5.4-work2
and could isolate this to a simple C code. The memory allocation
pattern in both version is pretty different, probably thanks to the
new GC in 5.4.

This code [1] will grow the memory steadily and will exhaust the
computer memory given enough iterations.

In my 8GB machine, this happened in 11 seconds when compiled with 5.3.

[2][3] both show the memory allocation pattern using massif from valgrind.

[1] https://pastebin.com/KCsHduKB
[2] https://pastebin.com/p2b6NE0Q
[3] https://pastebin.com/3hEkk2j9