Re: stepping old data in generational GC

[dormando <dormando@...>]

> On Sep 15, 2023, at 12:11 AM, Francisco Olarte <folarte@peoplecall.com> wrote:
> 
> Hi:
> 
> On Fri, 15 Sept 2023 at 03:35, dormando <dormando@rydia.net> wrote:
> ...
>> - Rarely the base objects are updated. I need the __gc() call to run on
>> the old base objects soon after they are replaced to free resources.
> 
> There is your problem, NEED COLLECTION (__gc). GC is good for
> tracked-memory-only resources, not that good for others ( including
> objects which hold a lot of memory unbeknownst to the GC ). You may
> solve this with GC tunning, but I've had some similar problems, in Lua
> and other GC languages and I have ended up having to manually manage
> them ( I normally have an idempotent close() method, influence by Java
> naming, which I call on __gc()just in case, and now on __close() too,
> as you say your objects are ocasionally updated hence neading
> "destruction", you will probably be better trying to encapsulate their
> managemente there.

Thanks!

To clear up a little: I did check the ML history and saw people with a pattern like:
- alloc userdata (8 bytes), which holds a malloc'ed item (8 megabytes) and the GC can't see the malloc so it doesn't run often enough.

...but that's not really happening here. The objects are all just 60 bytes without external memory. The GC is firing plenty, but in GCGEN mode it never visits the older data at all, as it can take a very very long time to alloc enough "old" objects to cause a major collection.

I'm in this pattern taking advantage of lua's memory management to cope with changing the base objects while "requests" are in flight with the old base objects. I can't easily predict when they're not in use anymore.

In some testing I did after my last post I found setting the GCINC params to "199, 200, 13" (default-1, default, default) is able to keep up... but now I'm repeatedly visiting a lot of objects that don't change in searching for all of the recently made garbage. I've also tried to get __close() to do what I want but without luck yet.

In some idealistic scenario you could either manually step or once-per-minor-young-cull step through specifically old objects. You can switch back to incremental temporarily but I can't predict _when_ the old objects are freeable.

-Dormando