Re: [BUG] table will be garbage collected multiple times

[Philippe Verdy <verdy_p@...>]

As well:

  setmetatable(o, {__gc =false})

would mean that the object can be finalized immediately: it would make the object explicitly weak

It would still need to be marked: if  it's still reachable, including notably in the context where the previous statemyn is used, where it is still reachable via (o) so it cannot be finalized before (o) gets out of scope and no other references to (o) remains. So if (o) is marked by the mark phase, it cannot be put into the finalization list.

So:

  setmetatable(o, {__gc =false})

would be mostly equivalent to:

  setmetatable(o, {__gc =nil})

or:

  setmetatable(o, {})

or:

  setmetatable(o, nil)

May be we can imagine another use for __gc=false (notably with a generation-based GC: meaning don't finalize in the current generation, but keep the object in the older generation, in which case __gc will be reset automatically to nil, and then the GC running on the older generation will allow finalizing it at this time: when the object will be old)

We could also tweak the value given to __gc (or the value returned by the function) to mean we want the object to be part of specific generations identifiable as any object;

This would be useful for example to create different pools, notably for caches (Lua appliations would be able to manage efficiently their "cache eviction policy", which is something very important to avoid DOS attacks that attempt to clear caches used by concurrent threads, and to avoid time attacks similar to Meltdown, measuring the time to honor requests, which is shorter if an object is still in cache than when it is not because the object has to be reconstructed, meaning that a third party can know if an object was recently used by another thread).

To avoid Meltdown-like attacks, we must be able to restrict the cache eviction by "segregating pools in caches": different pools are allocated for different security contexts or different threads (Meltdown is not affecting just CPUs, it concerns all computing systems that manage caches, notably thoise using the very common LRU eviction policy). The GC in Lua can easily become an easy target of Meltdown and DOS attacks if the Lua-written software is used to service many users on the internet.

Le mar. 20 nov. 2018 à 18:55, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

This proposed behavior would have no impact oin finalizers:

- they can still use getmetatable(o) and see the unmodiied metatable of the object (no need then for the GC to temporarily set clear it or set it to a dummy (different) metatable.

- they can use setmetatable(o,mt) as they want: no quirk needed in the implementation of setmetatable (whic hthen does not need to care about the fact a finalization of the object is pending)

- all is determined ONLY when the finalizer returns.

A simple finalizer like:

  setmetatable(o, {__gc =function() return true end})

or even just:

  setmetatable(o, __gc =true)

(if we also honor a boolean value of __gc as equivalent to a function returning a boolean) would be enough to say that the object "o" must NEVER be finalized. (the alternative using a boolean would avoid the need to perform any costly calls in repeated attempts to finalize the object, such object would not even be in a finalization list, the GC would then automatically consider the object as "marked", and reachable, the object will never be freed, i.e. will remain permanent in memory)

Le mar. 20 nov. 2018 à 18:42, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

So in summary, it is the effectively fact that we call "setmetatable(o,mt)" on the object (o) passed in parameter to the finalizer that effectively changes its state (a bit in the object).

In my opinion, this is not the best way to handle it 

- this requires a specific behavior of setmetatable: it inspects the metatatable to see if there's a __gc function in it, then sets the FINALIZEDBIT if it is so, then returns to the finalizer, which may still change the metatable after this.

- the only safe behavior would be that the finalizer **returns** an effective status. For now finalizer are functions that return nothing (or nil), they could return a non-nil value (probably non-false as well, e.g. true) to indicate its desire to indicate that the FINALIZEDBIT must be CLEARED on the object. A finalizer that does nothing (an empty function) would not return anything, so the GC can still safely look at the metadata at this time, see if it has a __gc function, and if not it will set the FINALIZEDBIT (allowing the object to be swept and freed).

Le lun. 19 nov. 2018 à 19:54, Gé Weijers <ge@weijers.org> a écrit :

You'd think that Lua source code is unavailable if you read this thread. The finalizer implementation is fairly simple to understand:

• if you call 'setmetatable' on an object, and the metatable has a __gc field the object is moved from the 'allgc' list to the 'finobj' list. The FINALIZEDBIT is set in the object's GC state.
  
• when the object becomes unreachable it is moved back to the 'allgc' list, and the FINALIZEDBIT is reset. The finalizer is then called.
  
• when the object becomes unreachable again (i.e. when it's on the 'allgc' list and there are no references to it) it's freed.

If you call 'setmetatable' in the finalizer, and that metatable has a '__gc' field the object is moved back to the 'finobj' list, so the process starts again and the object is never freed.

Observations:

• an object is only finalized once, unless you explicitly call 'setmetatable' again to set the finalizer.
• it's not a bug, the original poster's program explicitly requests that the object be finalized again using 'setmetatable'. The behavior is consistent with the documentation.
• there are other things you can do in the __gc function, like storing a reference to the object somewhere so it stays reachable. The finalizer has run, but the object will have to be kept around anyway because it just became reachable again. Having the finalizer 'decide' what to do is therefor unsafe. Freeing an object whose finalizer has not run yet requires two GC cycles, at the end of the first one the finalizer runs, at the end of the second one the object is freed.
• It's not clear to me what the use case is for changing the metatable in a finalizer to begin with. In the normal case you'd expect the object to be unreachable after the finalizer returns, so it will be freed in the next pass.

"It's not a bug, it's a feature!"

--

Gé