> The another solution is a reentrant/recursive GC, as far as I know.
I still fail to see your point. The GC in Lua already is
"reentrant/recursive": you can freely call a garbage collection inside
a __gc metamethod. (Although, as others pointed out, this is smelly.)
It disables debug hooks.
And I do not see how this feature would solve the problem of
non-terminating finalizers set by bad actors.
It would no longer disable debug hooks.
Moreover, finalizers set inside a sandbox can get to run outside the
sandbox, so finalizers set by bad actors seem to have many other
problems besides non termination. The best solution here is, as
Pierre pointed out, not to allow setting __gc inside sandboxes.
(In general, both 'getmetatable' and 'setmetatable' should not be
available in sandboxes.)
How would they run outside the sandbox?