Re: Crash Analysis: Finalizer Logic in singlestep function can lead to Sandbox Escape Exploit

[Roberto Ierusalimschy <roberto@...>]

> ------------------------------------------------------------------
> 
>   *Part 2. Defining the root cause of the crash.*
> 
>    So, In short, the root cause is...
> 
> 1. singlestep function with case GCScallfin can change the mode of garbage
> collection, because GCTM can recursively call garbage collection logic.
> 
> 2. runtilstate function can internally call singlestep function with case
> GCScallfin.
> 
> 3. Some functions are developed with the assumption that runtilstate cannot
> change the mode of garbage collection, as stepgenfull function is.
> 
> 4. As a result, the functions may run increment mode logic in generational
> mode.
> 
> 5. Such behavior can break linked list in global state, lead to various
> crashes for example SEGV in separatetobefnz function.

Many thanks for the bug report, and special thanks for the throughout
analysis. GC bugs are notoriously hard to debug.

About the fix: we have to think more carefully, but at first sight it
seems that item 3 in your list is the root of the problem. But the
problem is actually deeper than described there:: some functions are
developed with the assumption that runtilstate cannot call the GC
recursively, period: Changing the mode is just one of the possible bad
consequences.  As such, maybe a fix could be adding an extra parameter
to luaC_runtilstate to signal whether it should skip finalizers
altogether.

-- Roberto