Re: Interesting interaction between debug hooks, the garbage collector, and "open" VM instructions

[Patrick Donnelly <batrick.donnelly@...>]

On Sat, Feb 7, 2009 at 5:11 AM, Roberto Ierusalimschy
<roberto@inf.puc-rio.br> wrote:
>> In extremely convoluted situations, if a debug hook is called
>> immediately before an open VM instruction, and then proceeds to do a
>> garbage collection, then the behaviour of the program can change. This
>> is shown in the example code below, which will print "fail" if the
>> debug hook is set, and "pass" if it isn't:
>>
>> [...]
>>
>>   // Change a the {...} construct to use the un-used b through to f stack slots.
>> [...]
>
> Can you explain the problem? (I mean, why your example has that
> behavior.)  May this problem happen with "original" Lua opcodes?
>
> -- Roberto
>

This is caused by the top of the stack being reduced below the top
slot for the running function. The string and table will both be
collected (during the hook) by the GC before the SETLIST operation is
executed. What follows is a patch which corrects the problem.

batrick@waterdeep:/src/lua-5.1.4/src$ diff -Naur /home/batrick/tmp/lvm.c~ lvm.c
--- /home/batrick/tmp/lvm.c~    2009-02-08 01:35:04.000000000 -0700
+++ lvm.c       2009-02-08 01:35:04.000000000 -0700
@@ -745,7 +745,8 @@
           Protect(luaD_checkstack(L, n));
           ra = RA(i);  /* previous call may change the stack */
           b = n;
-          L->top = ra + n;
+          if (L->top < ra+n)
+            L->top = ra + n;
         }
         for (j = 0; j < b; j++) {
           if (j < n) {

I don't believe there is any way to corrupt Lua using this.

-- 
-Patrick Donnelly

"One of the lessons of history is that nothing is often a good thing
to do and always a clever thing to say."

-Will Durant