very deep recursion in 5.1 GC

["Jasmin Patry" <jpatry@...>]

We've encountered a situation in which we get extremely deep recursion
in the Lua 5.1 GC.  The problem occurs when collection userdata objects
with __gc metamethods definied:

* singlestep calls GCTM
* GCTM calls luaD_call
* luaD_call calls luaC_checkGC
* luaC_checkGC calls singlestep
...ad infinitum (almost)

I've fixed this by disabling the garbage collector during the call to
GCTM, by applying this patch to singlestep() in lgc.c:

@@ -594,7 +594,10 @@
     }
     case GCSfinalize: {
       if (g->tmudata) {
+        lu_mem thresh = g->GCthreshold;
+        g->GCthreshold = MAXLMEM;
         GCTM(L);
+        g->GCthreshold = thresh;
         lim -= GCFINALIZECOST;
       }

This seems to fix the problem.  If this is an incorrect fix, or if there
is a better fix, please let me know.

Thanks,
Jasmin