bug of GC "step"

[Makoto Hamanaka <naraxx800@...>]

Hello,

In the cource of writing a book about using scripting language in game 
development (in japan), I may found some bugs in Lua GC "step" functionality.

 (1) GC "step" sometimes eat much CPU cycle than exepected, or even worse, 
     infinitely loops under some situation.
 (2) GC "step" doesn't return "end of cycle" flag sometimes, when it should.

I'm experiencing with Lua5.1.3, WindowsXP SP2.

- steps to reproduce problem (1)

Just run this script.

collectgarbage("setpause", 100) -- small value
collectgarbage("setstepmul", 2000) -- large value
collectgarbage("stop")

print("start GC step")
collectgarbage("step",0) ------- Lua freezes here!
print("finish GC step")


- steps to reproduce problem (2)

1. Run this script with C debugger,

collectgarbage("setpause", 100) -- small value
collectgarbage("setstepmul", 1000) -- large value
collectgarbage("stop")
for i=0, 1000000 do
  collectgarbage("step",0)
end

2. while in "for" part, set breakpoint here, in singlestep()

      else {
        g->gcstate = GCSpause;  /* end collection */
        g->gcdept = 0;
        return 0;
      }

so it will break at that point,

3. then step out to luaC_step() and also to lua_gc().

This while loop in lua_gc() sometimes just keep looping, 
and the "end of cycle" flag is ignored.

      while (g->GCthreshold <= g->totalbytes)
        luaC_step(L); <-------- it may loop without checking "end of cycle"



To fix these problems, I modified the code to just check the "end of cycle" after every call to luaC_step().
I'm not so familiar with Lua VM code, so please review it.

in lua_gc() : 

    case LUA_GCSTEP: {
      lu_mem a = (cast(lu_mem, data) << 10);
      if (a <= g->totalbytes)
        g->GCthreshold = g->totalbytes - a;
      else
        g->GCthreshold = 0;
#if 0
      /* ----- original Lua code ----- */
      while (g->GCthreshold <= g->totalbytes)
        luaC_step(L);
      if (g->gcstate == GCSpause)  /* end of cycle? */
        res = 1;  /* signal it */
#else
      /* ----- modified code ----- */
      while (g->GCthreshold <= g->totalbytes) {
        luaC_step(L);
        if (g->gcstate == GCSpause) {  /* end of cycle? */
          res = 1;  /* signal it */
          break;
        }
      }
#endif
      break;
    }


Thanks and Regards,
Makoto Hamanaka <naraxx800@yahoo.co.jp>