calling lua_yield in return hook seems buggy

[Thomas Jericke <tjericke@...>]

Hi,
In a previous post I wrote that I am working on a project where I yield 
in count hooks for pre-emptive scheduling. This still works very well.

Currently I am working on a debugger for the same project. So this 
debugger doesn't stop the whole world when a breakpoint is hit but only 
that certain Lua task. So what I do is when I stop one task is to take 
it out of the schedulers list and yield it.

In general this works, I just had once a strange effect:
I implemented the step over functionality in this way

void StepOver() {
    lua_sethook(m_pL, l_stepoverhook, LUA_MASKLINE | LUA_MASKRET | 
LUA_MASKCALL, 0);
    m_uStepOverDepth = 0;

    ResumeTask(); /* Adds this lua task to the scheduler */
}

void l_stepoverhook (lua_State *L, lua_Debug *ar) {
  CINOSMcLua* pMcLua = (CINOSMcLua*) lua_getud(L);
  if(ar->event == LUA_HOOKLINE) {
      StopTask(); /* Removes this lua task from the scheduler */
      lua_sethook(L, NULL,0,0);
      lua_yield(L, 0);
  }
  else if(ar->event == LUA_HOOKCALL) {
    lua_sethook(L, l_stepoverhook, LUA_MASKCALL | LUA_MASKRET,0);
      pMcLua->m_pCurLuaTask->m_uStepOverDepth++;
  }
  else if(ar->event == LUA_HOOKRET) {
      pMcLua->m_pCurLuaTask->m_uStepOverDepth--;
      if(pMcLua->m_pCurLuaTask->m_uStepOverDepth == 0) {
          StopTask(); /* Removes this lua task from the scheduler */
          lua_sethook(L, NULL,0,0);
          lua_yield(L, 0);
      }
  }
 }

Now it seems if the return hook is within a C closure the lua_yield 
function doesn't handle this correctly.
At this line ldo.c:567
 if (isLua(ci)) {  /* inside a hook? */

isLua(ci) will return false, lua_yieldk will then make luaD_throw(L, 
LUA_YIELD);
After that the whole interpreter seems to be in a inconsistent state, as 
for example the value L->allowhook will be '0' even we are outside a hook.
Now I don't ask to get this feature implemented, I just think a check 
for this case should be in place. The simplest solution for now might be 
just a assert:
 if (isLua(ci)) {  /* inside a hook? */
    api_check(L, k == NULL, "hooks cannot continue after yielding");
  }
  else {
    lua_assert(!(ci->callstatus & CIST_HOOKED));  /* must be outside a 
hook <=== ADDED ASSERT*/
    if ((ci->u.c.k = k) != NULL)  /* is there a continuation? */
      ci->u.c.ctx = ctx;  /* save context */
    ci->u.c.extra = savestack(L, ci->func);  /* save current 'func' */
    ci->func = L->top - nresults - 1;  /* protect stack below results */
    luaD_throw(L, LUA_YIELD);
  }
  lua_assert(ci->callstatus & CIST_HOOKED);  /* must be inside a hook */
  lua_unlock(L);


Just to add, I found a solution for my problem. Instead of yielding 
inside the return hook I register a line hook at that place. Then I 
yield from the line hook which will be called later.

Cheers

--
Thomas