Re: Lua and light threads

["Sebby " <beast@...>]

Howdy...

> 1) it is possible (the way Bret did years ago) to unwind the stack 
(i.e.:
> not rely on C recursive calls) at least for pure Lua-to-Lua function
> invocation, but it would mean a lot of work and a lot of changes to 
the
[...]

I'v worked on trying to get that approach working last weekend. If 
you want to go all the way through with it, there is lots of code to 
change, which makes merging with new versions difficult and in 
addition, you risk introducing new bugs to Lua. But if you're willing 
for a halfway implementation it's not too bad. From my tests, i only 
had to change code in the luaD_call function and some in the begining 
of the luaV_execute function and it's CALL/TAILCALL and RETURN/END 
opcodes. Well my thing didn't work totaly right since it failed on 2 
of the lua test files but i think both of them were related to the 
same bug but i wasn't able to put the finger on it yet

> 2) it would be easy to stop the scheduler from intruding in the 
case of a
> Lua->C->Lua sequence of calls, so that's not an issue;

Yeah, that's what i did in my simplified try in making Lua iterative. 
It does restrict the threading however, since a sleep command can be 
refused if it's called within the context of a Lua->C->Lua execution. 
So, in other words, whoever would write scripts would need to know 
about thoes restrictions and understand how the recursion can occur 
(tagmethods, foreach, ...)

> 3) I don't quite see (it's my ignorance on the subject) how 
setjmp/longjmp
> could help in this matter. If anybody wanted to share their 
experience with
> me on the subject, I would be more than grateful.

Ok, well after the discussion on it yesterday, i decided to play with 
the idea a bit to see if it would be doable. I'v tried a limited test 
(starting a task, sleeping it and resuming it right away) and it 
seemed to work fine. So here is the basic idea.

When you start up your thread, you do a setjump to capture the state 
of the machine where you currently are (so you can return to that 
point in your scheduler when the thread sleeps or ends). Then, you 
substitute the system stack with a stack that is part of your thread 
structure (or part of your lua_State) and then call lua to execute 
the thread.

When a sleep (or equivalent) function is called, Then you do another 
setjmp to capture the current state of your thread (and store it in 
your thread structure), do whatever maintnance your need to do on 
your thread to reschedule it and call longjmp for the setjmp you 
called when you started the thread. And the next time you want to 
resume execution of your thread you'd simply call longjmp on the 
state you stored for your thread (well, you'd also have to setjmp 
before to store the state in case of another sleep command).

This technique wouldn't work with serialization as mentioned earlier. 
But it does seem to give simpler code in general and doesn't imply 
any modifications in lua code. Here is my sample code from what i'v 
written yesterday. Keep in mind it's just a test i'v done so some 
stuff is hacked in and it hasn't been stress tested. But i though i'd 
share my results with the mailing list :

// Lua state to show things i'v added to it for my thread test
struct lua_State {
  /* thread-specific state */
  StkId top;  /* first free slot in the stack */
  StkId stack;  /* stack base */
  StkId stack_last;  /* last free slot in the stack */
  int stacksize;
  StkId Cbase;  /* base for current C function */
  struct lua_longjmp *errorJmp;  /* current error recover point */
  char *Mbuffer;  /* global buffer */
  size_t Mbuffsize;  /* size of Mbuffer */

  // SEBBY : Jump buffer for threading and thread local stack
  jmp_buf	LongJump;
  char		TmpBuf[1024*16];
[...]

// Function which is called to parse a lua file and which does 
// my thread test for now
char* gFileName;
lua_State*	gState; 
Bool zslLuaParseFile(ZSLProcess* Process, char* FileName)
{
	int Res;
	lua_State*	State; 
	void*		Stack;

        // My thread management stuff, associates 2 lua_State together
	State = zslBindState(&Process->DummyThread);
      
        // Save the current state so i can return here on a sleep
	if (setjmp(State->LongJump)==0)
	{
                // Change the system stack to the thread local stack
		Stack = &State->TmpBuf[1024*16-1];
		gFileName = FileName;
		gState = State;
		_asm 
		{ 
			mov esp, Stack;
		}

                // Call lua to execute the thread
		Res = lua_dofile(State,FileName)==0;
	}
	else
	{
		// We get here on a sleep, so for now i simply resume
                // the thread
		longjmp(State->LongJump,1);
	}
    
        // Dissociate the 2 thread states
	zslUnBindState(&Process->DummyThread);

	return Res;
}

// My sleep function, note the usage of gotos isn't really necessary
// anymore but i was too lazy to clean up
int zslLua_Sleep (lua_State *L) 
{
  jmp_buf	JumpBuffer;
  jmp_buf	TmpJumpBuffer;

  // Save the current state of the thread so i can resume it later on
  if (setjmp(JumpBuffer)==0)
  {
	  // Copy the buffer so we can return
          // This is just some switcharoo to put the current state
          // in lua state without deleting the state i previously 
          // saved so i can return to the caller of the thread
	  zmemCopy(&TmpJumpBuffer,&L->LongJump,sizeof(jmp_buf));
	  zmemCopy(&L->LongJump,&JumpBuffer,sizeof(jmp_buf));
	
	  goto jump;
  }
  goto end;
jump:
  // Return to the thread's caller
  longjmp(TmpJumpBuffer,1);
end:
  return 0;
}

Like i said, this is just experimental code and is far from clean. 
But i thought it might inspire other people :)


Sebastien St-Laurent
Software Engineer
sebby@z-axis.com