Re: Using lua_yield with common C functions

[Sean Conner <sean@...>]

It was thus said that the Great Peter LaDow once stated:
> I've spend the last several hours pouring through the archives trying
> to understand the interaction between lua_resume, lua_yield, and
> coroutines.  The problem I'm having is that when I have multiple
> scripts running in separate threads, only the last created thread
> seems to run correctly.
> 
> This is my initial foray into coroutines.  Initially I used pthreads,
> but I thought the overhead of threads wasn't worth it, so I'm trying
> out this method.  This seems far more straightforward, cleaner, and
> more efficient.  But I'm not getting something with using lua_resume
> and lua_yield.  I'd prefer not to use Coco, and I'm not even sure it
> is necessary in this case.  I'm sure I'm misunderstanding something.

  I recently finished a project using Lua threads (extending a network
daemon---sockets handled by C, logic by Lua) here's what works for me, and
I'll concentrate primarily on the Lua side of things (and skimp on the error
handling as it would only cloud the issue at hand).

  First, I initialize Lua:

	g_L = lua_open();

	/*-------------------------------------
	; do I need the GCSTOP/GCRESTART?  I saw
	; the lua interpreter do it, so I was
	; unsure enough to include it here
	;---------------------------------------*/

	lua_gc(g_L,LUA_GCSTOP,0); 
	luaL_loadlibs(g_L);
	lua_gc(g_L,LUA_GCRESTART,0);

	/*---------------------------------------
	; create a table to store references to
	; userdata and coroutines
	;------------------------------------*/

	lua_newtable(g_L);	
	lua_setglobal(g_L,"sockets");

	/*------------------------------------------
	; load the script and make sure it's okay
	; error handling of lua_pcall() omitted ...
	;-------------------------------------------*/

  	luaL_loadfile(g_L,scriptfile);
	rc = lua_pcall(g_L,0,LUA_MULTRET,0);

  The easy stuff is out of the way.  The program then waits for a network
connection, and upon a connection, some userdata is created, a Lua thread is
created, and we call a predefined Lua function.  When that returns, we
handle the result, and go back into the main loop of the program (written in
C).

	void newluathread(void)
	{
	  lua_State *new;
	  UserData  *n;

	  /*---------------------------------------------
	  ; after each command, I show the stack of
	  ; each state at the end of the command.  Top
	  ; of stack is to the right
	  ;--------------------------------------------*/

	  lua_getglobal(g_L,"sockets");	/* g_L = ( sockets ) */
	  new = lua_newthread(g_L);	/* g_L = ( sockets new ) */
	  lua_getglobal(new,"main");	/* function we'll be calling */
					/* new = ( main )

	  n = lua_newuserdata(new,sizeof(UserData)); /* ( main n ) */

	  lua_getglobal(new,"sockets");	 /* new = (main n sockets) */
	  lua_pushvalue(new,-2);	 /* new = (main n sockets n ) */
	  n->userref = luaL_ref(new,-2); /* new = (main n sockets ) */
	  lua_pop(new,1);		 /* new = (main n ) */
	
	  n->coref = luaL_ref(g_L,-2);	/* g_L = ( sockets ) */
	  lua_pop(g_L,1);		/* g_L = ( ) */

	  /*--------------------------------------------------	
	  ; at this point, we save n on a list, but that's
	  ; not important to this dicussion, but just note
	  ; that we have a reference to it.
	  ;-------------------------------------------------*/	
	
	  n->L = new;
	  rc   = lua_resume(new,1);	/* start main(userdata) */
	
	  if (rc == 0)
	    thread_exit(n);	/* we'll get to this later */
	  else if (rc != LUA_YIELD)
	    /* error - handle as needed */
	  else
	    thread_schedule(n);	/* we'll get to this later */
	}

  Note that I save a reference to the userdata and coroutine in the global
table "sockets".  This is to keep the garbage collection from reclaiming the
space before we're done.  

  Now, I too, have a few routines that call lua_yield() (in my case,
whenever you read from a socket, or maybe even write to a socket, if
everything can't be written in one shot).  Those functions tend to look a
bit like:

	int function(lua_State *L)
	{
	  UserData *n;
	
	  luaL_checkudata(L,1);
	  n = lua_touserdata(L,1);

	  /*-----------------------------
	  ; do whatever processing
	  ;------------------------------*/

	  if (takes_too_long)
	  {
	    n->state = WAITING_FOR_WIDGET;
	    return lua_yield(L,0);
	  }
	  else
 	    return items_to_return;
	}

  Oh, I should mention that each C function that does yield sets a state in
the userdata structure indicating what it's waiting for (so I know what to
return when whatever what the thread is waiting for is available).  Now,
onto the two functions thread_schedule() and thread_exit():

	void thread_schedule(UserData *n)
	{
	  switch(n->state)
	  {
	    case WAITING_FOR_WIDGET:
	    case WAITING_FOR_GADGET:
	    case WAITING_FOR_GODOT:
	         /* put n into a waiting queue */
	         break;
	    case RUNNING:
		 /* put n into a run queue */
		 break;
	  }
	}

	void thread_exit(UserData *n)
	{
	  /* remove n from any queue it might be in */
	  /* free any resources allocated from C    */
	  /* and stored in n as well ...	    */

	  /*--------------------------------------------
	  ; remove the references to the userdata and
	  ; coroutine from our global table
	  ;--------------------------------------------*/

	  lua_getglobal(g_L,"sockets");
	  luaL_unref(g_L,-1,n->sockref);
	  luaL_unref(g_L,-1,n->coref);
	  lua_pop(g_L,1);

	  /*-----------------------------------------
	  ; purely optional, but I do this since the
	  ; program as a whole is a long lived daemon
	  ; and I might as well do it now.
	  ;------------------------------------------*/

	  lua_gc(g_L,LUA_GCCOLLECT,0);
	}

  The only thing left to do is show the main loop of the program, which I'll
kind of summarize here:

	while(1)
	{
	  if (run_queue_length > 0)
	    timeout = 0; /* return if no events */
	  else
	    timeout = INDEFINITE;

	  /*---------------------------------------------------------------
	  ; check for events, and handle them (say, WIDGET #2 is now ready,
	  ; Godot returns, etc).  The code in wait_for_various_events()
	  ; will find the appropriate userdata, and set the state
	  ; appropriately (or even create a new thread).  There's lots of
	  ; handwaving here because I don't want to overwhelm the example
	  ; with a ton of extraneous, non-Lua based details.
	  ;--------------------------------------------------------------*/

	  wait_for_various_events(timeout);

	  /*---------------------------------------------------
	  ; We now might have threads scheduled to run ... so
	  ; run the first one on the queue ... 
	  ;---------------------------------------------------*/

	  if (run_queue_length > 0)
	  {
	    n = remove_first_item(run_queue);
	    rc = lua_resume(n->L,lua_gettop(n->L));
	  
	    if (rc == 0)
	      thread_exit(n);
	    else if (rc != LUA_YIELD)
	      /* error */
	    else
	      thread_schedule(n);
	  }
	}

  The way I've set this up is that the Lua code can call coroutine.yield()
and it will still work properly (only that thread will then get rescheduled
to run---I use a simple round-robin scheduler).  I should also note that the
C code is single-threaded and all the "multi-threading" is purely Lua
coroutines.  Also note than when you call lua_newthread() you basically get
a copy of the global state, so any global variable changed in one Lua thread
is instantly visible in any other Lua thread, so tread carefully.

  I've found that what I've done works.  It may not be the best way to do
it, but it works.  Hope this helps some.

  -spc