Thread Safety in Lua, part 1

[Rici Lake <lua@...>]

(This is sort of a free-floating post, since the context comes from 
other messages.)

The question is: Can you run different (Lua) threads in different OS 
threads? And the answer is a qualified "yes", but with a great deal of 
caution. Here are some of the issues:

In theory, you can define lua_lock and lua_unlock and then execute 
different Lua threads (from the same context) in different (OS) 
threads. That's what Diego's luathread module does. But there are still 
a number of issues.

The most important one is that you either have to ensure that each 
thread has its own globals table, or that you never modify any global, 
or that you surround all uses of mutable globals with an explicit lock 
function, presumably implemented by the thread library you're using.

For example, you'd need to replace:

  someglobal = someglobal + 1

with

  mylock()
  someglobal = someglobal + 1
  myunlock()

Or something like that. And then you'd have to worry about the code 
within the lock throwing an error, an issue which we've been discussing 
in a different (mailing-list) thread.

You would also have to lock similar expressions with upvalues, if the 
upvalues could end up being executed in different threads.

Beyond that is the part the Lua script can't control: C modules.

There is only one Registry per Lua context, and some library functions 
modify the Registry; in particular, the lua_ref mechanism uses the 
Registry to store references. (That is, it passes LUA_REGISTRYINDEX to 
luaL_ref.) So any add-on module which uses lua_ref/lua_unref would need 
to call those functions within a mutex, or the module would have to be 
recompiled with a different macro definition of lua_ref and friends 
which called the corresponding lauxlib routines inside of a mutex. (I 
may be missing something but I don't see anywhere in luathread where 
this is addressed.) This doesn't affect any standard Lua library but 
I'm given to understand that a lot of people use lua_ref in their 
binding code.

The luaL_newmetatable mechanism is also not threadsafe, in the sense 
that it uses the Registry; I believe the 5.1 package system does that 
as well. So there may be race conditions associated with dynamically 
loading modules in different threads.

Finally, although there is not usually much code executed between a 
lua_lock and a lua_unlock, there is no guarantee. The VM itself is 
always executed within the lock; the lock is released when a CFunction 
is called. It is not easy to execute a lot of code without calling some 
library function or other (particularly as there is an implicit call to 
'next' in a 'for k, v in pairs(t)') but it is possible:

  for i = 1, 1e8 do
    j = j + i
  end

will hold the lock for an appreciable amount of time. Also, the garbage 
collector (or at least some phases of it in 5.1) run within the lock.

The main purpose of running multiple (Lua) threads in multiple (OS) 
threads would be to allow the threads to communicate, and any such 
communication would have to be carefully coded to avoid race 
conditions. On the other hand, if you were writing specific functions 
in order to allow threads to communicate with each other, you could 
probably write not much more complicated functions which would allow 
communication between unrelated lua_State's; the cost of creating a 
lua_State is not that great, and then you avoid all the overhead of 
lua_lock/lua_unlock, and the various complications of introducing 
explicit locking into your Lua code. At least, that's the approach I 
prefer.