Re: Replacement for luaL_openlib in Lua 5.1

[Rici Lake <lua@...>]

On 2-Dec-05, at 2:16 PM, Shannon Stewman wrote:

> On Fri, Dec 02, 2005 at 12:34:15AM +1100, David Burgess wrote:
> > I would vote for some more detail on LUA_ENVIRONINDEX in the
> > 5.1 manual. Anyone else?
>
> Hear hear.

Quick explanation:

In Lua 5.0.2, Lua closures (but not C closures) have "environment 
tables". The environment table is used to look up unbound names (that 
is, "global variables"). When a new closure is created, by executing a 
function expression or statement, the new closure acquires the 
environment table of the currently executing closure, so in the absence 
of any changes, all closures share the same environment table, which 
makes it pretty much a "globals" table as in previous versions of Lua.

The environment table of a closure can be accessed in C with 
lua_getfenv and in Lua with getfenv; it can be set to a different table 
with lua_setfenv / setfenv. See the manual for these functions.

In 5.0.2, C closures effectively share a single globals table, which is 
associated with the currently executing thread. This table is available 
to the C closure by using the pseudo-index LUA_GLOBALSINDEX (and indeed 
can be modified with lua_replace(L, LUA_GLOBALSINDEX)). This is a 
useful but can be a bit messy.

In 5.1, every C closure has its own environment table reference. 
However, the LUA_GLOBALSINDEX pseudo-index has not been altered, so 
that existing code which references LUA_GLOBALSINDEX will still be 
referring to the "globals" table of the currently executing thread. 
(That table is now referred to as the thread's environment table, and 
can be accessed and modified with lua_getfenv and lua_setfenv applied 
to the thread object itself.) In order to access the closure's 
environment table, one uses LUA_ENVIRONINDEX; a closure environment 
table (for either a C closure or a Lua closure) can be accessed and 
modified with lua_getfenv and lua_setfenv, applied to the closure 
object.

As far as I know, lua_replace will not work with the LUA_ENVIRONINDEX 
pseudo-index, so if you want to replace "your own" environment, you'll 
need to get a reference to "your own" closure object, which is only 
available through the debugging API. However, the more common case is 
to call lua_setfenv on a newly-created closure.

(Full) userdata also have environment tables, although as I have argued 
elsewhere, the name is confusing. Userdata environment tables are 
*only* a place to store information relevant to the userdata; no 
pseudo-index exists for them.

Newly created closures (and userdata) acquire their environment tables 
from the currently executing closure, if there is one, and otherwise 
from the current thread. (Again, as I have argued elsewhere, this is 
not a particularly useful default for userdata, but I won't go into 
that again.)

C closures don't really have a concept of bound and unbound variables. 
However, the C closure does have a number of places to put Lua values:
-- transiently on the stack, at a known index
-- or persistently, in order of privateness:
   -- in the upvalue array, which is a fixed-length array created as 
part of the C closure by lua_pushcclosure. (That is, the size is fixed 
at creation, but individual elements can be modified by lua_replace.)
   -- in the closure's own environment table
   -- in the thread's environment table
   -- in the Lua registry.

Figuring out which of these options to use for what can be a challenge, 
but I offer my own very personal guide:

Use upvalues:
  1) for reference data which is frequently used by the closure. This 
is particularly useful in closures, such as iterators, which need to 
return other closures.
  2) to persist state information between calls to a particular 
closure. This can be useful for iterators themselves, for example, if 
the iteration state cannot be captured adequately by the iteration 
protocol

Use the closure's own environment table:
  1) as a table shared between related closures. For example, this can 
be a good place to store the method table for the class for which the 
closure is a method, and/or to store persistent state information 
shared by methods for a given class.

Use the thread's environment table:
  1) To keep state information which is thread-specific. (Duh!) In an 
application which uses Lua threads for multithreading, this is more or 
less equivalent to thread local state; it might be, for example, 
something related to the event loop running in that particular thread.

Use the registry:
  1) To keep persistent information which applies to the entire 
application, and
  2) Only in desperation.

Note:

The only options which are threadsafe are the stack and the thread's 
environment table. With appropriate definitions of lua_lock and 
lua_unlock, you cannot corrupt the Lua state by storing into the 
registry or the closure's upvalue array or environment table, but 
nothing stops another (OS) thread from storing something else at the 
same time, leading to the standard race condition where access and 
update are interrupted by another process. In particular, this applies 
to lauxlib's implementation of references (if stored in the registry). 
With all the various environment tables available, however, this should 
now be much easier to avoid.

The thread's environment table is threadsafe provided that each Lua 
thread is mapped onto a single OS thread (it can be a many-to-one 
mapping); that is, a given Lua thread is always run in the same OS 
thread. That makes it an attractive option for keeping lauxlib-style 
references, if it is not possible to avoid using them altogether.

The closure's upvalue array and environment table are threadsafe under 
the same conditions, but this is often harder to guarantee. It will 
likely be the case for short-lifetime closures, such as ones returned 
to implemented an iterator, but it unlikely to be the case for 
long-lifetime closures, such as library functions.

If you do find yourself modifying the registry in a multi-threaded 
environment, you should think seriously about protecting the 
modification with a lock of some form. This also applies to the use of 
registry-based lauxlib features such as luaL_newmetatable, if its use 
is not restricted to the pre-thread initialization stage.

Hope that helps someone.

Rici