The "Environment Stack" (+ lexically scoped string metatables)

["Soni \"They/Them\" L." <fakedme@...>]

Hey Lua team!

I had this idea some years ago, I call it the "environment stack". It's 
perfectly compatible with existing Lua syntax, requiring only changes to 
the code generator, VM and libraries, and eliminates one of the pain 
points of Lua: metatable handling. (I've been told you plan on changing 
metatable handling, so I thought I'd pitch in)

Current Lua uses an actual register/upvalue for _ENV, and uses 
register/upvalue lookup to find _ENV. Thus, code of the form:

print(getmetatable(""))

First transforms into:

_ENV.print(_ENV.getmetatable(""))

Which then transforms into:

<register or upvalue for _ENV>.print(<register or upvalue for 
_ENV>.getmetatable(""))

I am not proposing we change the first transformation. Instead, I am 
proposing we change the second transformation.

What the "environment stack" does is, it provides a stack of environments.

With the 6 opcodes: GETTABST, SETTABST, PUSHST, POPST, GETST, SETST, we 
can build an "environment stack". From the given Lua code:

do
  local _ENV = {print=print}
  print()
end

We emit the following opcodes, with added comments detailing code 
generator behaviour, as well as explaining how the opcodes operate:

NEWTABLE 0 0 1 ; creates table at reg 0
GETTABST 1 0 -1 ; gets key "print" from upvalue 0, puts it at reg 1
SETTABLE 0 -1 1 ; sets key "print" at reg 0 to value at reg 1
; (frees reg 1)
PUSHST 0 0 ; pushes table at reg 0 onto "environment stack" slot 0
; (frees reg 0)
GETTABST 0 0 -1 ; gets key "print" from "environment stack" slot 0, puts 
it at reg 0
CALL 0 1 1 ; calls function at reg 0
; (frees reg 0)
POPST 0 ; pops "environment stack" slot 0 onto reg 0 -- corresponds to 
"end" in above example
RETURN 0 1

This can then be easily extended to metatables:

_ENV --> slot 0
_MT_STRING --> slot 1
_MT_NUMBER --> slot 2
and so on

On a simple block of code like the above, this is indeed a simple 
push/pop mechanism. However, if we add in some closures:

local f
local g
do
  local _ENV = {}
  function f() return _ENV end
  function g(newenv) _ENV=newenv end
end
g(1)
print(f()) -- prints 1

Up until "local _ENV = {}" it's pretty simple: make a table. push it. 
However, when you create the closure, the CLOSURE opcode captures the 
current values/slots on the stack(s). This is similar to how upvalues 
work, really. As with upvalues, you can set the environment (with _ENV = 
whatever) in the closure and it'll change the environment for everything 
using the same environment.

A small edge case: you can currently accept _ENV as an argument:

local function f(_ENV)
  print()
end
f({print=function() print "hello world" end})

This is a rare edge case, but easily handled: just emit a PUSHST as the 
first instruction in a function.
As for repeating it:

local function f(_ENV, _ENV, _ENV) end

Same idea applies, just PUSHST every one (from left to right, please!).

As for metatables and things: every time you use an opcode, it has 
access to the current environment stack. It can easily look up which 
metatable it needs to use. On the other hand, getmetatable and 
setmetatable now strictly only apply to tables and full userdata - all 
other metatables are accessed through the reserved names _MT_*.

To sum it up, here are some cases and their handling:

function f() local _ENV = {} return _ENV end -- handled by above rules - 
"return _ENV" generates a GETST for _ENV, then "return" itself pops all 
relevant PUSHSTs, and then does RETURN.
do local _ENV = {} return _ENV end -- same as above
do local _ENV = {} end -- simple PUSHST followed by POPST
function f(_ENV) end -- has to generate additional PUSHSTs for each _ENV 
in the argument list.
do ::test:: local _ENV = {} goto test end -- "goto test" needs to POPST. 
perhaps similar to `do ::test:: local f; local function g() return f 
end; goto test; end`.
-- (most cases can be compared to closures and upvalues)

And here are some benefits:

- Modules can use things from caller environment using debug library. 
(I've seen modules do this with coroutines before. See below.)
- SIGNIFICANTLY simplifies virtualization modules. Previously I've used 
coroutines for this: I installed global metatables for all types (using 
debug.setmetatable), then used the currently running coroutine to figure 
out which environment made the call (with 
`metatables[coroutine.running()][currentType]`), then passed it on to 
the correct function. This feature would completely eliminate all that code!
- Modules would be able to locally customize string, number, etc 
metatables, without affecting the main program. Modules will finally be 
able to use `"%s %s" % {"hello", "world!"}`!
- Among other things.

I'm not providing an implementation because I know you don't accept 
patches. However, if you do this, please don't make the Lua 5.1 mistake 
of implementing this as a call-stack-sensitive standard library function 
- if we've learned anything from Lua 5.1 it's that it's unsafe and 
dangerous, and makes sandboxing rather difficult.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.