Improved functions sandboxing in Lua and LuaJIT2

[Tymur Gubayev <tymur.gubayev@...>]

I apologize for my english, i know it isn't well.

I'm writing some server application in pure Lua (pure mostly because
of very limited experience in C/C++), and want to sandbox dangerous
Lua code (scripts from users etc). Hiding dangerous functions from
user script is more or less simply done with setfenv;
 Now the problem is: how to prevent scripts to loop forever and/or
grab whole CPU?
 There's a possibility to disable loops/tail calls at all (i've seen a
patch somewhere, but can't find it right know), but I don't like this
solution.
 A better one is to use debug.sethook with "count" mask. The basic
idea is to just check the time script is running on every N
instructions (with some large N like 1e4). This effectively eliminates
the indefinitely loops problem, but not the CPU overhead.

My very first idea was to just call sleep(0.001) from hook ( i'm using
luasocket library, and it provides access to sleep() for Lua). This is
ok in general, but not for server application: if the timelimit for
user scripts is high the server will lag on such scripts (and will
propably call sleep() at least twice)

My second naive idea to solve this problem was to call
coroutine.yield() every N instructions (giving control to server
application), but you can't yield across metamethod/C-call boundary.

Now I'm out of ideas (ok, the very last one is to just stop executing
script after, say, 1e5 instructions, but it's not so elegant as I
want). Any suggestions?

-------------
Now the LuaJIT (version 2.0.0-beta6) part:

  it's behavior is undefined if calling error() from debug hook (it's
the only way i know to stop scripts execution). With same source (see
below) i can get

1. error catched correctly (in second result from coroutine.resume)
2. error from hook function propagated to the main script (and the
main program stops with that error)
3. hook is being not called, thus letting script run indefinetly

Here are example outputs from my test script:
-------------------------------------------------------------------
>lua limittime.lua
some buggy script, that never ends and loads cpu
hook: limittime() called 11 times, i.e. 110000 instructions
Got script error:       limittime.lua:27: spent too much time:
0.010002136230469 seconds
987     1.000057220459
1987    2.000114440918
2987    3.000171661377
3000    cycles  3.0141735076904 seconds
>luajit
LuaJIT 2.0.0-beta6 -- Copyright (C) 2005-2011 Mike Pall. http://luajit.org/
JIT: ON CMOV SSE2 SSE3 AMD fold cse dce fwd dse narrow loop abc fuse
> dofile'limittime.lua'
some buggy script, that never ends and loads cpu
hook: limittime() called 42838 times, i.e. 42838 instructions
Got script error:       limittime.lua:27: spent too much time:
0.010000000000048 seconds
988     1
1988    2
2986    3
3000    cycles  3.014   seconds
> dofile'limittime.lua'
some buggy script, that never ends and loads cpu
---------------------------------------------------------------------
This was output for cases 1 and 3, and i just called
dofile'limittime.lua' twice from luajit console
And if uncomment line 31 (use sleep(dt)) i'm getting this (i.e. case 2):
---------------------------------------------------------------------
hook: limittime() called 37 times, i.e. 37 instructions
hook: limittime() called 38 times, i.e. 38 instructions
luajit: limittime.lua:27: spent too much time: 0.01 seconds
stack traceback:
	[C]: in function 'error'
	limittime.lua:27: in function 'select'
	limittime.lua:51: in function 'step'
	limittime.lua:68: in main chunk
	[C]: ?
--------------------------------------------------------------------

Also, there are things with LuaJIT I probably don't undestand correctly:
  - if use debug.sethook(co, limittime, 'count', 1e4), LuaJIT
definitively hangs, and with 1 works.
  - scr_time, an upvalue for the hook function (line 21) must be set,
even if updated just before the function call (line 50).

(luajit problem places marked with `--!` in the code)

Wbr,
 Tymur

------------
limittime.lua:

-- settings:
local hook_count = 1e4
local scr_timelimit = 0.01 -- time to wait for script
local dt = 0.001 -- time to sleep between steps
local steps_limit = 3000 -- "server" runs only ~3 seconds (steps_limit * dt)

local sleep, clock
local ok, ffi = pcall(require, 'ffi')
if not ok then -- plain Lua or FFI disabled
	require'socket'
	sleep = socket.sleep
	clock = socket.gettime
else -- LuaJIT2
	ffi.cdef'unsigned int Sleep(unsigned int msec);' -- this is probably win-only
	sleep = function (sec) ffi.C.Sleep(1000*sec) end -- socket.sleep compatibility
	clock = os.clock
	hook_count = 1 --! for some reason luajit fails if it differs from 1
end

local i = 0
local scr_time = math.huge --! must be initialized here for luajit. bug?
local limittime = function()
	i = i+1
	local new_time = clock()
	if new_time - scr_time > scr_timelimit then
		print('hook: limittime() called '..i..' times, i.e.
'..i*hook_count..' instructions')
		error('spent too much time: '..(new_time - scr_time)..' seconds')
	end
	-- coroutine.yield() -- can't use it in hooks: `attempt to yield
across metamethod/C-call boundary`
	--sleep(dt) --! error propagated to the top
end

local f = loadstring[[
	print('some buggy script, that never ends and loads cpu')
	while true do end
]]

-- sandbox f
local co = coroutine.create(f)
debug.sethook(co, limittime, 'count', hook_count)
f = function()
	return coroutine.resume( co )
end


local ret, err
local step = function ()
	--do stuff
	if not err then
		scr_time = clock()
		ret, err = select(2, pcall(f))
		if err then print('Got script error:',err) end
	end
end

local time = clock()
local start_time = time
local i = 0
while i<steps_limit do -- the main loop
	-- just to illustrate the progress
	local new_time = clock()
	if new_time - time >= 1 then
		print(i,new_time-start_time)
		time = new_time
	end
	i = i+1
	
	step()
	sleep(dt)
end
print(i,'cycles',clock()-start_time, 'seconds')