Malicious Lua 5.1 bytecode in the real world

[Peter Cawley <lua@...>]

I'm not aware of a documented instance of using malicious Lua bytecode
to escape from a real world Lua 5.1 sandbox, so I felt it worthwhile
to document a recent case of escapism.

The sandbox in question is in a 32-bit Windows program called
RelicCOH2.exe. It uses Lua 5.1.3 for many things, and in one instance
as a data description language for font metadata files. For these font
files, the program calls lua_newstate followed by luaopen_base
followed by luaL_dofile, which results in an interesting sandbox: the
base and coroutine libraries are present, but useful things like the
string, table, and package libraries aren't present. I wanted to
escape from this sandbox and call package.loadlib from one of these
font metadata files in order to load one of my DLLs into the program,
and with some malicious bytecode I was able to do so.

My previous warnings about bytecode in sandboxes have had the caveat
that the bytecode needs at least to be able to call string.gsub and
table.sort. At least for Lua 5.1.3 on 32-bit Windows, that caveat no
longer applies: malicious bytecode in this environment can perform
arbitrary TValue-sized memory reads and writes without needing any
other functions to be present. In this case of escapism, the presence
of coroutine.wrap presented a quick way of achieving my goal, but its
presence is not necessary. With it present, the malicious bytecode can
be summarised by the following three lines of Lua, which should scare
any sandbox author:

co = coroutine.wrap(function() end)
-- ... section of code that doesn't assign to co ...
co(dll_name, function_name) -- actually calls package.loadlib

If you're interested in the actual malicious bytecode or the Lua 5.1.3
virtual machine internals that lead to this escape capability, then
I've attempted to explain things at
https://gist.github.com/corsix/6575486