disable the code path that loads compiled VM code altogether, in the Lua C source. The small CPU time savings are not worth the risk when you're about to run potentially malicious code.
only allow those functions that you know are safe and useful for your purposes. Anything unnecessary goes. Ideally the code should be commented out in the source code. Stuff in the debug and os modules are candidates for removal, as are any file primitives. Be ruthless, the attackers will be too.
find some way to kill off Lua programs that use an inordinate amount of CPU time or memory. Using a hook in the interpreter to detect runaway programs may not be enough, one can write regular _expression_ matches that for all practical purposes never terminate.