coroutine library can cause panic with low memory

[Patrick Donnelly <batrick@...>]

Using the attached script one can cause Lua to panic. The problem is
the API call lua_checkstack which may throw a memory error. When
auxresume in lbaselib.c fails to allocate stack space, there is no
error handler set on the coroutine so Lua will panic.

batrick@li50-61:~$ ulimit -v 10000 # simulate a memory constrained env
batrick@li50-61:~$ lua test.lua
1       thread: 0x532c70        true    nil
2       thread: 0x539ac0        true    nil
3       thread: 0x537df0        true    nil
4       thread: 0x534650        true    nil
5       thread: 0x5365c0        true    nil
6       thread: 0x53b500        true    nil
7       thread: 0x53d470        true    nil
8       thread: 0x566c20        true    nil
9       thread: 0x53f3b0        true    nil
10      thread: 0x541500        true    nil
11      thread: 0x557480        true    nil
12      thread: 0x5593f0        true    nil
13      thread: 0x55c9b0        true    nil
14      thread: 0x55e920        true    nil
15      thread: 0x560890        true    nil
16      thread: 0x562800        true    nil
17      thread: 0x564770        true    nil
18      thread: 0x566e50        true    nil
19      thread: 0x568dc0        true    nil
20      thread: 0x588530        true    nil
PANIC: unprotected error in call to Lua API (not enough memory)

lua_checkstack should probably not throw memory errors because it
cannot knowingly throw the error on the currently running thread. The
thread you are checking stack space for may not actually be running!

-- 
-Patrick Donnelly

"One of the lessons of history is that nothing is often a good thing
to do and always a clever thing to say."

-- some constants they may need to be tuned
local NUM = 100; -- number of garbage tables made each iteration

-- A large stack of values to unpack (near the max C stack size)
local t = {}
for i = 1, 7.5e3 do t[i] = i end

local f = {}; -- garbage values
for i = 1, 1e300 do
  for i = 1, NUM do f[{}] = true end
  local co = coroutine.create(function() end);
  local s, r = coroutine.resume(co, unpack(t))
  print(i, co, s, r)
  collectgarbage"collect"
end