RE: PiL 3 and pcall

[Dan Tull <dtull@...>]

> > (from PiL 3)
> > "Can you find any value for f such that the call pcall(pcall,f)
> > returns false as its first result?"

> (Pierre Chapuis [catwell@archlinux.us])
> Has anyone found a way to do it *without* debug?

After a few failed attempts, I got had to look at the code.

There is a way to make pcall itself have false as its first result
besides the function it is asked to call throwing an error.

If the function it calls returns so many values that it exhausts
the stack and there's no room for pcall to push the boolean
value to replace the nil placeholder it put in the 1st position,
it will return false.

LUAI_MAXSTACK is big enough even on 32-bit systems that
I doubt this happens in practice, but assuming I'm reading
it right, it's an interesting edge case.

Of course, now that I have a hypothetical way to make it 
happen I feel compelled to confirm experimentally...

DT

/* for reference (from 5.2.1 source) */

static int finishpcall (lua_State *L, int status) {
  if (!lua_checkstack(L, 1)) {  /* no space for extra boolean? */
    lua_settop(L, 0);  /* create space for return values */
    lua_pushboolean(L, 0);
    lua_pushstring(L, "stack overflow");
    return 2;  /* return false, msg */
  }
  lua_pushboolean(L, status);  /* first result (status) */
  lua_replace(L, 1);  /* put first result in first slot */
  return lua_gettop(L);
}