segfault in lua(L)_getmetatable on invalid stack index

[benjamin sunshine-hill <bsunshin@...>]

calling lua_getmetatable() on a stack index that is greater than lua_gettop() produces a segfault when lua_getmetatable() dereferences a null pointer returned by luaA_indexAcceptable(). This has the effect of causing a segfault in the IO library:

fh = io.open("somefile.txt")
fh.read() -- with no arguments, including no self, segfaults

IIRC, another function exists which can cause the same result. What's happening here is that the IO library relies on luaL_getmetatable() to ensure that it is acting on a filehandle. luaL_getmetatable(), in turn, relies on lua_getmetatable(), but without first checking whether the stack index it passes in actually references a valid metatable. This can be seen as a bug in:

1, the IO library, for not doing a lua_isuserdata() check before luaL_getmetatable,
2, luaL_getmetatable(), for not doing the same, or
3, lua_getmetatable(), for not checking to see whether the obj == NULL before dereferencing.

I'd be inclined to conclude 2 or 3, especially since I first noticed the error when working with my own code which made use of luaL_getmetatable(). Here's an informal patch:

/* in lauxlib.c */
/* change: */
LUALIB_API void *luaL_checkudata (lua_State *L, int ud, const char *tname) {
  const char *tn;
  if (!lua_getmetatable(L, ud)) return NULL;  /* no metatable? */

/* to: */
LUALIB_API void *luaL_checkudata (lua_State *L, int ud, const char *tname) {
  const char *tn;
  if (!lua_isuserdata(L, ud)) return NULL;
  if (!lua_getmetatable(L, ud)) return NULL;  /* no metatable? */

I didn't want to change lua_getmetatable, even though it seems slightly more likely to be at fault, since I really don't know what its effect on an invalid index "should" be.

Ben