Re: segfault in lua(L)_getmetatable on invalid stack index (CORRECTION)

[benjamin sunshine-hill <bsunshin@...>]

er, whoops. I meant, of course, luaL_checkudata instead of luaL_getmetatable in all instances. The patch references the correct function.

Ben

----- Original Message -----
From: benjamin sunshine-hill <bsunshin@usc.edu>
Date: Sunday, July 6, 2003 2:08 pm
Subject: segfault in lua(L)_getmetatable on invalid stack index

> calling lua_getmetatable() on a stack index that is greater than 
> lua_gettop() produces a segfault when lua_getmetatable() dereferences a 
> null pointer returned by luaA_indexAcceptable(). This has the effect of 
> causing a segfault in the IO library:
> 
> fh = io.open("somefile.txt")
> fh.read() -- with no arguments, including no self, segfaults
> 
> IIRC, another function exists which can cause the same result. What's 
> happening here is that the IO library relies on luaL_getmetatable() to 
> ensure that it is acting on a filehandle. luaL_getmetatable(), in turn, 
> relies on lua_getmetatable(), but without first checking whether the stack 
> index it passes in actually references a valid metatable. This can be seen 
> as a bug in:
> 
> 1, the IO library, for not doing a lua_isuserdata() check before 
> luaL_getmetatable,2, luaL_getmetatable(), for not doing the same, or
> 3, lua_getmetatable(), for not checking to see whether the obj == NULL 
> before dereferencing.
> 
> I'd be inclined to conclude 2 or 3, especially since I first noticed the 
> error when working with my own code which made use of luaL_getmetatable(). 
> Here's an informal patch:
> 
> /* in lauxlib.c */
> /* change: */
> LUALIB_API void *luaL_checkudata (lua_State *L, int ud, const char *tname) {
>  const char *tn;
>  if (!lua_getmetatable(L, ud)) return NULL;  /* no metatable? */
> 
> /* to: */
> LUALIB_API void *luaL_checkudata (lua_State *L, int ud, const char *tname) {
>  const char *tn;
>  if (!lua_isuserdata(L, ud)) return NULL;
>  if (!lua_getmetatable(L, ud)) return NULL;  /* no metatable? */
> 
> I didn't want to change lua_getmetatable, even though it seems slightly 
> more likely to be at fault, since I really don't know what its effect on 
> an invalid index "should" be.
> 
> Ben
> 
> 
>