Potential Null Pointer Dereference in resizebox method from lauxlib.c

[Yongchao Wang <chaowyc@...>]

Hi all,
We have detected that the resizebox method may trigger a null pointer dereference. Here is a possible vulnerable trace:

1. Return null to caller at lapi.c:457
// From lapi.c
l_sinline void *touserdata (const TValue *o) {
  switch (ttype(o)) {
    case LUA_TUSERDATA: return getudatamem(uvalue(o));
    case LUA_TLIGHTUSERDATA: return pvalue(o);
    default: return NULL;  // line 457 Return null
  }
}
2. Return the return value of function touserdata, could be null, to caller at lapi.c:462.
//From lapi.c
LUA_API void *lua_touserdata (lua_State *L, int idx) {
  const TValue *o = index2value(L, idx);
  return touserdata(o); // line 462 Return null
}

3. Function lua_touserdata executes and stores the return value to box (box can be null) at lauxlib.c:476
and load value from box->size at lauxlib.c:477, which will lead to null pointer dereference
// From lauxlib.c
static void *resizebox (lua_State *L, int idx, size_t newsize) {
  void *ud;
  lua_Alloc allocf = lua_getallocf(L, &ud);
  UBox *box = (UBox *)lua_touserdata(L, idx);  // line 476 box could be null
  void *temp = allocf(ud, box->box, box->bsize, newsize); // line 477 dereference box
  if (l_unlikely(temp == NULL && newsize > 0)) {  /* allocation error? */
    lua_pushliteral(L, "not enough memory");
    lua_error(L);  /* raise a memory error */
  }
  box->box = temp;
  box->bsize = newsize;
  return temp;
}

This could lead to a program crash or other unwanted behavior. Please fix it as soon as possible.
Best
Yongchao