Bug in debug.sethook

["Rici Lake" <lua@...>]

(Reported by sanooj in #lua)

Summary: the debug library's hook function can call lua_rawget() on nil,
causing unpredictable behaviour (probably a segfault).

The rawget is at line 212 in ldblib.c; it assumes that the lookup of
KEY_HOOK in the registry will return a table. The table is created by
gethooktable() which is called by db_sethook (debug.sethook). Under
normal circumstances, this means that the table will be present when
the hook function is invoked.

Unfortunately, it is possible for the lua_createtable() in gethooktable()
to fail and throw a memory error. Since db_sethook calls lua_sethook()
before calling before gethooktable(), this could result in the hook
function being installed without a valid hooktable, leading to the
call of rawget() on nil when the return hook subsequently fires.

Of course, this is highly unlikely. It was apparently discovered by
use of the memfault tool.

It is only fair to quote a bit of the conversation on #lua:

sanooj: spent a lot of time yesterday hammering lua with memfault
rici: did you find one?
sanooj: just one
sanooj: the rest of the time.. wow..I've *never* seen software as stable
as this.