I'd say in this
if (globalhook || (L->hookmask & LUA_MASKRET)) { /* is return hook on? */
and other places where you introduced a second conditional, it was not really necessary.
You could define another hook flag LUA_GLOBALHOOK, and set it up in L->hookmask whenever G(L)->globalhook is set up with a non-null value, and also when switching from one Lua thread to another.
You would still have to check G(L)->globalhook again, but that would happen only in the slow path.
And, since I am talking about your code, your use of 'volatile' is technically incorrect, you need an atomic type instead, but I am afraid it was not you who started this in the Lua codebase.
Cheers,
V.
