> In particular, it would have the original race condition since L->hookmask isn't even checked if ci->u.l.trap doesn't get set, and it wouldn't work to interrupt other coroutines anymore since that's in L and not in G(L).
I am not so deep in these details as you are. Still, I have thought more about this, and have another idea, FWIW.
#define LUA_HOOKGLOBAL 5
#define LUA_MASKGLOBAL (1 << LUA_HOOKGLOBAL)
Add to global_State:
_Atomic int hookmask;
_Atomic lua_Hook globalhook;
When a Lua thread starts or resumes, it copies its own hook mask from L->hookmask into G(L)->hookmask, while preserving LUA_MASKGLOBAL in G(L)->hookmask - atomically. Ditto when hooks are changed.
All the code that currently checks for any hooks, should check G(L)->hookmask, with LUA_MASKGLOBAL where appropriate. G(L)->globalhook should be checked after the mask is checked positively.
The point of the above is to have just one check for hooks in the hot path, like it is the case today.
Cheers,
V.
