Use of pointer comparisons to static data in Lua C code

[Tom Sutcliffe <tomsci@...>]

Hi list,

An interesting problem just hit me when trying to embed Lua in a large and complicated Win32 application, which boils down to Lua making some unfortunate assumptions in its C implementation. They aren't unreasonable assumptions to make, but since they turn out to be pretty critical and I stumbled into a situation which violates them I'd be interested in what people think about it.

I have a Windows project which embeds Lua 5.3. Because of the structure of the project, Lua is built as a static library (.lib in Windows parlance) which is eventually linked into the final EXE. This EXE hosts the Lua runtime and loads scripts etc. So far so good.

Unfortunately I also want these scripts to (in some circumstances) be able to load additional native code libraries, which are build as DLLs which also (because Lua is built as a static library) end up including a copy of the Lua runtime. This is hardly ideal from a disk usage point of view, and I hope to eventually clean this up, but I figured that since Lua doesn't use any global data it won't actually matter providing I can guarantee that the 2 versions of the Lua code are identical and I don't mix unrelated lua_States.

However there are a couple of places where Lua uses pointer comparisons to static data - in luaO_nilobject and dummynode are the two I found after a quick search. Because there are now multiple copies of the Lua code and static data, one coming from the lib compiled into the host EXE and one from the lib in the extension DLL, the static data will have different pointer addresses depending on which code is checking it, which means that code which checks for "if (x == luaO_nilobject)" and similar will do the wrong thing leading to almost immediate crashes.

In an ideal world I would build Lua as a DLL, have both the EXE and the extension DLL link against the Lua DLL, then there'd only ever be 1 version of the static data as nature intended. However the size and structure of the project and where the Lua runtime fits into it makes this rather tricky without a massive refactor. Ideally I'd like for Lua to be able to tolerate my slightly weird setup as it does so successfully in all the other crazy situations I've thrown at it.

I had a go at modifying isdummy(n) so that it checked for the key type being nil rather than just comparing its pointer value against &dummynode_, which seemed like it might work, but the places luaO_nilobject is used looked a bit too complicated for me to try hacking with. Maybe something could be added to the global_State which would remain unique across multiple instances of the code, or something. I still haven't really decided whether what I'm doing is reasonable or not - on the one hand I'm running multiple copies of the Lua code at different addresses in a single process and passing lua_States between them, which is a weird thing to be doing. On the other hand, my mental model of how generally stateless a lua_State is makes me feel like this should work. Your thoughts appreciated!

Cheers,

Tom