Found it.
The problem:
It's similar to the same problem I had on GCC/Linux and MinGW builds, except that compiler presented it to me as a linker error rather than a runtime failure that I couldn't quite understand.
The error is tossed from luaL_checkversion (luaL_checkversion_ is what it points to in lauxlib.c, 5.2.4 source tree):
LUALIB_API void luaL_checkversion_ (lua_State *L, lua_Number ver) {
const lua_Number *v = lua_version(L); // uh oh...
if (v != lua_version(NULL)) // --------------- HERE!!!
luaL_error(L, "multiple Lua VMs detected");Linking the same static library in a Windows Executable and a Windows DLL means that the static library has the same code dumped out twice. This means there are 2 versions of the code, and the check being done here places a number inside of its final compiled target to uniquely identify the Lua VM to the binary it was compiled into (whether that was from being statically linked or if it was baked into a DLL).
I'm not sure what Lua 5.1 didn't have or Lua 5.3 changed/introduced to make it work, but this is the gist of the problem.
Explanation and Solutions:
On GCC/Linux and MinGW, trying to link 2 libraries like this ultimately produces a failure at link-time, stating that I have multiply-defined symbols attempting to be packed into my executable. There's also a separate problem of when you only try to link one on later versions of GCC, where it will perform an optimization where it strips out unused symbols. So, if I were to link the static lib into the DLL only and then hope that all of Lua's functions were available when linking the DLL into the Executable, it would complain about undefined references to functions it couldn't find (because the Linker optimizes for space by default and cuts unused functions).
On GCC/Linux, the solution is to link the library only once in the DLL, and surround it with the flags -Wl,-whole-archive {lua library name} -Wl,-no-whole-archive.
On Windows, they just privatize the static library into each executable / dynamic library's virtual memory space and load them separately. Supposedly, I have the ability to do much the same as GCC with the flag /WHOLEARCHIVE[:libname], where in this case I would put the static library name of Lua there.
Hopefully, this helps somebody else who comes across this.