Lua 5.4 lists a change in "Incompatibilities in the API" for 'lua_version' that "The Lua core should work correctly with libraries using their own static copies of the same core, so there is no need to check whether they are using the same address space.".
This is also documented in some commits to the codebase.
What is the intention behind this and what does it actually enable? Under what circumstance would one need to share a state (L) between different instances of the same Lua library? Would such separation of instances not mean a separation of the states to begin with?
Also are there other benefits to having Lua states without global memory in them (such as luaO_nilobject) which was what enabled this feature?
Someone can correct me if I'm wrong, but my interpretation is that your program can use two copies of the (same version of) the Lua library without a problem.
For example, your program embeds Lua via static linking, so contains a complete copy of Lua in its executable. A Lua script written for your program calls `require "mylib"`. Your Lua runtime looks up the "mylib" module, which turns out to be a C extension module dynamically linked to a shared Lua library (i.e. .so or .dll) somewhere on your user's system. From what I understand, so long as that shared library and your embedded library are the same version (5.4), this should just work (whereas it would fail in 5.3 and before due to luaO_nilobject and possibly others).
As for a separation of states, the state is just a blob of bits in memory. Lua cores of the same version should be ABI compatible, so both copies of the core interpret the bits in the same way (assuming the compilers are the same or compatible, which isn't always the case).
Remember that the Lua library is just a collection of functions. With the 5.4 setup, it doesn't matter whether a particular function is called in your statically linked copy or in the dynamic library; they run the same machine instructions and manipulate the bits in memory in the same way.