> Can a Lua executable compiled as one (C or C++) successfully load and use an extension module (*.dll or *.so) written in the other?
If you mean that your main application executable has Lua compiled and statically linked into it, then you will not be able to load a shared/dynamic library that needs to use the Lua API - because it expects the Lua API to be exposed by a shared/dynamic library. That is, unless you develop some private interface between your application and "extension modules", or patch the low-level library loader code, either way going quite beyond the C/C++ question.
Normally, if you need to load libraries that use Lua, the Lua engine must itself be a shared/dynamic library, in which case the C/C++ question applies to this library first and foremost.
Linking Lua either statically or dynamically, there are two major things to consider:
1. C++ name decoration/mangling. If the Lua library is compiled in the C++ mode, by default, the Lua API functions will be decorated as such, making them unavailable to C-mode code. This can be remedied by making the Lua API defined as export "C". Unfortunately, this is not available out of the box, and you will have to patch luaconf.h, for example, as follows:
#define LUA_API extern "C"
The patched luacnf.h must be used when compiling
the Lua library and any other code that uses it. If the Lua library is compiled in the C mode, and it needs to be used by C++ code, the same solution works, in principle, but you could use the stock headers, too:
2. If the Lua library is compiled in the C++ mode, long jumps vs C++ exceptions for the Lua error mechanism. Unfortunately, there is no one size fits all solution. The only truly portable solution is the long jump mode, but then the C++ code that is invoked by Lua and itself uses the Lua API, must essentially become pure-C code and not have any automatic storage objects with non-trivial destructors. Such code must also not leak any C++ exceptions (in fact, this restriction stays in the other case, too). Depending on your situation, this may be easy or pretty much impossible. For example, consider a function callable from Lua.
int foo(lua_State *L)
std::string ret = "some lengthy string that needs memory allocation";
auto inp = luaL_checklstring(L, 1, std::nullptr);
The ret variable has a non-trivial destructor that won't be called if luaL_checklstring() raises an error. And its constructor could, in principle, throw an exception. While in this toy functions both problems can be easily mitigated, in a more complex situation that may get quite a bit more involved.
Using C++ exceptions for the error mechanism, and using some C-mode code that is invoked by Lua, you would have to ensure that such exceptions can propagate correctly through C/C++/Lua boundaries, which would be highly platform dependent, where the platform includes such details as static vs dynamic linking and compiler options.
> I'm most interested in a C-compiled Lua and C++ extension module
In this case, the two major decisions are automatically settled, with the above mentioned implications on the C++ code.