Re: dependencies between C modules, RTLD

Wow, my original message got mangled, sorry about that. Such a shame, I did nice ASCII art and everything. :)

I'm going to try once more and see if my original formatting is preserved.

--

Short question: what is the rationale for Lua loading C
extensions as RTLD_LOCAL by default?

Long story: I have several C extensions with dependencies
between them.

+---+
+-->| A |<--+
| +---+ |
| |
+---+ +---+
| B | | C |
+---+ +---+

These dependencies are C-level dependencies; ie. A contains
a C function foo() that B and C call.

Now I could tell the linker about these dependencies when
I compile these modules. That would make the OS aware of
these dependencies, and make it automatically load A when
B or C are loaded.

But this approach has a couple downsides:

1. Users now need to set both LUA_CPATH *and* LD_LIBRARY_PATH
correctly. Kind of a big pain, especially since the two
have different failures modes so it's not immediately
obvious which one you got wrong. Also, the LD_LIBRARY_PATH
part is inherently OS-specific, and is different on OS X.

2. If the user wants to install your library, they will expect
to *not* have to set LD_LIBRARY_PATH, so you have to install
into /usr/lib or /usr/local/lib, and give it an soname that
won't conflict with anything else on the system. But this
is a little weird, because the library isn't intended to
be used from C, it's *only* intended to be loaded by Lua.

What I wish I could do is guarantee, from Lua, that A is always
loaded before B or C. This is easy to do with some trivial
wrappers:

-- b.lua
import("a")
return import("b_cext")

-- c.lua
import("a")
return import("c_cext")

This would be a really nice solution that solves both of the
above problems. But unfortunately it is not possible because
require() loads C libraries with RTLD_LOCAL. That means that
the symbols loaded in A are not available to be linked against
from B or C.

Any thoughts or workarounds?

Thanks,
Josh

On Tue, May 12, 2015 at 4:14 PM, Josh Haberman <jhaberman@gmail.com> wrote:

Short question: what is the rationale for Lua loading Cextensions as RTLD_LOCAL by default?
Long story: I have several C extensions with dependenciesbetween them.
+---+ +-->| A |<--+ | +---+ | | | +---+ +---+ | B | | C | +---+ +---+
These dependencies are C-level dependencies; ie. A containsa C function foo() that B and C call.
Now I could tell the linker about these dependencies whenI compile these modules. That would make the OS aware ofthese dependencies, and make it automatically load A whenB or C are loaded.
But this approach has a couple downsides:
1. Users now need to set both LUA_CPATH *and* LD_LIBRARY_PATH correctly. Kind of a big pain, especially since the two have different failures modes so it's not immediately obvious which one you got wrong. Also, the LD_LIBRARY_PATH part is inherently OS-specific, and is different on OS X.
2. If the user wants to install your library, they will expect to *not* have to set LD_LIBRARY_PATH, so you have to install into /usr/lib or /usr/local/lib, and give it an soname that won't conflict with anything else on the system. But this is a little weird, because the library isn't intended to be used from C, it's *only* intended to be loaded by Lua.
What I wish I could do is just provide, from Lua, that A is alwaysloaded before B or C. This is easy to do with some trivialwrappers:
-- b.luaimport("a")return import("b_cext")
-- c.luaimport("a")return import("c_cext")
This would be a really nice solution that solves both of theabove problems. But unfortunately it is not possible becauserequire() loads C libraries with RTLD_LOCAL. That means thatthe symbols loaded in A are not available to be linked againstfrom B or C.
Any thoughts or workarounds?
Thanks,Josh