Re: How to find out which modules a binary Lua .so declares?

[Hisham <h@...>]

On Fri, 31 Aug 2018 at 14:22, Dirk Laurie <dirk.laurie@gmail.com> wrote:
>
> Op Vr., 31 Aug. 2018 om 13:30 het Hisham <h@hisham.hm> geskryf:
> >
> > So, here's an interesting problem: since multiple Lua modules can be
> > stored in a single .so file (looking at you, LuaSec!), one can't
> > really tell which Lua modules a rock contains by scanning the
> > filesystem.
> >
> > This is currently causing `luarocks show` display incomplete data, and
> > has also caused bugs because other LR commands were naively assuming a
> > filesystem-to-modules correspondence. I do keep a list of module files
> > in rock_manifest, so those bugs are fixable.
> >
> > But for luarocks.loader (a package searcher that resolves the right
> > version of a module based on LuaRocks metadata, so that if you require
> > a module A version 1 and it depends on module B version 3, it will
> > pick the right one even if you have multiple B's installed in your
> > system), the only way LuaRocks can resolve a require()d module is if
> > it knows which rock it belongs to. So missing this data means that
> > luarocks.loader is currently unable to resolve the LuaSec binary
> > modules.
> >
> > I guess the only way to build complete module data will be to use nm
> > on Unix (and dumpbin on Windows I guess), at `luarocks build` time,
> > and search for "luaopen_*" symbols.
> >
> > But then how to know if luaopen_foo_bar_bla means
> > require("foo_bar_bla"), or require("foo.bar.bla"), or
> > require("foo.bar_bla"), or require("foo_bar.bla")?
> >
> > AFAICT, the right answer is "all of the above", so my current plan is
> > to "normalize" on one of those (say, dots) when storing the metadata
> > (i.e., "luaopen_foo_bar" means "foo.bar"), and have luarocks.loader
> > "normalize" the require() input when searching for .so files, in case
> > a non-"normalized" .lua filename cannot be found. Does that make
> > sense?
>
> Would it be cheating to run a little Lua program that requires the
> "main" module and then snoops around in package.loaded and/or
> debug.getregistry() to see what else came with it?

It would, as there is no general way to know what is the "main" Lua
module — in LuaSec, for example there's ssl.so which exports ssl.core
and others, but require("ssl") loads ssl.lua, and there's no guarantee
it will luaopen all modules at startup. Also, requiring modules may
cause side-effects, so it's not something that LuaRocks can blindly
do.

At this point, if we were to ask the rock author for help to identify
the main module or something to that effect, we could just follow
Enrique's suggestion here:

"Honestly, I would just add a "list your modules here if they don't
match the filesystem" field to the rockspec and from that moment on
consider it a bug by the libraries that don't use it when they need
to."
-- https://twitter.com/otikik/status/1035489554621517824

This is sounding more and more like a good idea, since doing it
automagically via catching symbols would still wouldn't resolve the
"how do I translate a luaopen_foo_bar name to the intended module
name", and I haven't bumped into that many modules that do this (I
think cqueues also bundles multiple modules into a single .so), and it
doesn't sound like an unreasonable thing to ask from rockspec authors.

One slight complication is that binary module filenames change
according to systems (ie, ssl.so vs ssl.dll) so it would need some
agnostic notation, like

exports = {
   ["ssl"] = { "ssl.core", "ssl.config", "ssl.context", "ssl.x509" },
   ["parentdir.binarymod"] = { "parentdir.binarymod.foo",
"parentdir.binarymod.bar" },
}

-- Hisham