since we have that new flashy module standard in place for Lua 5.1
we should think about a standard and portable way to build and install
extension modules (both Lua and C modules, where the latter is the more
difficult part).
I had a little E-Mail conversation with Diego about this and we both
identified the need to have something like a 'lua-config' program or
similar functionality. This would allow a developer to modify a single
installed file and put all knowledge about the local build system into
it.
When we standardize on something this could be used by all 3rd party
extension modules and ease the task of adding modules to your local
installation considerably.
And extension module authors would benefit, too. It just does not make
sense to reinvent the wheel (the build process) in thousand different
ways
and then see every wheel break on a different platform the author
hasn't
thought about.
However I'm not sure what the best approach for this would be. Here are
a few candidates:
- Many OSS packages now include a '${package}-config' program or a *.pc
file for pkg-config. This is usually a shell script which prints
stuff like the required compiler or linker options.
Alas, this approach is mostly useful for autoconf style builds of
applications that need a specific library. And it is more or less
limited to POSIX systems unless you have a fully-fledged shell at
hand.
And for building extension modules you want to know different kind of
things, too: e.g. how to build a shared or static library and where
and how to install it.
- A variant of this would be to derive the required info from the
package
table. E.g. the install directory for modules is more or less already
present there, hidden in package.path and package.cpath. It would be
pretty simple to explicitly store this and other info in other
elements
of the package table.
Then retrieving these is as simple as: lua -e 'print(package.foo)'
But then you still have the knowledge about the build process split
between a makefile and some Lua variables (or functions for more
complex
stuff). It is fairly unlikely that you'll succeed in making this work
for the whole range of systems where Lua runs on and the associated
build processes (think about cross-compiling and get worried).
Yet, having programmatic access to some key elements required for the
build process is a step in the right direction.
- Python replaces the whole build process with the 'distutils' package.
You basically get programmatic access to a set of classes that have
all the required functionality (dependency checking, calling the
compiler,
the linker and even the OS specific packaging system). Your extension
has to include a tiny Python file that imports distutils and passes
a few key information elements (such as the C or Python files that
are part of your extension). Of course you have all the power of a
programming language at hand to change the build process.
As an extension author you don't need to write a makefile anymore.
When building extensions you don't need the make program anymore.
As a local admin or developer you can change the local setup in one
place
and all extension modules are then built incorporating your changes
to the process.
I like this approach because you can add arbitrary complex rules and
adapt
the whole build process to your needs. After all a makefile is just
a bad
excuse for a turing-complete language -- so why not use a 'real'
language?
Since you obviously (*) must have one at hand when you are building
modules for it (Python in this example, Lua in ours).
However this approach is a bit heavyweight (9.000 lines of Python
code)
since it incorporates a lot of knowledge about the tool chain
(compiler,
linker, archiver, OS packager) and reimplements make functionality.
It also depends on a lot of system dependent functionality to call
subprocesses or check the local OS environment. No problem for Python
here with its giant library, but somewhat of a problem for Lua.
Still it's doable and we could start small.
(*) Not so obvious for cross-compiles though. But Lua qualifies since
it is very likely to compile both on the developer host and the
target host.
- Perl has an abominable system that creates a Makefile on the fly and
does preprocessing on source files. I found this difficult to
integrate
and maintain (IMHO of course). Maybe it's just me or maybe it could
be adapted to integrate better (like generating an includable
Makefile).
- Ruby has mkmf.rb and seems to create a makefile on the fly, too.
However I have no experience in building modules for Ruby.
- There are other build systems (like a-a-p) that have their own
flavour
of a configuration language and claim better portability than
makefiles.
However they need to be installed before you can build an extension.
And you are basically stuck on systems where the build system is not
available.
I'd rather have something self-contained, written mostly in Lua and
maybe a few OS specific parts in C. Since this is a rather
lightweight
install, it is likely that we can convince everyone to install it and
accept this as part of the tool chain for building Lua extensions.
- And then there's the possibility (or need) to integrate this with
IDEs.
Think about Eclipse or the countless Windows tools for this purpose.
I feel not qualified to comment on this other than to say that it's
probably a bad idea to depend on any of these tools. Still, it would
be nice for the build process to work smoothly within these
environments.
Of course this list can be extended ...
Maybe we should collect both requirements and solutions systematically
and then somehow seek to find the best compromise that satisfies many
or
even most needs.
So what are the needs for your build process? Anyone has stories to
share about building for different platforms? Suggestions? Opinions?
Bye,
Mike
