Building extension modules

[Mike Pall <mikelu-0412@...>]

Hi,

since we have that new flashy module standard in place for Lua 5.1
we should think about a standard and portable way to build and install
extension modules (both Lua and C modules, where the latter is the more
difficult part).

I had a little E-Mail conversation with Diego about this and we both
identified the need to have something like a 'lua-config' program or
similar functionality. This would allow a developer to modify a single
installed file and put all knowledge about the local build system into it.

When we standardize on something this could be used by all 3rd party
extension modules and ease the task of adding modules to your local
installation considerably.

And extension module authors would benefit, too. It just does not make
sense to reinvent the wheel (the build process) in thousand different ways
and then see every wheel break on a different platform the author hasn't
thought about.

However I'm not sure what the best approach for this would be. Here are
a few candidates:

- Many OSS packages now include a '${package}-config' program or a *.pc
  file for pkg-config. This is usually a shell script which prints
  stuff like the required compiler or linker options.

  Alas, this approach is mostly useful for autoconf style builds of
  applications that need a specific library. And it is more or less
  limited to POSIX systems unless you have a fully-fledged shell at hand.

  And for building extension modules you want to know different kind of
  things, too: e.g. how to build a shared or static library and where
  and how to install it.

- A variant of this would be to derive the required info from the package
  table. E.g. the install directory for modules is more or less already
  present there, hidden in package.path and package.cpath. It would be
  pretty simple to explicitly store this and other info in other elements
  of the package table.
  
  Then retrieving these is as simple as: lua -e 'print(package.foo)'

  But then you still have the knowledge about the build process split
  between a makefile and some Lua variables (or functions for more complex
  stuff). It is fairly unlikely that you'll succeed in making this work
  for the whole range of systems where Lua runs on and the associated
  build processes (think about cross-compiling and get worried).

  Yet, having programmatic access to some key elements required for the
  build process is a step in the right direction.

- Python replaces the whole build process with the 'distutils' package.

  You basically get programmatic access to a set of classes that have
  all the required functionality (dependency checking, calling the compiler,
  the linker and even the OS specific packaging system). Your extension
  has to include a tiny Python file that imports distutils and passes
  a few key information elements (such as the C or Python files that
  are part of your extension). Of course you have all the power of a
  programming language at hand to change the build process.

  As an extension author you don't need to write a makefile anymore.
  When building extensions you don't need the make program anymore.
  As a local admin or developer you can change the local setup in one place
  and all extension modules are then built incorporating your changes
  to the process.

  I like this approach because you can add arbitrary complex rules and adapt
  the whole build process to your needs. After all a makefile is just a bad
  excuse for a turing-complete language -- so why not use a 'real' language?
  Since you obviously (*) must have one at hand when you are building
  modules for it (Python in this example, Lua in ours).

  However this approach is a bit heavyweight (9.000 lines of Python code)
  since it incorporates a lot of knowledge about the tool chain (compiler,
  linker, archiver, OS packager) and reimplements make functionality.
  It also depends on a lot of system dependent functionality to call
  subprocesses or check the local OS environment. No problem for Python
  here with its giant library, but somewhat of a problem for Lua.
  
  Still it's doable and we could start small.

  (*) Not so obvious for cross-compiles though. But Lua qualifies since
      it is very likely to compile both on the developer host and the
      target host.

- Perl has an abominable system that creates a Makefile on the fly and
  does preprocessing on source files. I found this difficult to integrate
  and maintain (IMHO of course). Maybe it's just me or maybe it could
  be adapted to integrate better (like generating an includable Makefile).

- Ruby has mkmf.rb and seems to create a makefile on the fly, too.
  However I have no experience in building modules for Ruby.

- There are other build systems (like a-a-p) that have their own flavour
  of a configuration language and claim better portability than makefiles.
  However they need to be installed before you can build an extension.
  And you are basically stuck on systems where the build system is not
  available.
  
  I'd rather have something self-contained, written mostly in Lua and
  maybe a few OS specific parts in C. Since this is a rather lightweight
  install, it is likely that we can convince everyone to install it and
  accept this as part of the tool chain for building Lua extensions.

- And then there's the possibility (or need) to integrate this with IDEs.
  Think about Eclipse or the countless Windows tools for this purpose.
  
  I feel not qualified to comment on this other than to say that it's
  probably a bad idea to depend on any of these tools. Still, it would
  be nice for the build process to work smoothly within these environments.

Of course this list can be extended ...

Maybe we should collect both requirements and solutions systematically
and then somehow seek to find the best compromise that satisfies many or
even most needs.

So what are the needs for your build process? Anyone has stories to
share about building for different platforms? Suggestions? Opinions?

Bye,
     Mike