Building Modules

wiki

For instructions on how to build the lua core, see BuildingLua.

Introduction

Building a C extension module generally means that you need to create a shared library aka dynamic link library. This library gets loaded at runtime by require() (new module system in Lua 5.1) or by loadlib() (low-level call, available in Lua 5.0, too).

Static linking of modules against the core is also possible, but not recommended for generic Lua distributions (it makes sense for embedded systems, though).

Here should be a link to a tutorial for the new module system. Someone needs to write it first, though. :-)

Please feel free to correct or extend this list. -- MikePall

A related page CreatingBinaryExtensionModules exists, specific to Windows. It needs to be reviewed and have relevant information incorporated into this page, then deleted.

Build Instructions

Note: Typically, Lua core header files (lua.h, lauxlib.h, etc.) are not stored directly in the standard search path, but rather in a subdirectory. You'll need to specify their location to the compiler by adding -Ilua_header_file_directory to the compile command, which is not shown in these examples. If your platform has pkg-config and associated Lua data installed, you can get the needed compile flags as follows (assuming Lua was installed under package name "lua5.1":

LUA_CFLAGS=`pkg-config lua5.1 --cflags`

shared libraries with libtool

Although instructions specific to several platforms are provided below, on most systems libtool is available and will do the right thing while hiding platform idiosyncrasies.

LIBTOOL="libtool --tag=CC --silent"
$LIBTOOL --mode=compile cc -c module.c
$LIBTOOL --mode=link cc -rpath /usr/local/lib/lua/5.1 -o libmodule.la module.lo
mv .libs/libmodule.so.0.0.0 module.so

Note that cc is an alias for the "main compiler" on most systems. Plug in a specific compiler and optimization setting as necessary. The -rpath value is not critical, but the option is required to trigger a shared library build.

As a bonus, the above commands simultaneously produce a static version of the library at .libs/libmodule.a. (Yes, it's compiled without -fpic.)

dlfcn shared libraries with GCC (Linux, *BSD etc.)

gcc -O2 -fpic -c -o module.o module.c
gcc -O -shared -fpic -o module.so module.o

Add -fomit-frame-pointer on x86 for a significant performance gain (unfortunately this prevents debugging on x86, too). Note: this does not make any sense on other CPUs (x64 aka x86_64/AMD64/EM64T in particular). E.g. gcc -O automatically enables -fomit-frame-pointer on all systems where it does not interfere with debugging. Oh and x64 has no problems with -fpic, too.

Avoid -fPIC since it comes with a big performance penalty on some RISC architectures (Sparc in particular). The linker will warn you when you are exceeding the limits for -fpic and then you have to switch to -fPIC. It's very unlikely that you'll ever hit this limit with Lua modules, though.

Read more than you ever wanted to know about building shared libraries [here] (PDF, ~500K).

dlfcn shared libraries with Sun CC (Solaris)

cc -O -Kpic -c -o module.o module.c
ld -G -o module.so module.o

dlfcn shared libraries with HP CC (HP-UX)

cc +O3 +Z -c -o module.o module.c
ld -b -o module.so module.o

dlfcn shared libraries with IBM XL C (AIX 4.2 or later)

xlc -O2 -c -o module.o module.c
xlc -G -bexpall -o module.so module.o

dlfcn shared libraries on some older Unix systems

cc -O -Kpic -c -o module.o module.c
ld -Bshareable -o module.so module.o

Mac OS 9/X shared libraries (CFM)

You need a patched Lua, with support for oldskool CFM libraries.

For each module, create a matching foo.exp file with a single entry like:

luaopen_foo

Mac OS X bundles (requires Lua 5.1)

Compiling and Linking

MACOSX_DEPLOYMENT_TARGET="10.3"
export MACOSX_DEPLOYMENT_TARGET
gcc -O2 -fno-common -c -o module.o module.c
gcc -bundle -undefined dynamic_lookup -o module.so module.o

It took me a long time to figure this out, but the above link line is not sufficient on (at least) OS X 10.3, and perhaps 10.4. If you have mysterious crashes in your module, attempt to run it with DYLD_BIND_AT_LAUNCH=1 set in the environment. If the module now runs correctly, you probably have a problem with "lazy" initialization of data. Instead of setting the environment variable, add -Wl,-bind_at_load after the -bundle option. In paticular, modules implemented in obj-c, or that call obj-c have this problem, though it appears it can also be an issue with C++ as well.

Note: do not strip the dynamic symbols from the Lua executable (use strip -x). liblua.a was not (and shouldn't be, see notes at end of page) linked into the module, so the bundle will need to find the lua_ API symbols in the executable when it is dynamically loaded.

Loading and Unloading

Modules that use obj-c have another problem, they CANNOT be unloaded, which lua attempts to do during shutdown. Symptom is something like this when lua is exiting:

objc: cannot unmap an image containing ObjC data

Program received signal SIGTRAP, Trace/breakpoint trap.

The only fix I have found for this so far is to remove the __gc method from the _LOADLIB metatable:

static void lua_objc_kill_dlclose(lua_State* L){

luaL_getmetatable(L, "_LOADLIB");

lua_pushnil(L);

lua_setfield(L, -2, "__gc");

}

luaopen_objc(lua_State*L) {

lua_objc_kill_dlclose(L);

... register my module....

}

or to hack loadlib.c so it never unloads C modules.

Module Naming Conventions

For consistency with other platforms the modules should be installed with the .so extension on Mac OS X, too, and this assumed by the default definitions of the package cpath.

Q: Why should other platforms naming affect Mac OS X ? A more natural naming would be: .bundle

A: Because it is a POSIX platform and other multi-platform apps and tools do the same. Please don't confuse marketing ("being different") with developer realities. Multi-platform support is already tough enough.

Comment: it is a reality that .so is split into .dylib and .bundle, on Mac OS X (and Darwin). Perl chose to name them .bundle, Python chose to name them .so. Either way works... Just find the whole "consistency with other platforms" argument to be somewhat lame ?

For those not familiar with the issue, OS X uses a different binary format for statically linked shared libraries and dynamically loaded shared libraries . The former are often called .dylib (libc is a classic example), and the run-time loader will link this in when an application runs. The latter are loaded programatically by application code using dlopen() or NSBundle, the loader doesn't know about them. These are usually called .bundle, but the recommendation above is to call them .so if (and only if) the bundle is a lua module. ELF doesn't distinguish between the two in the binary format, its distinguished by how the .so is used.

Anyhow, it could be considered lame if you are writing code specific to OS X, but most/all of lua's module loading code for other unix-like systems works fine just as it is on OS X. Using a different extension would require more OS X-specific hacks to the lua core. The goal here is to get multi-platform support with a minimum of platform-specific code. Since OS X's library loading facilities don't care what the bundle's extension is, having lua C modules end in .so on all platforms (where this is possible) is easier and works. If you really disagree, you can hack the loadlib.c in lua! Personally, if I was going to throw compatibility away and hack the lua core, I wouldn't use .bundle... I'd use something specific to lua, like ".module"! :-)

Windows DLLs with MSVC

• Use the IDE to create a DLL project (File, New, Projects).
• Select the multithreaded DLL runtime library (C/C++, Category = Code Generation).
• Add the Lua include directory to the include path (C/C++, Preprocessor).
• Add the Lua directory to the library path (Link, Input).
• Add the Lua import library lua51.lib to the input libraries (Link, Input).
• Build the DLL (Build).

Someone please verify/correct this and remove this line. Thanks!

Windows DLLs with GCC (MinGW, Cygwin or MinGW cross-compiler)

gcc -O2 -c -o module.o module.c
gcc -O -shared -o module.dll module.o -Llua_dir -lluaXX

Old GCC versions (pre 4.0) have problems with exception handling and stdcall's when using -fomit-frame-pointer ([bug report]). You can either upgrade or always use -maccumulate-outgoing-args with -fomit-frame-pointer.

Do not strip the relocation info from the DLL (use strip --strip-unneeded).

You may want to add -Wl,--enable-auto-image-base to the link command to reduce DLL load times when you have many extension modules.

Final Notes

Recompile When You Upgrade

Lua generally offers some level of API stability, but not necessarily ABI stability (binary compatibility). This means you need to recompile all your modules when you upgrade the Lua core even between minor versions (e.g. from 5.0 to 5.1). Be sure to always point your compiler to the correct include path (-I...) which contains the Lua core header files you want to compile against.

It's a good idea to recompile everytime when you use the development versions (the 'work' releases), because there are no API/ABI stability guarantees at all.

Do Not Link Modules to the Lua Core Libraries

Although it may be tempting to link modules against the static libraries containing the Lua core (lua*.a), this is not a good idea.

Essentially you are adding a copy of the whole Lua core (or at least large parts of it) to every module. Apart from being a waste of space this may lead to very difficult to diagnose problems. The different instances of the core code don't know anything about each other and may not necessarily be in sync.

In case you built a shared library containing the Lua core (*), please do not link any modules against it, too. I.e. do not specify the name of the Lua core library on the linker line (Windows DLLs are an exception). This makes a hard forward dependency where you really want a lazy backward dependency. Loading this module with a statically linked Lua interpreter would essentially drag in a 2nd Lua core and you get the same problems mentioned above.

Lua modules expect that the Lua core is already present before they are loaded. This works because the Lua core symbols are exported globally (e.g. with -Wl,-E for GCC on most ELF systems -- see the Lua Makefiles).

Related caveat: this is also why you shouldn't link an application with the dynamic library containing a Lua module. This usually does no harm, but doesn't do what you want because the luaopen_foo() function is never called. Modules are to be loaded by require() and not via hard dependencies in the image (package dependencies are ok).

(*) Compiling the Lua core with -fpic on x86 platforms has even more serious performance drawbacks than doing this just for modules (see above). The recommended way (shown in the Lua 5.1 Makefiles) is to link the Lua core statically and export all symbols (enabled with -Wl,-E or by default on Solaris, AIX and others).

Windows and luaXX.dll

The above build instructions for Windows assume that you have compiled the Lua core in a specific way:

• Compile all source file except lua.c, luac.c, print.c and build luaXX.dll (i.e. either lua50.dll or lua51.dll).
• Compile the standalone executables (lua.exe from lua.c and luac.exe from luac.c, print.c) and link them to luaXX.dll.

It's important to compile and link both the standalone executable and every module to the same lua header files and the same luaXX.dll.

A related note: almost every compiler for Windows seems to bring its own standard C library to link against (e.g. the MSVC*.DLL variants). You need to be extra careful that all executables and DLLs working together in a single process share the same C library. Especially when you use malloc and free from different modules (the Lua core carefully avoids this, though).

Stripping Symbol Information

In general I don't recommend to strip binaries. With modern binary formats (such as ELF) and virtual memory systems all debugging info is stored in data pages that never get mapped into memory. Thus all debugging information only takes up space on the hard disk (which is usually abundant), but not in memory. The situation may be different for embedded systems, though. I've only added warnings for systems where you need to be careful with stripping.

Compiler Optimization Levels

The settings listed above are conservative defaults. Of course you can go crazy with -O3 -march=xyz and various -f<something> flags. Do it if your application really needs it. Don't bother, if you have no need.