[ANN] cffi-lua: a portable (and mostly LuaJIT compatible) C FFI for Lua 5.1-5.4

["Daniel Kolesa" <daniel@...>]

I always liked the LuaJIT FFI, so I decided to mostly replicate it as a plain Lua module, from scratch. I've been working on it for a while and it's just about getting to the point where it's nearly usable.

First, the repository:

https://github.com/q66/cffi-lua

It's based on libffi, so it's portable to most CPU architectures and operating systems (unlike luaffifb, which only works on x86 and maybe ARM). While at it, it adds compatibility with modern versions of Lua, with the functionality exposed depending on which Lua version you're using, so e.g. cffi.metatype will support metamethods supported by the current Lua, and with 5.3 you'll get lossless 64-bit integer conversions for cdata.

It's also written from scratch, including its own C parser, and has minimal dependencies (only meson, lua, libffi, and a C++14 compiler). It does not aim for bug-for-bug compatibility with LuaJIT's parser, and has a bit stricter error checking.

In addition to being buildable as a module, which you can load using the standard require() (it's a pure C module, so all you need is a single .so/.dll file), it's also buildable as a library that you can embed in your thing. That is useful for things that intend to ship it embedded, or use with Lua builds that don't or can't (e.g. Windows UWP, though support for that is currently unimplemented) have dynamic library loading set up etc.

It's tested on Linux, FreeBSD, OS X and Windows, supporting GCC (tested 7+ but should be 5+ or even 4.8+), Clang (tested 8+ but should work on as old as 3.4) and Visual Studio (2017 with updates or newer) toolchains and any architecture supported by libffi. There's Travis CI set up with active per-commit builds/test suite runs on x86_64/ppc64le/s390x/aarch64 Linux (GCC/Clang, Lua 5.1-5.3), OS X (Clang, Lua 5.3) and Windows (mingw-w64 and MSVC++, Lua 5.3), and local testing on some more (32-bit PowerPC and so on).

Additionally, it tries to make not too many assumptions about how the Lua build is configured. Different configurations for lua_Number and lua_Integer from the defaults (e.g. configuring numbers to always be integers) are handled on best effort basis, but not tested. I accept patches if anything is broken, though.

Supported Lua implementations are the reference implementation starting with version 5.1 (it works under LuaJIT itself as well, as that has 5.1 compatibility). It's always written against the latest API, with backwards compatibility being implemented through custom compat funcs centralized in one place. It's been confirmed to work under the latest 5.4 dev snapshots. 5.0 was deemed not worth supporting, but in theory it could be added too.

It's also MIT licensed just like Lua, so you don't have to worry about legal nonsense regardless of your project. There is full documentation as well, including noting all differences from the LuaJIT FFI.

Some of those differences:

- Equality comparisons of cdata and nil always yield false (this is a Lua metamethod semantics limitation; use cffi.nullptr instead)
- Passing/returning unions (or structs containing a union somewhere) by value is not supported (libffi limitation: passing unions is ABI dependent, so pretending it's the largest type won't help - e.g. on x86_64 floats are passed in SIMD registers but a union containing a larger float and a smaller integer will be passed like an integer)
- Bitfields are not supported (libffi limitation)
- There are some API extensions (cffi.addressof etc)
- Parser extensions for 64-bit cdata literals are not available (you can use cffi.eval to get value cdata, though)
- Some functionality is currently still missing
- tonumber() and type() are not overridden to support cdata semantics (use cffi.tonumber and cffi.type)
- Windows calling conventions are not automatically guessed, you need to decorate your Windows API bindings with stdcall properly (which is yet to be implemented)

Feature TODO:

- Support for Windows calling conventions
- Some more C language extensions
- Complex types
- alignas
- non-integer constants
- variable lookups in expressions
- symbol renaming
- vector types
- treat typedefs as qualifiers (to allow 'from typedef to' syntax), same with signed/unsigned (to allow arbitrary order)
- some edge cases with enums
- and maybe a few vendor specific extensions that are still unimplemented
- also optimize and refactor the codebase, it's currently not optimal everywhere, could be made use less memory and so on

Other than that, the test suite really needs growing. If anyone feels like reporting bugs, writing test cases, or something else, that'd be helpful. Currently the module is not battle tested, and I don't recommend using it in production yet. There's also a chance that some semantics may still change.

Future plans:

- Add a C preprocessor
- Even more language extensions?
- Figure out implementing unions by value and bitfields somehow
- Possibly extra features as long as they don't bloat it up too much

Constructive feedback welcome.

Daniel