Re: LuaJIT 2 ffi (casting / force hotpath)

[Mike Pall <mikelu-1101@...>]

Michal Kottman wrote:
> On Sat, 2011-01-22 at 21:49 +0100, Florian Weimer wrote:
> > Wouldn't the humongous size of C++ header files causes problems with
> > application startup time?
> 
> The idea was to prepare the headers beforehand and convert it into some
> easily digestable format, like some form of a key-value database. We
> already do the parsing while generating the bindings, and it's
> relatively fast (i.e. not more than a second of processing on my 2.4GHz
> processor).
> 
> Let's say that a user wanted to create qtgui.QPushButton. An __index
> metamethod would be triggered and the definitions would be read from the
> database and constructed so that LuaJIT FFI could parse it. The
> resulting type would be saved back in qtgui.QPushButton and used to
> construct new objects.
> 
> This way, we would not have to distribute the whole generator, only a
> single 'live-binding' module and a pre-processed database of types.

Thanks for the inspiration! I think I know how to handle the
deployment problem for the LuaJIT FFI now:

On the developers machine:
- Feed all of the headers to the C/C++ parser of the LuaJIT FFI.
  This may be somewhat slow of course.
- Then dump the whole internal database of C type declarations in
  some kind of output format.
- To enable fast access, this would need to be a mix of a hash
  table (for names) and a direct index (for C type IDs). The FFI
  internally uses something like this, but it needs to be extended
  to work with non-interned strings.
- The ideal format would be an object file, which can either be
  statically linked or turned into a shared library. The data
  structures should be pure read-only data without the need for
  relocations (indexes instead of pointers).
- Generating C code that holds a huge const array is probably the
  easiest way. A small luaopen_*() wrapper could be added, so it
  behaves like any Lua/C module.

On the users machine:
- Just load the module containing the pre-compiled database with
  require.
- Internally, whenever the namespace of the module is indexed, it
  recursively copies/interns the needed declarations from the
  read-only data to the FFI declarations database and then runs
  the standard C library namespace handler.
- This should be quite fast, because the internal FFI data
  structures already have support for interning. Also, it works
  incrementally and only the needed definitions end up in
  read-write memory.
- Since the read-only data is shared across processes and pages
  are loaded on-demand by the kernel, the impact on the memory
  footprint would be limited.
- One could shrink the declaration database further by adding some
  kind of post-execution analysis tool.

Alas, this sounds like quite a bit of work ...

> I'm sure this idea could work on smaller libraries, but now it seems to
> me that Qt is too complicated for this (it has it's own nuances, like
> the signal-slot mechanism, parent-child memory management etc.)

That's why I said Qt has a rather quaint idea of C++. :-)
But given the rigid rules for the Qt code base, it's still more
manageable than trying to handle all of C++.

--Mike