Re: LuaJIT 2 ffi (casting / force hotpath)

[Mike Pall <mikelu-1101@...>]

Florian Weimer wrote:
> Is there a way to mark af FFI calls which may trigger callbacks (via C
> extensions)?  What's the general story about callbacks?

Right now the call chain 'FFI -> C -> lua_*()' with the same
lua_State is a big no-no. I'm planning to add an extra GCC-like
attribute that could be used to mark C functions which do that.
So you can still call these functions from Lua, but it's only done
from the interpreter (e.g. for the entry into a GUI main loop).

[You may however call luaL_newstate() from the FFI to start up an
_independent_ Lua state and play with it. Works fine.]

Converting a Lua function into a callback for a C function is a
different issue (the canonical example would be qsort()). This is
quite tricky and has some inherent issues, e.g. dynamically
generated trampolines cannot be deallocated. So I've postponed
this for now.

> Do you plan to make popular POSIX types (off_t, time_t, struct
> timeval, struct timespec), constants (those for errno, for instance)
> and errno itself available without meddling with header files?  It's
> great if you can get this data from header files, but reading them has
> downsides (performance, availability at run time, compatibility issues
> caused by new GCC extensions).

Once I add an internal pre-processor to the C parser, you could
just ffi.cinclude("sys/types.h") (don't try this, NYI). Though
that doesn't really help for deployment, as not every installation
provides header files.

One possibility would be to convert headers into Lua files while
LuaJIT is built, like h2ph does for Perl. But that has its own
share of problems. And should it really convert all headers from
the developer's system? I don't think so.

Another option would be to add a ffi.osdef convenience module,
which has all of the generic POSIX types, or the LPCTSTR mess for
Windows. The latter wouldn't be too bad, because the Windows types
are cast in stone. But the former requires tricky pre-processing
to get the needed OS-specific types without including lots of
extra cruft.

And errno is an entirely different story: all modern libc's are
multithreaded and provide their own (non-standard) mechanism for
getting errno. The only real spec is the errno macro, which
contains C code of course. Figuring this out for all OS and libc
combinations is going to be difficult. And even if you do, it's
probably not safe to simulate this from Lua at a higher level.
Some memory allocation or a debug hook could get inbetween and
clear the errno value.

So I've thought about adding an __errno attribute. If you declare
a C function with this attribute, you get the errno as an extra
return value (local ok, errno = ffi.C.mkdir(...)). This is fetched
by low-level code immediately after the function returns. Alas,
this means I have to figure out all of the variants for getting
the errno value myself and teach them to the JIT compiler. :-/

--Mike