Re: [ANN] LuaJIT x64 port sponsorship program

[Florian Weimer <fw@...>]

* Mike Pall:

> Florian Weimer wrote:
>> I raised this on the x86_64 ABI mailing list.  The belief over there
>> is that all 2**64 pointer values are valid per the ABI.  It's just
>> that *addresses of objects* will always fall in the 2**48 sub-range.
>
> Well, C99 makes no such distinction. It ties the definition of a
> valid pointer to where it can legally point to.

Right, and seen in that light, LuaJIT2's behavior matches the spec in
the Lua reference manual (if you assume that it's invariant under C
extensions, and not covariant---I hadn't considered that possibility
before).

>> Lua could do something similar to Objective Caml here.  To certain
>> routines in their VM, you may only pass valid pointers to objects you
>> created outside the VM (or pointing to somewhere in those objects).
>> All other pointers can be mutilated by the GC.  But adopting this
>> would change the Lua API.
>
> A way to pin some pointers (interior and/or exterior) would at
> least pave the way for a moving GC. That would be the prerequisite
> for a fast bump allocator and a generational GC.

There are non-compacting generational collectors.  For example, the
main collector in Hotspot's low-latency, concurrent mark-and-sweep
collector (CMS/-XX:+UseConcMarkSweepGC) is mostly non-moving.
Compaction is used only as a fallback if the collector cannot recover
sufficient space, and you'll likely miss your latency targets if it
happens (this part is stop-the-world).

Generational GCs do need some sort of write barrier, but that is not a
problem with the Lua C API (and one already exists, of course).

And a bump-pointer allocator will not be able to immediately reuse
free space created by table resizing.

> Something like lua_pin_touserdata() and lua_pin_tolstring() might
> work. It would be a no-op right now, since the current guarantees
> for pointer stability are more restrictive.

lua_pushstring doesn't return a pointer to the string, so objects can
be created unpinned.  lua_tosltring could pin them.  When the C
function returns, the string is unpinned again.  The result would
match what is described in the reference manual ("there is no
guarantee that the pointer returned by lua_tolstring will be valid
after the corresponding value is removed from the stack"), but it will
probably break some existing code.

Userdata is different, you really shouldn't move that because I
suspect many extensions use lua_newuserdata as a special form of
malloc.

>> Another option would be to box light userdata if it's outside the
>> 2**48 range.

> Comparing the complexity cost vs. the benefit, I'd think this is
> not a good deal.

Yes, that seems the case.