Re: LuaJIT: referring to Lua values from C/FFI?

[Josh Haberman <jhaberman@...>]

Mike Pall <mikelu-1104 <at> mike.de> writes:
> Cdata objects do not hold interior pointers to GC objects. This
> means they don't need to be traversed by the GC. If you view the
> entire state of all Lua objects as a graph, they are always leafs.

Good to know.
 
> I was under the impression that protocol buffers are mostly
> transient objects. Then I'd worry more about the speed of access
> than their memory usage.

The vision I have for protocol buffers is far broader than just messaging.
For example, I plan to use them as ASTs and a lot of the things XML
and JSON are often used for today.  Space efficiency is definitely
important.

> > One idea I had was to have a userdata but use its metatable to store
> > the references to other Lua values.  That requires a metatable per
> > instance which is a little extra overhead and it means that the
> > luaL_checkudata() scheme won't work for type checking, but overall
> > it's the best idea I have so far.
> 
> Avoid writing to metatables after their initial creation. This
> kills the internal negative cache for metamethods on every write
> (this is true for Lua, too).

Good to know.

> Also, one metatable per instance would defeat any kind of
> metatable specialization (if I were to implement that). Use the
> environment table of userdata to store per-instance data.

I totally didn't know that every userdata has an environment table -- that's
perfect for this purpose!

> > I forgot to mention one related idea I had: the protobuf data itself
> > could live in a separate data structure that the userdata only has a pointer
> > to.  In that case the Lua object is just a wrapper, and I can cache these
> > Lua objects in a weak table indexed by the address of the actual data
> > object (ie. keyed by lightuserdata).  When I want to go from one Lua
> > object to another, I look up the sub-object in the weak table by address.
> 
> Umm, isn't this overdesigning a bit? Yeah, ok, so is the whole
> protobuf thing. Sorry, I couldn't resist. 

I can offer with <50k of object code the capabilities of a whole stack
of XML specs (XML, Schema, Namespaces) at 20x the speed.  I know,
I know, picking on XML is way too easy, but protobufs are actually
pretty lean and solve the serialization problem better than anything
else I've ever seen.  They have a killer combination of efficiency and
ease of use.

> But wouldn't it make sense to first study the access patterns and
> then adapt the data structures?

To me that's like trying to study the access patterns of C "struct."
Applications will do such wildly divergent things that you want
something that is efficient in all cases.

I will offer more specialized APIs for special cases, like a streaming
API (ie. SAX for XML) and an API for selecting a few specific values
from a message.

> Assuming that every element is only
> accessed once, it might be easier to decode each one on-the-fly
> from the binary representation.

Protobufs aren't seekable, so this would have quadratic cost.

Josh