Re: Userdata/Table Unification

[Jamie Webb <j@...>]

On Mon, Jul 26, 2004 at 02:51:00PM +0200, Mike Pall wrote:
> Maybe it's time to start a 'Userdata/Table Unification' thread ...
> 
> The same topic pops up every so often. Summing up all advantages and
> disadvantages in one thread may be helpful. And it may be a first step
> in convincing the authors of Lua to add this to the core.
> 
> Just a few thoughts to start the discussion:
> 
> - Adding a void* to the table structure incurs no additional memory
>   overhead on most systems. I.e. the userdata pointer is free!
>   The current table structure is 32 bytes long (on 32 bit systems) and
>   anything between 29 and 36 bytes leads to the same effective memory
>   allocation due to malloc overhead and alignment constraints. YMMV.

Looks like this hole appears on 16-bit systems as well, but not on
most 64-bit systems (except Itanium) since they still only use 8-byte
alignment.

> - A plain userdata object has 16 bytes (on 32 bit systems) plus the
>   userdata itself plus any malloc and alignment overhead.
>   The savings provided by a trivial userdata object over a unified
>   userdata/table object are negligible. If you really need to save space
>   there is still lightuserdata (I think this type is valuable and should
>   not be affected by the unification).
>   More complex libraries actually need less memory because they no longer
>   need table wrappers, auxiliary tables or per-object metatables.

Makes sense.

> - Current userdata objects are allocated by the Lua core and cannot be
>   resized nor moved nor shared. Quite a few libraries end up storing
>   a single void* in a userdata object. This is wasteful.
>   Leaving the allocation management to the C library seems to be the better
>   solution in the long term IMHO.

The trouble with that is that it forces everyone to write __gc
metamethods, even if they only want the simple case. The
implementation at present caters quite well for both simple uses (e.g.
lmd5), and complex ones, in this respect.

> - Many libraries need to store additional Lua objects along with userdata
>   values and end up abusing the metatable or weak tables. This complicates
>   type checking and/or garbage collection.

Absolutely. I don't think unifying tables and userdatas alone provides
a solution though: that data should be private. If __index and
__newindex were changed to always be called, that would work since
they could completely disallow access to a table (or to certain
members), but that might get expensive.

> - Tables do not have a __gc() metamethod. This prevents wrapping userdata
>   objects into a table because __gc() metamethods often need to have
>   a look at the full object context and not just at individual userdata
>   objects. Unifying userdata and tables would provide such a __gc()
>   metamethod for tables. This is advantageous even for pure Lua code.

I agree that __gc metamethods for tables would be very useful, but
there are complications: the reason AIUI that tables do not have them
is because a destructor might create new references to an object.
Unless Lua code can somehow be prevented from setting __gc (it's not
sufficient to require that it be a C function), probably the only good
solution is to move to a two-stage destruction, where the object has
to be shown unreachable again after the destructor has been called.
This is how Java does it.

One other possibility might be to make the destruction hook not a
metamethod, but give it some specialised C interface. That would also
make it easier to create a convenience function that created and
allocated a userdata and handled its deallocation, given a change to
pointer-only userdatas.

> - Current userdata objects cannot be subclassed easily. Unless you have
>   the cooperation of the C library you need to wrap them up in a table and
>   store any subclass fields there. But this approach is troublesome
>   due to the lack of a __gc() metamethod for tables. See above.
>   Subclassing gets trivial with a unified userdata/table object.

Yes, provided there is no private data in the table that the
subclassing might clobber. 

> - Every unified table object has a void*. This may be useful for many more
>   scopes than just plain library objects (e.g. module namespace tables).

True.

> - I have not taken a closer look at the issues that the unification rises
>   for the Lua garbage collector. I may be wrong, but I think it gets simpler
>   because the tricky handling for userdata objects can be dropped.

Why? The reasons for that tricky handling don't seem to change.

> - Reducing diversity usually simplifies the implementation and this leads
>   to fewer bugs. Some people believe this is the only measure that counts.

...provided the complexity of what's left doesn't increase too much.

-- Jamie Webb