Re: Why table.pack() does not use __len?

[nobody <nobody+lua-list@...>]

On 2016-07-25 03:58, Rodrigo Azevedo wrote:
> DISCLAIMER: I'm just wondering about this topic
> 
> table.pack() is used to pack "the stack", that can hold nils. To accomplish
> this behaviour it uses the  ".n" field (I'm not analysing the merit here).
> Since this is somewhat arbitrary, look this example:
> 
> [...]
> 
> Why not use __len instead of ".n"?

On top of the table used for storing '...', that will cost an extra
metatable (and in your formulation an extra two closures, but that can
be avoided) per pack.  'n' costs a single key-value entry in the table.
For the common situation of packing '...' into a table to iterate over
it once or do some random access and then throw it away, that roughly
doubles the amount of work for the garbage collector.

table.pack, in the way that it currently works, basically does the
"minimum amount of work" needed to turn '...' into a table without loss
of information.  As such, it can be used as a building block for
producing any other desired representation.  Your version does (and
produces) more work, even if that's not needed.

In the case of long(er)-lived objects, when setting a custom metatable
(e.g. when packing '...' in an object constructor and then applying the
class metatable), your version will also produce extra garbage for
nothing.  That makes it less suitable as a "basic building block".



The same applies to just about any other way to get __len:  You cannot
compute the length, so you have to store it somewhere.  Walking
outwards, we have the following options:

(a) In the table: that's what we currently have with 'n' (just without
the metatable/__len).

(b) In the metatable: That means you need a separate metatable per
table, creating more garbage.  (With some more work you only need one
metatable per length, but that runs into the same problem as the next
variant and constantly replacing the metatable potentially also ruins
the fasttm hack, so you'd be doing a trade-off between code speed and GC
speed by doing what amounts to crappy manual memory management...)

(c) "globally" accessible (e.g. in a table in an upvalue of the
metamethod(s)): With lengths[t] = n, __len(t) --> lengths[t], and
mt(lengths).__mode = "k".  Looks ok at first glance. Problem is, you can
get a _huge_ lookup table.  The way Lua allocates the backing storage
(increase by factor of two), this potentially wastes a lot of memory
(especially now that there can be _really_ many objects because we have
more than 2GB addressable space).  With just over a million objects with
registered lengths, you could be wasting space for a million more,
that's just sitting there, empty & unused.  Worse, even if all the
objects are collected, the lengths table will look empty, yet still
occupy the same amount of space.  (Lua can't silently resize it because
that would break a pairs/next traversal if someone was currently doing
one.  And Lua simply cannot detect that a key (or the string.reverse'd
key or the count of steps to get to that key or...) was remembered
somewhere for later continuing of a suspended traversal.)

So you can essentially choose between warts, garbage factory, or space
hog.  The current choice of Lua/table.pack is warts.