Re: [ANN] Lua 5.4.0 (work1) now available

On Wed, Mar 14, 2018 at 9:24 PM, Roberto Ierusalimschy wrote:

The problems I am trying to solve are these:

1) A constructor like {x, y, z} should always create a sequence with
three elements. A constructor like {...} should always get all arguments
passed to a function. A constructor like {f(x)} should always get
all results returned by the function. '#' should always work on these
tables correctly.

2) A statement like 't[#t + 1] = x' should always add one more element
at the end of a sequence.

...

t[i]=undef is a special syntax form that means "remove the key 'i' from table 't'"

Remark #1
At first sight, I like this idea with undef.
Syntax is a bit confusing, but it is a step towards more convenient
work with lists/tuples.
nil is not merely a substitution for missed values anymore.
nil now is a full-fledged value!

Question:
Once we could have nil as a valid value in tables,
could we use nil as a valid key?
t[nil]=42

Remark #2
At second sight, this "undef" idea has big problems with compatibility.
If I understood correctly, in Lua with NILINTABLES table keys
with nil values will NOT be GC-ed.
To release a key in a table, undef must be pseudo-assigned instead of nil.
Unfortunately, this breaks most of old Lua code.
Almost all Lua projects will need to change its internal logic
to avoid memory leakage. (Memory leakage here is about having a lot of
unneeded keys with nil values which are not available for GC)
Yes, searching for literal nil assignment will find most of places
we should change.
But there would be a lot of hard-to-find places such as
   -- func may return nil sometimes
   t[k] = func(k)
or
   -- intentionally skips nils and stores only non-nils
   table.insert(t, get_next_val())
Rewriting old projects would be hard and error-prone.
We all have a lot of old code we don't remember in details how it works.
The price of rewriting Lua 5.3 code to Lua 5.4 will be too high.

Remark #3
Probably, metatable-based approach would be convenient?
We could define special field "__undef" in table's metatable for storing
a value which is used to delete table entry from this table.
If there is no "__undef" metafield, then the value nil is used to delete
table key (this behavior is compatible with previous Lua versions).
But in Lua 5.4 we could store nils by assigning non-nil value to "__undef":
   t = setmetatable({}, {__undef = 'A'})
   t[1] = nil   -- adding key 1 with value nil
   assert(#t == 1 and t[1] == nil)
   t[1] = 'A'   -- deleting key 1 from the table
   assert(#t == 0 and t[1] == nil)
But what if user needs string 'A' in this table?
To avoid accidental collision, we could generate unique __undef value:
   t = setmetatable({}, {__undef = {}})
To delete a key:

function table.delete(t, key)

t[key] = (getmetatable(t) or {}).__undef

end

__undef metafield also affects #/next/pairs/ipairs:
iterating must skip all pairs (key, value) with value equals to __undef.

The interesting question is about table constructors:
   t1 = {1, 2, nil, 4, a=nil, b=nil}
   t2 = { func() }
Table constructors are applied at the time when a table doesn't have
a metatable yet, so table constructor must store all values it has
(including nils).

   t = {a=nil}
   setmetatable(t, {__undef = t})
   -- the following line will print "a", nil
   for k, v in pairs(t) do print(k, v) end

   t = {nil}      -- table actually has key 1 with nil value

setmetatable(t, {__undef = t})
print(#t) -- prints 1

The backward compatibility is guaranteed due to next/pairs/ipairs
respect __undef metafield.

   t = {a=nil}    -- table actually has key "a" with nil value
   -- the following line prints nothing due to next() skips
   -- values equals to __undef (__undef==nil)
   for k, v in pairs(t) do print(k, v) end

   t = {nil}      -- table actually has key 1 with nil value

print(#t) -- prints 0

   -- the following line prints nothing due to ipairs() returns
   -- an iterator which skips values equals to __undef (__undef==nil)
   for k, v in ipairs(t) do print(k, v) end

As you can see, this approach allows storing nils in tables
without syntax compatibility problems and without new keywords.

Both Roberto's problems (see quoted text) are solved
(for problem #2 user must prepare metafield prior to assigning nil).

You can prepare simple global metatable:

_NILINSIDE = { __undef = {} }
and use it for every table which you want to insert nils into:

local function f() return nil, nil end

local t = setmetatable( { f() }, _NILINSIDE )

t[#t+1] = nil

assert(#t == 3)

Assigning a metatable is verbose, but only some tables need this feature.

More details:
1) __undef metafield is compared by rawequal().
2) You can change __undef metafield on-the-fly, it may (or may not,
depending on the implementation) delete existing keys which have
value equals to __undef.
3) While iterating with ipairs(), depending on implementation,
__undef may be read once at the moment of ipairs invocation
or may be reread on every iteration.
4) next() may (or may not, depending on the implementation)
delete skipped table items from the table
(that is, keys having value equals to the current __undef metafield)
5) Iterator returned by ipairs() may (or may not, depending
on the implementation) delete skipped table items from the table