Iteration and nils

["Greg Falcon" <veloso@...>]

The recent thread about lists and nil got me thinking about the ways
in which nil isn't a first-class data type in Lua.  Of course, the
focus of that thread was on sparse arrays and vararg lists.  I've been
thinking about another limitation of nil -- that the generic for loop
doesn't allow an iterator to provide nil as a value.

This is a very unfortunate limitation in the face of multiple-value
expressions.  There's simply not a good way to iterate over all the
values resulting from ... or from a function call.  It seems to me
that there can't be one, since nil is a legal value which iterators
can't emit.  This is highly unfortunate when writing a vararg
function: surely iterating over the provided arguments should be easy.

PiL2 suggests two approaches for looping over varargs.  The first approach,
  for i, v in ipairs{...}
is straightforward, but of course it doesn't support nils.

The second, use of select("#", ...) and select(n, ...), has two major
problems.  First, it involves passing N+1 arguments to a function N+1
times, so performance for large data sets is abysmal.  Second, it
would be nice to be able to iterate over any multiple-value
expression.  Unfortunately, the select() approach only works in the
vararg case.  You couldn't use the same approach for iterating over
the values returned by a function, because it would mean calling that
function N+1 times.

Passing or returning nil, and passing or returning nothing at all, are
two different operations.  It occurred to me, however, that this cause
of the problem could also be the solution.

What I mean is this.  I'm proposing a version of Lua where nil gets no
special treatment in the generic for loop.  It could be returned by an
iteration function just like any other Lua value, and it would not
terminate the loop.  To indicate the end of iteration, the iterator
function would have to return nothing.

I think the ability of iterators to emit nil would be generically
useful.  Notice that If this was supported, it'd be possible to
provide a standard library function that iterates over a multi-value
expression:
  for v in values(...) do
or:
  for v in values(f()) do
which fixes the nil problem and seems an even clearer syntax than
  for _,v in ipairs{...} do.

I see problems with my idea.  The implementation of OP_TFORLOOP would
become more complicated.  This change could break existing code
(notably, this change breaks user-written iterators valid in Lua 5.1).
 The exact behavior of generic for could no longer be replicated with
simple Lua code, at least not without calling select().

I'd like to clarify that I'm not advocating nil as a valid table key.
next(t, nil) and next(t) have to stay equivalent; I think the
alternative would break too much code.  On the other hand, next{} must
be changed to return nothing instead of nil.  I can't imagine that
would break any real code.

Am I totally crazy?  I think I might be, but I can't shake the idea.

Greg F