Re: '__iter', yet again!

[Jay Carlson <nop@...>]

On Wed, Dec 16, 2009 at 10:32 AM, Mark Hamburg <mark@grubmah.com> wrote:

> There's an interesting general argument (meta-argument) there about the notion that the Lua language has a certain number of syntactic/semantic constructs and of those a subset can be overridden via metamethods. The structure of that subset is guided by concerns over issues such as speed, size, and semantics but growing that subset is not a fundamental change in the language though it may introduce new capabilities.

> Besides iteration, some other metamethod extensions that come up include:
>
> * First class treatment in some form for the colon operator -- i.e., allow it to be more than just syntactic sugar
> * Altered semantics for __index to handle recursive usage -- e.g., pass the base table along
> * Altered semantics for __newindex and/or a new metamethod to catch all writes without needing to use a proxy table

The last is just reversion to the behavior in Lua 4.0.  In 4.0, it
always called __index to catch all reads as well.

I found it a lot more *fun* to work with objects in 4.0 because I
tended to think in terms of property accessors with filtering or
triggers.  In 5.x, you need a blank proxy, and then some way to get
from the proxy to the underlying private data.  You can create a fresh
metatable for every object to store data in (now you have two object
identities to manage), or you can gensym a table to use a key in the
blank proxy, and then store your data in there (two objects to manage,
but at least it's containment.) [1]

The blank proxy leads directly to __pairs.  If I know I have an
table-object-like value, I can call o:m() and have something
reasonable happen.  If I'd like to iterate over the values in that
table-object-like value, I can't just iterate over 5.1 pairs(o),
because o is a blank proxy.  If I took the second implementation
approach, I'll even get the private data instead of a reasonable
result.

I believe the current Lua 5.1 attitude is "you should know which
iterator to use". [2]  If I'm only dealing with values inside my own
object system, I can standardize on Object:pairs().  But there isn't a
single root.  t:pairs() fails on arbitrary tables.  A C extension to
work with directories may want me to use os.path.iter(dir).

Lua doesn't need a single object system because, in practice, o:m() is
enough to let many modules make method calls to each other without
having to agree on what an object system even is.  Iterating over
arbitrary table-like objects *does* require knowledge spread out all
over the place now.  I don't see how to solve this without changes to
lbaselib.c or ltablib.c, because any solution requires agreement
between all parties, and the Lua library is the only thing we all have
in common.  Thus __pairs and __ipairs.

> The proposed semantics, if I gather correctly, are "if the value of the first expression is not a function (fast exit path) and it has an __iter metamethod then invoke that metamethod to get the iterator". What happens, however, if there are multiple expressions in the list? Does the rule just not apply? Does the __iter metamethod receive the additional values as parameters?

Gross proposal: make iter a keyword.  Then:

iter k,v in t do
  print(k, v)
end

I want the foreach-style __iter behavior more often than I want the
general for.  Instrument luac to count both forms and report tallies.
We can hold a popularity contest!  ;-)

Jay

[1]: I think weak tables are just as tedious.
[2]: I wrote a weird XML document model with lots of different
iterators over document nodes.  I know there isn't necessarily just
one reasonable iterator choice for a given object.  I think the
strongest argument against trying to solve this "problem" is that most
of the time it makes no sense to arbitrarily iterate over an arbitrary
object; if you have an *object*, you probably have some knowledge
about its behavior.  I don't care.  I want a terse foreach because
most of the time it would be more *fun* to write code in that
language, and I don't see how it will get me into trouble versus
typing pairs() or _p() all the time.  Having fewer noise words on the
screen is good.