what to make of this fellow named __ipairs?

[David Manura <dm.lua@...>]

What are we to make of this fellow named __ipairs in 5.2.0-alpha?

The Reference Manual specifies the mechanics of `__ipairs` [1] but not
its intended purpose.  This leads to many questions....  How should
`ipairs(t)` behave if `t` represents a sparse array? a 0-indexed
array? an instance of an OO class?  Should the keys always be numbers?
integers? returned in a particular order?  What should we do if we
need both forward and reverse iterators for an object?  Similarly, for
`__pairs`, it may tell you in arbitrary order the values of `k` such
that `o[k]` is defined (e.g. methods and fields), or perhaps it tells
you other values of `k` not exposed via the indexing operator (e.g.
things conceptually contained in the object).  Due to this ambiguity,
for arbitrary objects `__pairs` may best be limited to introspection
or debugging purposes, like how `__tostring` is often not intended for
the rigors of serialization.  For some custom object I probably would
just create my own iterator with its own more descriptive name.

At one point `ipairs(t)` was redefined in terms of `for i=1,#t`, but
this was rejected [2].  Perhaps that is good for cases where you don't
want to start at 1 or stop at #t or when random access may be
inefficient (linked list).

One motivation for `__ipairs` is to allow a piece of code to be
polymorphic.  `f(t)` will work identically regardless whether `t` is a
raw table, proxy, or userdata.  However, this doesn't work in
practice.  table functions ignore metamethods.  Even `next` ignores
metamethods, and `next` is arguably as important for hash tables as
'#' is for array tables since `next(t) == nil` is a common idiom to
test for the null set.   Then some have specifically asked for
`rawlen` and `rawtostring`, adding to the quagmire.

A reason we write `ipairs(t)` in the first place is that it's not
always suitable *in general* to make it a method, `t:ipairs()`, as we
might do in other languages.  In Lua, `t:ipairs`'s existence implies
`t.ipairs ~= nil` and `t["ipairs"] ~= nil`.  Therefore, we make it
instead a *meta*-method `__ipairs`, which `ipairs` uses to achieve a
polymorphism similar to a method call.  IMO, maybe what we need, more
generally, is a more convenient notation to call metamethods, as
opposed to `for k,v in getmetatable(o).__pairs(o) do ...`.  Most of
the time, however, it is just fine to expose the iterator through a
regular method, conveniently attached to the object, so we have
`o:iter()` for some custom object `o`.  Also, in Lua you can always do
things like `local ipairs = myipairs; for i,v in ipairs(t) do ...`.

On performance, __ipairs could slow down iterations, although my test
below indicates it doesn't really.  At the very least, it slightly
complicates the implementation and human interpretation.

  local t = {}; for i=1,1000 do t[i]=i end
  for j=1,100000 do
    for i,v in ipairs(t) do end
  end

Finally, `pairs` and `ipairs` are not essential functions, so we could
avoid the issue by just eliminating them (although that was tried and
rejected [3]).  Anything you can do with these can also be done via
`next` and `rawget`.  The only reason I sometimes use `ipairs` is that
code is a bit shorter than using `#` even though less optimal.
`pairs` can always be replaced with `next`.

So, `__ipairs` leaves me a little perplexed.

[1] http://www.lua.org/work/doc/manual.html#pdf-ipairs
[2] http://lua-users.org/lists/lua-l/2010-08/msg00011.html
[3] http://lua-users.org/lists/lua-l/2010-05/msg00416.html