Re: ipairs in Lua 5.3.0-alpha

[Jan Behrens <jbe-lua-l@...>]

On Sun, 17 Aug 2014 15:00:14 +0200
Jan Behrens <jbe-lua-l@public-software-group.org> wrote:

> There is a problem here: Sometimes you want to write a function
> (my_function) that accepts a sequence of values (e.g. to apply
> some functions to that sequence's values). There are several
> ways of accepting such a sequence that I can think of:
> 
> [...]
> 
> So when you say "iterators are not one of those things where I need
> consistency across libraries", I would dissent: It's really sad that
> there is no consistent interface to pass "iterable" values. The
> __ipairs metamethod of Lua 5.2 could be seen as such interface. Here,
> it's also defined whether an incrementing integer is returned as first
> value by the iterator triplet, or not:  It is included!  Therefore,
> every function knows how to handle the output of the ipairs iterator.
> The function my_function in the examples above could be implemented
> like this:
> 
>   function my_function(seq)
>     for i, v in ipairs(seq) do  -- assuming __ipairs is respected
>       -- do something with "v" here
>     end
>   end
> 
> And this function could work for a lot of different types (including
> raw tables that don't have a metatable set).
> 
> [...]
> 
> Additionally, improved_ipairs could even be further extended such that
> it accepts any form of iterator triplet (not just simple functions):
> 
> do
>   local function my_ipairs_funcaux(f, i)
>     local v = f()
>     if v then
>       return i + 1, v
>     else
>       return nil
>     end
>   end
>   function supercool_ipairs(x, s, i)
>     if type(x) == "function" then
>       if s == nil and i == nil then
>         return my_ipairs_funcaux, x, 0
>       else
>         local n = 0
>         return function()  -- closure is not avoidable here
>           n = n + 1
>           local j, v, v2, v3, v4 = x(s, i)
>           -- needs C implementation for variable number of values v1..vn
>           if j == nil then
>             return nil
>           else
>             i = j
>             return n, j, v, v2, v3, v4
>           end
>         end
>       end
>     else
>       return ipairs(x)
>     end
>   end
> end
> 
> 
> Considering all this, my proposal for Lua 5.3 would be to make ipairs a
> generic interface for (ordinal) iteration. That would mean:
> 
> * Keep the __ipairs metamethod to allow customized behavior
> 
> * Allow functions to be passed to the global ipairs function
>   (according to one of the implementations above:
>   improved_ipairs or supercool_ipairs)
> 
> I'm unsure, however, how the default ipairs should work if __ipairs is
> undefined but if __len and/or __index are defined. I guess it's a
> matter of taste. Having the default ipairs respect __len would require
> multiple evaluation of the length (as shown before). On the other hand,
> if we keep having __ipairs, then this problem could be circumvented if
> we really want/need to.


I just coded a more elaborated example for this "supercool_ipairs",
which I would like to share with you:

============================================================
do

  local function ipairsaux_raw(t, i)
    i = i + 1
    local v = rawget(t, i)
    if v then
      return i, v
    else
      return nil
    end
  end

  local function ipairsaux_meta(t, i)
    i = i + 1
    local v = t[i]
    if v then
      return i, v
    else
      return nil
    end
  end

  local function ipairsaux_metalen(t, i)
    i = i + 1
    local v = t[i]
    if v ~= nil then
      return i, v
    else
      -- Roberto's idea: evaluate len only if v == nil
      if i <= #t then
        return i, nil
      else
        return nil
      end
    end
  end

  local function ipairsaux_func(f, i)
    local v, v2, v3, v4, vn = f()
    -- variable arg number requires C implementation
    if v then
      return i + 1, v, v2, v3, v4, vn
    else
      return nil
    end
  end

  local empty = {}

  function supercool_ipairs(x, s, i)
    local mt = getmetatable(x) or empty
    local mt_ipairs = rawget(mt, "__ipairs")
    if mt_ipairs ~= nil then
      return mt_ipairs(x)
    elseif type(x) == "function" then
      if s == nil and i == nil then
        return ipairsaux_func, x, 0
      else
        local n = 0
        return function()  -- closure not avoidable here
          n = n + 1
          local v, v2, v3, v4, vn = x(s, i)
          -- variable arg number requires C implementation
          if v == nil then
            return nil
          else
            i = v
            return n, v, v2, v3, v4, vn
          end
        end
      end
    else
      local mt_call = rawget(mt, "__call")
      if mt_call then
        return supercool_ipairs(mt_call, s, i)
      elseif rawget(mt, "__len") ~= nil then
        return ipairsaux_metalen, x, 0
      elseif rawget(mt, "__index") ~= nil then
        return ipairsaux_meta, x, 0
      else
        return ipairsaux_raw, x, 0
      end
    end
  end

end
============================================================


Now this is possible:

(you can copy/paste this to your Lua interpreter, if a file named
"testfile" exists)

============================================================
do

  local function printentries(...)
    for i, v, v2 in supercool_ipairs(...) do
      if v2 == nil then
        print(
          "Entry #" .. tostring(i) ..
          ": " .. tostring(v)
        )
      else
        print(
          "Entry #" .. tostring(i) ..
          ": (" .. tostring(v) .. "," .. tostring(v2) .. ")"
        )
      end
    end
  end

  local letter = nil
  local function my_iterator()  -- some example iterator
    if letter == nil then
      letter = "a"
    elseif letter == "z" then
      return nil
    else
      letter = string.char(string.byte(letter) + 1)
    end
    return letter
  end

  local abc = {"a", "b", "c"}

  local lines = assert(io.open("testfile", "r")):lines()

  local tbl = {key1 = "value1", key2 = "value2"}

  local sparse_array = setmetatable(
    {[3] = true, [5] = true},
    {__len = function() return 10 end}
  )

  local shadow = {"alpha", "beta", nil, "delta"}
  local simple_proxy = setmetatable({}, {__index = shadow})
  -- NOTE: doesn't use __len

  print("printentries(lines):")
  printentries(lines)
  print()

  print("printentries(my_iterator):")
  printentries(my_iterator)
  print()

  print("printentries(abc):")
  printentries(abc)
  print()

  print("printentries(supercool_ipairs(abc)):")
  printentries(supercool_ipairs(abc))
  print()

  print("printentries(pairs(tbl)):")
  printentries(pairs(tbl))
  print()

  print("printentries(sparse_array):")
  printentries(sparse_array)
  print()

  print("printentries(simple_proxy):")
  printentries(simple_proxy)
  print()

end
============================================================


Giving the following output (assuming that "testfile" exists):

printentries(lines):
Entry #1: This is line #1 of my testfile.
Entry #2: This is line #2 of my testfile.

printentries(my_iterator):
Entry #1: a
Entry #2: b
Entry #3: c
Entry #4: d
Entry #5: e
Entry #6: f
Entry #7: g
Entry #8: h
Entry #9: i
Entry #10: j
Entry #11: k
Entry #12: l
Entry #13: m
Entry #14: n
Entry #15: o
Entry #16: p
Entry #17: q
Entry #18: r
Entry #19: s
Entry #20: t
Entry #21: u
Entry #22: v
Entry #23: w
Entry #24: x
Entry #25: y
Entry #26: z

printentries(abc):
Entry #1: a
Entry #2: b
Entry #3: c

printentries(supercool_ipairs(abc)):
Entry #1: (1,a)
Entry #2: (2,b)
Entry #3: (3,c)

printentries(pairs(tbl)):
Entry #1: (key2,value2)
Entry #2: (key1,value1)

printentries(sparse_array):
Entry #1: nil
Entry #2: nil
Entry #3: true
Entry #4: nil
Entry #5: true
Entry #6: nil
Entry #7: nil
Entry #8: nil
Entry #9: nil
Entry #10: nil

printentries(simple_proxy):
Entry #1: alpha
Entry #2: beta


The two big advangages here are:

* The interface is always the same:

  ipairs(lines)
  ipairs(my_iterator)
  ipairs(abc)          -- aka ipairs{"a", "b", "c"}
  ipairs(ipairs(abc))  -- Note the nested ipairs
  ipairs(pairs(tbl))
  ipairs(sparse_array)
  ipairs(simple_proxy)

* It doesn't require much change to Lua at all. There are no
  changes for the language itself, just minimal changes for
  the base library in lbaselib.c


Of course, this "supercool_ipairs" could be also implemented outside of
Lua's baselib. But its true power would be only available if there was
a general agreement on a common interface (__index, __len, __ipairs).

Otherwise there is a mess (as explained in my previous post) when every
library defines its own iterator interface.


... looking forward to feedback from you

Jan