Method Chaining Wrapper

wiki

At times we would like to add custom methods to built-in types like strings and functions, particularly when using method chaining [1][2]:

("  test  "):trim():repeatchars(2):upper() --> "TTEESSTT"

(function(x,y) return x*y end):curry(2) --> (function(y) return 2*y end)

We can do this with the debug library (debug.setmetatable) [5]. The downside is that each built-in type has a single, common metatable. Modifying this metatable causes a global side-effect, which is a potential source of conflict between independently maintained modules in a program. Functions in the debug library are often discouraged in regular code for good reason. Many people avoid injecting into these global metatables, while others find it too convenient to avoid [3][6][ExtensionProposal]. Some have even asked why objects of built-in types don't have their own metatables [7].

...
debug.setmetatable("", string_mt)
debug.setmetatable(function()end, function_mt)

We could instead use just standalone functions:

(repeatchars(trim("test"), 2)):upper()

curry(function(x,y) return x*y end, 2)

This is the simplest solution. Simple solutions are often good ones. Nevertheless, there can be a certain level of discordance with some operations being method calls and some being standalone global functions, along with the reordering that results.

One solution to avoid touching the global metatables is to wrap the object inside our own class, perform operations on the wrapper in a method call chain, and unwrap the objects.

Examples would look like this:

S"  test  ":trim():repeatchars(2):upper()() --> TTEESSTT

S"  TEST  ":trim():lower():find('e')() --> 2 2

The S function wraps the given object into a wrapper object. A chain of method calls on the wrapper object operate on the wrapped object in-place. Finally, the wrapper object is unwrapped with a function call ().

For functions that return a single value, an alternative way to unpack is to the use the unary minus:

-S"  test  ":trim():repeatchars(2):upper() --> TTEESSTT

To define S in terms of a table of string functions stringx, we can use this code:

local stringx = {}
for k,v in pairs(string) do stringx[k] = v end
function stringx.trim(self)
  return self:match('^%s*(%S*)%s*$')
end
function stringx.repeatchars(self, n)
  local ts = {}
  for i=1,#self do
    local c = self:sub(i,i)
    for i=1,n do ts[#ts+1] = c end
  end
  return table.concat(ts)
end

local S = buildchainwrapbuilder(stringx)

The buildchainwrapbuilder function is general and implements our design pattern:

-- (c) 2009 David Manura. Licensed under the same terms as Lua (MIT license).
-- version 20090430
local select = select
local setmetatable = setmetatable
local unpack = unpack
local rawget = rawget

-- http://lua-users.org/wiki/CodeGeneration
local function memoize(func)
  return setmetatable({}, {
    __index = function(self, k) local v = func(k); self[k] = v; return v end,
    __call = function(self, k) return self[k] end
  })
end

-- unique IDs (avoid name clashes with wrapped object)
local N = {}
local VALS = memoize(function() return {} end)
local VAL = VALS[1]
local PREV = {}

local function mypack(ow, ...)
  local n = select('#', ...)
  for i=1,n do ow[VALS[i]] = select(i, ...) end
  for i=n+1,ow[N] do ow[VALS[i]] = nil end
  ow[N] = n
end

local function myunpack(ow, i)
  i = i or 1
  if i <= ow[N] then
    return rawget(ow, VALS[i]), myunpack(ow, i+1)
  end
end

local function buildchainwrapbuilder(t)
  local mt = {}
  function mt:__index(k)
    local val = rawget(self, VAL)
    self[PREV] = val -- store in case of method call
    mypack(self, t[k])
    return self
  end
  function mt:__call(...)
    if (...) == self then -- method call
      local val = rawget(self, VAL)
      local prev = rawget(self, PREV)
      self[PREV] = nil
      mypack(self, val(prev, select(2,...)))
      return self
    else
      return myunpack(self, 1, self[N])
    end
  end
  function mt:__unm() return rawget(self, VAL) end

  local function build(o)
    return setmetatable({[VAL]=o,[N]=1}, mt)
  end
  return build
end

local function chainwrap(o, t)
  return buildchainwrapbuilder(t)(o)
end

Test suite:

-- simple examples
assert(-S"AA":lower() == "aa")
assert(-S"AB":lower():reverse() == "ba")
assert(-S"  test  ":trim():repeatchars(2):upper() == "TTEESSTT")
assert(S"  test  ":trim():repeatchars(2):upper()() == "TTEESSTT")

-- basics
assert(S""() == "")
assert(S"a"() == "a")
assert(-S"a" == "a")
assert(S(nil)() == nil)
assert(S"a":byte()() == 97)
local a,b,c = S"TEST":lower():find('e')()
assert(a==2 and b==2 and c==nil)
assert(-S"TEST":lower():find('e') == 2)

-- potentially tricky cases
assert(S"".__index() == nil)
assert(S"".__call() == nil)
assert(S""[1]() == nil)
stringx[1] = 'c'
assert(S"a"[1]() == 'c')
assert(S"a"[1]:upper()() == 'C')
stringx[1] = 'd'
assert(S"a"[1]() == 'd') -- uncached
assert(S"a".lower() == string.lower)

-- improve error messages?
--assert(S(nil):z() == nil)

print 'DONE'

The above implementation has these qualities and assumptions:

The wrapper does not expose most of the wrapped object's metamethods. It's not clear there is a need for this and that it can be implemented correctly. Operators don't follow the method chaining style, apart from the __call and __index operators, which also form the two halves of the method call. Operators like __len are not definable in 5.1 tables. True LuaVirtualization is not possible.
Special care is taken to avoid unintentionally exposing names through the wrapper object and thereby overriding names in the wrapped object. This imposes some overhead that can be avoided with less care or additional table allocation.
Still, the unpacking operation in the wrapper overrides any similarly named operation in the wrapped object. Currently, we use the __call operator () to unpack, which is the only operator allowing multiple return values. The code also supports unary minus as an alternative, which has the limitation of returning a single value (usual case) but perhaps has nicer syntax qualities (S and - together).
Calling and indexing on the wrapper mutate the identity of the wrapped object with the return value of the operation (i.e. side-effects).
Reads from the function table used by the wrapper are not cached (an earlier implementation did have this property).
Building the builder has some additional overhead (fixed cost) to allow the builder to allocate only one table per object (incremental cost).

There are alternative ways we could have expressed the chaining:

S{"  test  ", "trim", {"repeatchars",2}, "upper"}

S("  test  ", "trim | repeatchars(2) | upper")

but this looks less conventional. (Note: the second argument in the last line is point-free [4].)

Method Chaining Wrapper Take #2 - Object at End of Chain

We could instead express the call chain like this:

chain(stringx):trim():repeatchars(5):upper()('  test   ')

where the object operated on is placed at the very end. This reduces the chance of forgetting to unpack, and it allows separation and reuse:

f = chain(stringx):trim():repeatchars(5):upper()
print ( f('  test  ') )
print ( f('  again  ') )

There's various ways to implement this (functional, CodeGeneration, and VM). Here we take the latter approach.

-- method call chaining, take #2
-- (c) 2009 David Manura. Licensed under the same terms as Lua (MIT license).
-- version 20090501

-- unique IDs to avoid name conflict
local OPS = {}
local INDEX = {}
local METHOD = {}

-- table insert, allowing trailing nils
local function myinsert(t, v)
  local n = t.n + 1; t.n = n
  t[n] = v
end

local function eval(ops, x)
  --print('DEBUG:', unpack(ops,1,ops.n))
  local t = ops.t

  local self = x
  local prev
  local n = ops.n
  local i=1; while i <= n do
    if ops[i] == INDEX then
      local k = ops[i+1]
      prev = x  -- save in case of method call
      x = t[k]
      i = i + 2
    elseif ops[i] == METHOD then
      local narg = ops[i+1]
      x = x(prev, unpack(ops, i+2, i+1+narg))
      i = i + 2 + narg
    else
      assert(false)
    end
  end
  return x
end

local mt = {}
function mt:__index(k)
  local ops = self[OPS]
  myinsert(ops, INDEX)
  myinsert(ops, k)
  return self
end

function mt:__call(x, ...)
  local ops = self[OPS]
  if x == self then -- method call
    myinsert(ops, METHOD)
    local n = select('#', ...)
    myinsert(ops, n)
    for i=1,n do
      myinsert(ops, (select(i, ...)))
    end
    return self
  else
    return eval(ops, x)
  end
end

local function chain(t)
  return setmetatable({[OPS]={n=0,t=t}}, mt)
end

Rudimentary test code:

local stringx = {}
for k,v in pairs(string) do stringx[k] = v end
function stringx.trim(self)
  return self:match('^%s*(%S*)%s*$')
end
function stringx.repeatchars(self, n)
  local ts = {}
  for i=1,#self do
    local c = self:sub(i,i)
    for i=1,n do ts[#ts+1] = c end
  end
  return table.concat(ts)
end

local C = chain
assert(C(stringx):trim():repeatchars(2):upper()("  test  ") == 'TTEESSTT')
local f = C(stringx):trim():repeatchars(2):upper()
assert(f"  test  " == 'TTEESSTT')
assert(f"  again  " == 'AAGGAAIINN')
print 'DONE'

Method Chaining Wrapper Take #3 - Lexical injecting with scope-aware metatable

An alternate idea is to modify the string metatable so that the extensions to the string methods are only visible within a lexical scope. The following is not perfect (e.g. nested functions), but it is a start. Example:

-- test example libraries
local stringx = {}
function stringx.trim(self)  return self:match('^%s*(%S*)%s*$') end
local stringxx = {}
function stringxx.trim(self) return self:match('^%s?(.-)%s?$') end

-- test example
function test2(s)
  assert(s.trim == nil)
  scoped_string_methods(stringxx)
  assert(s:trim() == ' 123 ')
end
function test(s)
  scoped_string_methods(stringx)
  assert(s:trim() == '123')
  test2(s)
  assert(s:trim() == '123')
end
local s = '  123  '
assert(s.trim == nil)
test(s)
assert(s.trim == nil)
print 'DONE'

The function scoped_string_methods assigns the given function table to the scope of the currently executing function. All string indexing within the scope goes through that given table.

The above uses this framework code:

-- framework
local mt = debug.getmetatable('')
local scope = {}
function mt.__index(s, k)
  local f = debug.getinfo(2, 'f').func
  return scope[f] and scope[f][k] or string[k]
end
local function scoped_string_methods(t)
  local f = debug.getinfo(2, 'f').func
  scope[f] = t
end

Method Chaining Wrapper Take #4 - Lexical injecting with Metalua

We can do something similar to the above more robustly via MetaLua. An example is below.

-{extension "lexicalindex"}

-- test example libraries
local stringx = {}
function stringx.trim(self)  return self:match('^%s*(%S*)%s*$') end

local function f(o,k)
  if type(o) == 'string' then
    return stringx[k] or string[k]
  end
  return o[k]
end

local function test(s)
  assert(s.trim == nil)
  lexicalindex f
  assert(s.trim ~= nil)
  assert(s:trim():upper() == 'TEST')
end
local s = '  test  '
assert(s.trim == nil)
test(s)
assert(s.trim == nil)

print 'DONE'

The syntax extension introduces a new keyword lexicalindex that specifies a function to be called whenever a value is to be indexed inside the current scope.

Here is what the corresponding plain Lua source looks like:

--- $ ./build/bin/metalua -S vs.lua
--- Source From "@vs.lua": ---
local function __li_invoke (__li_index, o, name, ...)
   return __li_index (o, name) (o, ...)
end

local stringx = { }

function stringx:trim ()
   return self:match "^%s*(%S*)%s*$"
end

local function f (o, k)
   if type (o) == "string" then
      return stringx[k] or string[k]
   end
   return o[k]
end

local function test (s)
   assert (s.trim == nil)
   local __li_index = f
   assert (__li_index (s, "trim") ~= nil)
   assert (__li_invoke (__li_index, __li_invoke (__li_index, s, "trim"), "upper"
) == "TEST")
end

local s = "  test  "

assert (s.trim == nil)

test (s)

assert (s.trim == nil)

print "DONE"

The lexicalindex Metalua extension is implemented as

-- lexical index in scope iff depth > 0
local depth = 0

-- transform indexing expressions
mlp.expr.transformers:add(function(ast)
  if depth > 0 then
    if ast.tag == 'Index' then
      return +{__li_index(-{ast[1]}, -{ast[2]})}
    elseif ast.tag == 'Invoke' then
      return `Call{`Id'__li_invoke', `Id'__li_index', unpack(ast)}
    end
  end
end)

-- monitor scoping depth
mlp.block.transformers:add(function(ast)
  for _,ast2 in ipairs(ast) do
    if ast2.is_lexicalindex then
      depth = depth - 1; break
    end
  end
end)

-- handle new "lexicalindex" statement
mlp.lexer:add'lexicalindex'
mlp.stat:add{'lexicalindex', mlp.expr, builder=function(x)
  local e = unpack(x)
  local ast_out = +{stat: local __li_index = -{e}}
  ast_out.is_lexicalindex = true
  depth = depth + 1
  return ast_out
end}

-- utility function
-- (note: o must be indexed exactly once to preserve behavior
return +{block:
  local function __li_invoke(__li_index, o, name, ...)
    return __li_index(o, name)(o, ...)
  end
}

--DavidManura

Method Chaining Wrapper

Method Chaining Wrapper Take #2 - Object at End of Chain

Method Chaining Wrapper Take #3 - Lexical injecting with scope-aware metatable

Method Chaining Wrapper Take #4 - Lexical injecting with Metalua

See Also