Re: coroutine.clone and stateful web applications

[Mike Pall <mikelu-0601@...>]

Hi,

Markus Walther wrote:
> Of course, upvalues written to by one clone appear modified in the
> other, as is to be expected.

Yes, if the upvalue is _outside_ of the coroutine context.

> >One semantic problem: open upvalues remain open only in the
> >original coroutine. This is usually what you want, but may yield
> >some semantic surprises (pun intended).
> 
> I don't quite follow. What's the open/closed distinction here (not
> defined in the ref manual, though I see such terminology in the source)?

Ok, this is a bit difficult to explain, but I'll try:

Upvalues are kept open as long as the context holding them is
alive. An open upvalue points to a stack slot. Subsequently
created closures share the same upvalue (full read-write
sharing).

If the context of the upvalue is terminated (e.g. when a function
or a loop ends) the stack slot must be freed or recycled. To
avoid dangling upvalue pointers, the upvalue is closed by copying
the value from the stack slot to an extra area in the upvalue and
changing the internal upvalue pointer to point to this area.

The upvalue struct definition helps to understand what's going on:

typedef struct UpVal {
  CommonHeader;
  TValue *v;  /* points to stack or to its own value */
  union {
    TValue value;  /* the value (when closed) */
    struct {  /* double linked list (when open) */
      struct UpVal *prev;
      struct UpVal *next;
    } l;
  } u;
} UpVal;

The following example creates 10 closures. The context is kept
alive and all share the same upvalue u. If the function exits,
the upvalue is closed, but the closures can still access it and
it will be shared:

function create_table_of_closures_with_shared_upvalue()
  local t = {}
  local u
  for i=1,10 do
    t[i] = function(y) if y then u = y else return u end end
  end
  return t      -- <-- Close of u happens here.
end

In the next example the upvalue is local to the loop. This means
it needs to be closed after every loop iteration. Every closure
gets a unique upvalue and no sharing is going on:

function create_table_of_closures_with_unshared_upvalues()
  local t = {}
  for i=1,10 do
    local u
    t[i] = function(y) if y then u = y else return u end end
    -- <-- Close of u happens here.
  end
  return t
end

There are some more complications with coroutines because an
outer lexical scope may be part of a different coroutine (i.e.
hold upvalues pointing to a different stack). But since only one
context can hold an upvalue open at any time, the algorithm still
works. Garbage collecting a coroutine closes all its open
upvalues, too.

> Can you give an example of a semantic surprise?

Cloning a coroutine duplicates the stack. But it doesn't
duplicate open upvalues because this would violate the invariant
that only one context can hold an upvalue open. Also local access
to the stack slot in the lexical context holding the upvalue
would be unshared, anyway.

A new upvalue is created if the outer lexical scope holding the
upvalue (relative to the yield point where the clone happened)
is continued in the cloned coroutine.

This upvalue is not shared with the upvalue created by the
original coroutine, but it's shared by all closures subsequently
created in the clone:

local resume, yield = coroutine.resume, coroutine.yield
local co1 = coroutine.create(function(u)
  repeat until yield(function(y) if y then u = y else return u end end)
end)
local _, f1a = resume(co1, 1)
local _, f1b = resume(co1)
local co2 = coroutine.clone(co1)
local _, f2a = resume(co2)
local _, f2b = resume(co2)
print(f1a(), f1b(), f2a(), f2b()) -- Read-only sharing is ok.
f1a(2)
print(f1a(), f1b(), f2a(), f2b()) -- But read-write sharing is not cloned.
f2a(3)
print(f1a(), f1b(), f2a(), f2b()) -- A new upvalue is shared by the clone.
local _, f1c = resume(co1)
local _, f2c = resume(co2)
print(f1a(), f1b(), f1c(), f2a(), f2b(), f2c()) -- The split is persistent.

Usually outer lexical scopes holding upvalues are either outside
of the coroutine (this works fine) or they are read-only (like
cached functions). I don't think this is much of a a problem in
practice.

Bye,
     Mike