Re: Custom extensions to Lua

[Rici Lake <lua@...>]

On 11-Aug-05, at 9:17 AM, Lisa Parratt wrote:

> Rici Lake wrote:
> > Lisa,
> > I think what you are trying to do is semantically incoherent, so it 
> > is not surprising that it is difficult. It's semantically incoherent 
> > because it confuses values with references to values ("boxes").
>
> Not so much confuses, as intentionally seeks to make the two 
> semantically identical to the end user. The goal of the project is to, 
> in so far as possible, seamlessly integrate the native elements with 
> Lua.

I think you mean "syntactically identical" :) If so, I sympathize but I 
think you are going to end up fighting a semantic impedance mismatch.

If I understand you correctly, you'd like to be able to write in Lua 
something vaguely like Glenn Maynard's useful example of C++:

 locked<int> value = 1;  // constructor
 value = 2;              // mutator

which might translate into something like:

 local value = locked_int(1) -- constructor
 value = 2                   -- mutator

C++, as has been noted, embeds a little visual pun (several, actually, 
but I won't get into the other ones): the semantics of '=' are quite 
different in the two lines quoted (which I've annotated). In the first 
line the '=' is a syntactic element, whereas in the second case it's an 
operator.

Lua is somewhat more like Objective-C (or even C, if you think of all 
Lua locals as having the type LuaObject* rather than LuaObject) (note 
[1]). A Lua assignment is always an assignment; mutation needs to be 
expressed as a function call.

Consequently, in Lua the semantics of

  local value = locked_int(1)
and
  local value
  value = locked_int(1)

are the same. (The second one differs in that value is set to nil for a 
very short amount of time, but the difference is not user-visible.) A 
subsequent

  value = 2

changes 'value' from being a locked_int into being a regular number.

We all have our prejudices (particularly me) and perhaps "semantic 
incoherence" was an overstatement: I meant that the semantics is 
incoherent with the syntax (note [2]). Providing a syntactic 'mutate' 
operator, as proposed by Andrew Lauritzen in 
<http://lua-users.org/wiki/LuaPowerPatches>, strikes me as more 
semantically consistent (and I don't see why it is "un-Lua-like" 
either):

  local howmany = locked_int(1)
  howmany <- 3              -- mutate with lock
  howmany = locked_int(3)   -- new lock-protected value

The syntactic difference between lines 2 and 3 reflects a real semantic 
difference (note [3]) and it is not difficult to get used to, in 
practice.

> This is exactly how those aspects of the system operate. The main 
> issues occur when one is trying to proxy strings, integers, and 
> non-global variables. Locals aren't bound to a table,

OK, I've written enough about locals. As far as integers and strings 
go, I honestly think you'd be better off converting them between C and 
Lua at the call-boundary rather than trying to proxy them. For 
integers, that is straight-forward as long as lua_Number is double or 
your integers are short. For strings, the conversion requires a copy 
(possibly two copies), but that is not awful unless the strings are 
long. (Furthermore, since Lua interns strings, it can avoid the copy in 
the event that it already has an instance of the string.) On the whole, 
this is probably less inefficient than the metamethod comparison 
approach: you can copy quite a few bytes in the time that it takes Lua 
to start up a metamethod.

Notes:

[1] Internally, Lua avoids malloc'ing simple scalars (numbers, 
booleans, nil) and this often leads people to say that Lua 
distinguishes between scalars and composites. Here's a representative 
quote from the mailing list:

> Lua forces copy semantics for simple value types (nil, boolean, number,
> string, light userdata) and reference semantics for complex types 
> (function,
> table, userdata).

I don't think this aids understanding of Lua's semantics. Unlike Lua, 
Python allocates numbers and keeps pointers to the heap-allocated 
storage, but the semantics of Python numbers and Lua numbers are pretty 
well identical. Lua strings (which are immutable) are not actually 
copied even though they are heap-allocated.

The point is that if a datatype is truly immutable, it is not really 
possible to semantically distinguish between two (hypothetical) copies 
of it. If the datatype is mutable, then operations such as 
object-equality comparison and argument passing are naturally based on 
object identity. Storing a number or a boolean instead of storing a 
pointer to a heap-allocated instance of the same value is really just 
an optimization (or at least an attempt at optimization); not a 
semantic difference.

[2] Lua is not immune to visual puns. The semantics of '=' in the 
following two statements are radically different:

  local a = 3
  global_a = 3

Like all visual puns, this creates confusion.

[3] Also, Andrew's patch does not implement the unboxing operator '*', 
and I believe it uses a different symbol to represent the mutation 
statement.

The locked<T> example is interesting in other ways. For example, in 
order to get it to work correctly, it's necessary to overload all 
mutation operators:

  locked<int> howmany = 1;
  howmany++; // this is not the same as howmany = howmany + 1;

But what if the mutation is not anticipated?

  int complex_calculation(int a, int b);
  howmany = complex_calculation(howmany, 42); // Oops

There are several possible solutions to this problem:

1) lock howmany, do the calculation, change howmany, unlock howmany

2) save the value of howmany in a temporary, do the calculation, 
atomically change howmany if it still has the same value as was saved.

I don't think either of these is "right" or "wrong"; it would depend on 
how long the calculation was likely to take and how much contention 
there is on the lock. I'd probably go for the second one in most cases, 
though. I'll leave it to one of the C++ meisters on the list to come up 
with an ultracool C++ syntax for this, but I note that Andrew's patch 
allows the mutator to accept multiple values, so that the following is 
possible:

  do local temp = *howmany
    howmany <- temp, complex_calculation(temp, 42)
  end

or even:

  function protected(val, func, ...)
    return val, func(val, ...)
  end
  -- ...
  howmany <- protected(*howmany, complex_calculation, 42)