RE: Engram Proposal for Serialisation - RFC

["John Hind" <john.hind@...>]

Many thanks for this David, useful points all!

One improvement is suggested by your comments already: Conceptually it
should be possible to cache a table payload in an upvalue of the engram's
closure so the table only has to be constructed once per de-serialisation.
Not sure if it is possible for a function in a closure to expand that
closure, but worth looking into!

Any limitations of this scheme should be the limitations of Lua as a DDL -
there is no essential difference between this representation and the result
of compiling a constructor. If a version upgrade does not break the dump
format for existing uses it should not break it for this use either. But
this is certainly not aimed at Pluto level game state saves, more persisting
user settings or relatively modest control messaging.

Database queries and table synchronisation were very much in my mind when I
put the enclosed function capability in the proposal. You can send an engram
from A to B such that when its enclosed function is executed on B it
extracts just the required data from a large table and sends a reply engram
containing the extract back B to A. Or an engram can store a differences
table along with the code required to update a much larger table at the
destination.

Of course this flexibility may raise security concerns in some applications!

Space efficiency is not too bad - although obviously there is an overhead.
Without a function for a Boolean: 72 bytes, 79 for a Number, 76 + # for a
String. For a simple function that just returns its parameter, this rises to
181, 188, 185 + # (this seems to include debug information which it may be
possible to strip out). However I'm sure someone will point out the
corresponding Lua source code would take less (in most cases), which I admit
was a bit of a shock!

-----Original Message-----
From: David Manura Sent: 05 February 2009 06:26

This could have its uses.  Here's some things to consider:

  - Any limits on data structure size imposed by the implementation?
  - en() must be evaluated in entirety in order to extract any part of the
data
    (implications for large data structures).
  - Evaluating en() could take an indeterminate length of time: e.g.
    function en() for i=1,1e10 do end; return 1 end
    (implications for DOS)
  - Space efficiency of data representation?
    Compare to Pluto, ASN.1, Google Protocol Buffers, etc.
    (BTW, the libraries for these can be relatively large/complex.)
  - Noncanonical (non-unique physical representation -- see "Canonical
XML").
  - Not intended for long-term storage (e.g. versioning).
  - The format of Lua bytecode is not rigorously specified apart from the
    No Frills Guide (alternative: more portable Lua source).

Attachment: smime.p7s
Description: S/MIME cryptographic signature