Re: Pluto wishlist

[Peter Cawley <lua@...>]

On Thu, Jan 21, 2010 at 4:02 PM, Ben Sunshine-Hill <sneftel@gmail.com> wrote:
> On Thu, Jan 21, 2010 at 5:48 AM, Peter Cawley <lua@corsix.org> wrote:
>>    * Pluto's options for [de]persisting userdata are limited. The
>> literal persistence of userdata is inappropriate, as userdata based on
>> C/C++ structures may be different sizes or endianness with different
>> compiler settings and different platforms. This leaves just the fixup
>> Lua closure option, which is appropriate for some userdata, but not
>> for others. I would like to be able to have metamethods on userdata,
>> which operate on a persistence API to write (and later read) arbitrary
>> binary data and Lua objects.
>
> I'd be interested to hear what types of userdata you found that can't
> be appropriately handled through the closure method, as when I
> originally designed it I spent quite a while trying to think of
> something that it couldn't handle, and came up dry. In particular,
> your serialization system's method of userdata handing -- manually
> convert userdata to a bytestream and inject it into the file, then
> manually deserialize it on loading -- can be easily modelled in Pluto
> by a closure which encapsulates a call to the deserialization method,
> and a string to pass to it. TBH, my own dislike of the special
> persistence system was not its expressive power, to which I wasn't
> able to discover any important limits, but its wordiness.

Definitely, the two ways of serialising userdata are equivalent in
terms of their functionality (the manual method can choose to write a
single Lua object - the fixup closure, and the fixup method can write
a table containing alternating strings containing arbitrary binary
data and Lua objects).

My primary motivation of something which isn't suited to the fixup
closure approach is a C++ 2-dimensional array containing 2^14
elements, where each element has 7 integer fields and 2 lists of 0 or
more Lua objects (with references to said objects also from other
places outside the userdata, which are persisted at the same time).
This class is implemented as userdata because when it was implemented
in pure Lua, garbage collecting it took a very noticeable amount of
time - this would be mostly mitigated by Lua's incremental garbage
collector, but as an instance of this class is always in existence, it
always has to be garbage collected in full when the application exits,
resulting in the application appearing to hang for several seconds
when asked to close. This class could be persisted by creating a very
large table, and then persisting said table wrapped as a fixup
closure, or alternatively a string containing a large binary blob and
a large table containing all the object references. It would take some
amount of time to convert the C++ data structure to a Lua data
structure, and then take time in deserialisation to convert back, and
would also give the garbage collector a lot to work on after
serialisation and deserialisation - possibly affecting application
close time if the application was instructed to close immediately
after a deserialisation. Serialising this data structure manually
feels a lot more efficient than using a fixup closure as the data can
be written directly to the bytestream from C++ rather than being
converted to a valid Lua data structure before going into the
bytestream.