RE: [ANN] Luabins ? Trivial Lua Binary Serialization Library

["John Hind" <john.hind@...>]

I'm with Luiz on this one and I think your objections are misguided:

1. Lua will already read "bytecodes" through the same channel as source
files, so it had better be robust! If there are known problems with this,
hopefully they will be fixed before too long.

2. You can introduce an infinite loop through source code as easily as
through "bytecodes".

3. The serialisation process would generate the "bytecodes" algorithmically
and so the round trip should be just as robust as using an "arbitrary"
binary format - the process should not write any "bytecode" sequences that
are invalid or dangerous. Luiz's code looks dangerous because it exposes
internals to Lua and does the "heavy lifting" there. However this was just a
proof-of-concept prototype - in final code the serialisation would be done
internally in "C".

4. "Bytecodes" are the only practical way of serialising functions (unless
you decompile them to source code strings). Most complete serialisation
schemes embed "bytecodes" within the "arbitrary" binary format and so
inherit any problems with them.

One real issue however is that the "bytecode" format is not particularly
compact and I guess a "arbitrary" format could be made a good deal more
efficient in storage requirement. It is also worth pointing out that to do
bytecode serialisation at present you need to modify Lua internals (or
duplicate a lot of code).

I am not disparaging your good work, which I am sure has its niche, but if
serialisation is to be added to "Lua the language" as opposed to "Lua the
ecosystem", I think serious consideration should be given to using the
"bytecode" format, possibly improved with some compression, to get maximum
leverage of the existing code base.

See my paper here:

http://lua-users.org/wiki/EngramProposal

Which builds on Luiz's work (although I have not written the code for table
serialisation yet).

** NB I keep parenthesising "bytecode" because I find it a misleading term,
especially as Lua uses 32-Bit "bytecodes". "VM Codes" would be a better term
to use.

-----Original Message-----
From: lua-bounces@bazar2.conectiva.com.br
[mailto:lua-bounces@bazar2.conectiva.com.br] On Behalf Of Alexander Gladysh
Sent: 23 February 2009 01:53
To: Lua list
Subject: Re: [ANN] Luabins ? Trivial Lua Binary Serialization Library

On Mon, Feb 23, 2009 at 4:36 AM, Alexander Gladysh <agladysh@gmail.com>
wrote:
> On Mon, Feb 23, 2009 at 4:18 AM, Luiz Henrique de Figueiredo
<lhf@tecgraf.puc-rio.br> wrote:
> >> I wrote a small and simple library called "luabins" to serialize and
> >> unserialize trivial Lua values (including tables) into arbitrary
> >> binary data format.

> > My approach to this problem is to use Lua bytecode format instead of
> > inventing yet another binary format.

<...>

> I've seen your solution. But I have some problems with it:

> 1. Lua bytecode is unsafe and may crash Lua. <...>

Sorry, I've got carried away. Problem 1 is sufficient in our case.

Unless Lua bytecode is somehow proven as crash-safe as Lua sources, it
may not be used as a data format.

Of course, my custom binary format is not proven safe as well. But
both format and implementation are much simpler than Lua bytecode and
Lua VM -- they are designed to solve much simpler problems.

Also, how do I protect from, say, infinite loops embedded into
bytecode I'm about to load? Putting hooks or pre-parsing it is likely
to degrade performance considerably.

Alexander.

Attachment: smime.p7s
Description: S/MIME cryptographic signature