Re: Userdata and Serialization [From: Feature request: plain option for gsub]

[Coroutines <coroutines@...>]

On Sun, Aug 24, 2014 at 12:05 PM, Andrew Starks <andrew.starks@trms.com> wrote:

> Coro:
>
> A couple of things. You should really look at Nanomsg. It has features that
> provide for zero-copy inproc and IPC _may_ be zero copy on Windows, but it
> has been a work in progress.
>
> BUT, you have to use their variants of free and alloc: `nn_alloc` and
> `nn_free`. So, I assume you mean something that is somewhat similar, but for
> Lua.
>
> We've been puzzling with serialization, ourselves. I thought for a moment
> about a Lua type called "buffer", which would essentially be the same as a
> string, but not interned.
>
> But, then as I poked around more, I came to believe that there probably
> isn't much point in doing that. It would almost certainly require some
> amount of control over the allocator and at that point, you're better off in
> C.
>
> A quick example: it would be better for us to use Nanomsg's memory
> management functions, not Lua's, so that we can gain 0-copy IPC. So, those
> values are always pointers in our UD, which wouldn't make sense in that
> model.
>
> So, we decided to use RIFF in our nanomsg messages. The outer RIFF is always
> a "MSGP" (or something), which is associated with our msgpack library, which
> parses the rest of the message, creating a table of Lua values, as it goes.
>
> For userdata, we use an imbedded RIFF (usually [RIFF] = blob). The blob is
> understood to be a userdata that follows a very simple format, which is
> flexible enough for any opaque type. We use this UD format for everything we
> do, which makes serializing and deserializing more-or-less standard.
>
> In some cases, we can keep the UD in nanomsg's space (represented as a
> pointer in the UD). We do this when we're modifying the content and thus
> creating a new UD as the result.

Thank you for your post, I will have to take a closer look at nanomsg :-)

In regards to the serialization, IPC thing: I realize that it is much
simpler to pass around data like this through channels.  A lot of
complexity is avoided by avoiding shared, "raw" data.  I just want to
make the harder one more accessible.  Thanks for going into detail on
how you use nanomsg <3