Userdata and Serialization [From: Feature request: plain option for gsub]

On Sat, Aug 23, 2014 at 1:58 AM, Coroutines <coroutines@gmail.com> wrote:

On Fri, Aug 22, 2014 at 11:33 PM, Sean Conner <sean@conman.org> wrote:

> Okay, now I'm totally confused as to what you are trying to do. Let's
> take, for example, LPeg.
>
> > lpeg = require "lpeg" -- I'm using 0.12 BTW
> > x = lpeg.P "Coroutines"
> > print(type(x))
> userdata: 0x8835e7c

> Okay, what exactly, are you trying to modify?

Okay. From Lua you have a userdata object. You can use # to get the
size of that, assuming __len isn't doing anything deceptive. I want
to be able to string.sub() userdata, so I can read
character-by-character. Promoting a lua_Buffer/userdata to a string
hashes the userdata and from then on it's known as a string to Lua,
and I cannot modify it. But I don't need that hash or the inherent
deduplication -- I know that the data in that userdata is unique and
its lifetime is short (it's a packet!). I want to be able to read
userdata from Lua without touching C -- without writing a function to
dump userdata as a string. I want to operate *ON* the userdata, not a
string copy. The point is avoiding more allocation when I just need 1
buffer that is very volatile.

> typedef struct lua_State lua_State;
>
> and that's it. No indication of how big it is, or what's inside it. Try to
> compile the following:

I am not talking about opaque types in C :>

> I also don't fully understand what you mean by:
>
>> I dream of a world without data channels and serializing things

The cost of serializing an object and sharing it through some method
of IPC is more than it needs to be. I take issue with the concept of
it. When I have brought this up before I'm told to look elsewhere for
inefficiency, but I am someone who likes to solve benign problems.
Also lock-free algorithms are a thing. Message-passing will never be
as quick as zero-copied, shared data ;>

Coro:

A couple of things. You should really look at Nanomsg. It has features that provide for zero-copy inproc and IPC _may_ be zero copy on Windows, but it has been a work in progress.

BUT, you have to use their variants of free and alloc: `nn_alloc` and `nn_free`. So, I assume you mean something that is somewhat similar, but for Lua.

We've been puzzling with serialization, ourselves. I thought for a moment about a Lua type called "buffer", which would essentially be the same as a string, but not interned.

But, then as I poked around more, I came to believe that there probably isn't much point in doing that. It would almost certainly require some amount of control over the allocator and at that point, you're better off in C.

A quick example: it would be better for us to use Nanomsg's memory management functions, not Lua's, so that we can gain 0-copy IPC. So, those values are always pointers in our UD, which wouldn't make sense in that model.

So, we decided to use RIFF in our nanomsg messages. The outer RIFF is always a "MSGP" (or something), which is associated with our msgpack library, which parses the rest of the message, creating a table of Lua values, as it goes.

For userdata, we use an imbedded RIFF (usually [RIFF] = blob). The blob is understood to be a userdata that follows a very simple format, which is flexible enough for any opaque type. We use this UD format for everything we do, which makes serializing and deserializing more-or-less standard.

In some cases, we can keep the UD in nanomsg's space (represented as a pointer in the UD). We do this when we're modifying the content and thus creating a new UD as the result.

-Andrew