Re: RE: Pluto updated

["Grisha" <grisha@...>]

Hi, All Interested :)

Why is there a problem with cyclic references? There's no problem as long as
you first add object to already serialized list, and only then actually
serialize it. Consider pseudo-code example:

///////////////////////////////////////

struct A
{
    A *obj;
};

A *a = new A;
A *b = new A;
a->obj = &b;
b->obj = &a;

void A::serialize(serializer)
{
    serializer >> obj;
}

operator >> (serializer , A &*obj)
{
    if(saving)
    {
        if(saved[obj])
            write(saved[obj]);
        else
        {
            int index = ++lastIndex;
            saved[obj] = index;
            write(index);
            obj->serialize(serializer);
        }
    }
    else
    {
        int index;
        read(index);
        if(loaded[index])
            obj = loaded[index];
        else
        {
            obj = new A;
            loaded[index] = obj;
            obj->serialize(serializer);
        }
    }
}

serializer >> a >> b;

///////////////////////////////////////

The above code will cleanly serialize this cycle. This scheme makes file
format practically unreadable though.
Just my 2c :)

Best Regards,
Grisha

----- Original Message ----- 
From: "benjamin sunshine-hill" <bsunshin@usc.edu>
To: "Lua list" <lua@bazar2.conectiva.com.br>
Sent: Tuesday, June 15, 2004 10:21 PM
Subject: Re: RE: Pluto updated


> The big problem with the one-stage approach is that it does not correctly
handle cyclical references, since there's no way to create the first one and
let it refer to the other all in one step. The impact on permanents would
actually be fairly minor, assuming a slightly more complex file format which
could choose to refer to a full object or to a permanent reference. Overall,
though I don't think I'm willing to give up the current level of generality
which allows arbitrary cyclical references.
>
> Ben
>
> ----- Original Message -----
> From: Virgil Smith <Virgil@Nomadics.com>
> Date: Tuesday, June 15, 2004 11:14 am
> Subject: RE: Pluto updated
>
> > Go Ben go!  Sorry (again) that I haven't had time to work on this.
> >
> > One difference in implementation structure I noticed in Pluto as opposed
to
> > what I've done in the past is the 2 stage behavior.  The "reference
table"
> > need not be passed explicitly in the stream.  It can instead be spread
> > throughout the stream.  This eliminates the need for having 2 separate
> > stages for "__persist" specialization.
> >
> > In my implementation this merely meant "counting" objects as they were
> > serialized/deserialized.  As each object was serialized I performed a
check
> > to see if it had already been serialized and if it was then its "count"
was
> > retrieved in the process and could then be used as a reference.  Upon
> > deserialization the list of objects was formed in reverse and any
references
> > that came out of the stream necessarily had to refer to already
deserialized
> > objects.
> >
> > Another advantage of this approach is that references need not conflict
with
> > light-userdata (or any other type) as they can be their own "type" in
the
> > stream which thus signaled that they are a reference and not a "target
> > object".  These values then of course are translated to LUA_NUMBER for
use
> > in the tables of serialized/deserialized objects.
> >
> > A distinct disadvantage to this approach is that it does not integrate
as
> > directly with Pluto's table of "permanents" (sorry I've forgotten the
term
> > you used, but I am referring to your table of items that are assumed to
be
> > present in the deserialization environment).
> >
> > However, this still isn't too much of a departure from the Pluto high
level
> > stream components.  Currently (or rather last I knew) Pluto serializes
the
> > "references table" and then the reference that indicates the top level
> > serialized object.  The switch would just be that first the "permanents"
> > table is serialized, and then the top level object (rather than just a
> > reference to it).
> >
> > Of course this would involve a change in the code architecture that is
not
> > at all insignificant.  Also, I feel like I'm missing some impact on the
> > operation of the "permanents" table.
> >
> >
> >
> > -----Original Message-----
> > From: lua-bounces@bazar2.conectiva.com.br
> > [lua-bounces@bazar2.conectiva.com.br]On Behalf Of benjamin
> > sunshine-hill
> > Sent: Tuesday, June 15, 2004 11:00 AM
> > To: Lua list
> > Subject: Re: Pluto updated
> >
> >
> > Okeydoke, why not. I'll make it a conditional compilation option.
> >
> > Now, one caveat that needs to be worked out. __persist metamethods are
> > invoked during the first stage of persistence (so that all needed
objects
> > referenced by the closure can be visited) but nothing is written out
until
> > the second stage (since not all references are assigned until then). So
in
> > order for this to work, Pluto will have to invoke a special stage-2
persist
> > metamethod during stage 2.
> >
> > Here's my idea: For userdata that need this special behavior, set the
> > __persist mt member to "true". Then, set some other member
> > "__persistlowlevel" to the function to execute during stage 2. The
writer
> > and ud will be pushed on the stack as lightuserdata.
> >
> > Unpersisting is a bit more tricky, since we have to know, somehow, which
> > userdata to invoke low-level unpersist methods for, and what methods to
> > invoke for them. This needs to happen in stage 1 of unpersisting (since
> > that's when we read everything). Normal unpersist methods are executed
after
> > stage 2. Ideas?
> >
> > Ben
> >
> > ----- Original Message -----
> > From: Grisha <grisha@eagle.ru>
> > Date: Tuesday, June 15, 2004 5:10 am
> > Subject: Re: Pluto updated
> >
> > > Hi, Ben!
> > >
> > > We have such a system up and running, and it's not that complex at
all, I
> > > can send it to you if you wish. You are right, Pluto doesn't have to
know
> > > anything about how C++ objects are serialized, same as C-side must
know
> > > nothing about how Pluto does it's job. What I'm asking for is just to
> > > recieve those context parameters passed to Pluto (reader/writer &
(void
> > > *ud)) back in persist/unpersist methods. This will not complicate
things
> > > much for either you, as Pluto developer, or any other programmer
working
> > > with Pluto (they will just ignore extra parameters).
> > >
> > > -- modified text from Pluto\readme
> > > Here's an example of special persistence for an object:
> > >
> > > vec = CreateMyObject()
> > > setmetatable(vec, { __persist = function(oldtbl, writer, ud)
> > > local mt = getmetatable(oldtbl)
> > > local state = DoCSideSaving(oldtbl, writer, ud)
> > >
> > > return function(newtbl, reader, ud)
> > >  setmetatable(newtbl, mt)
> > >  DoCSideLoading(newtbl, reader, ud, state)
> > > end
> > > end })
> > >
> > > Note that original example will work too, since it will just ignore
extra
> > > parameters. We really need this, we can live without function or
thread
> > > persistance, but without proper uderdata persistance we can't.
> > >
> > > Thanks in advance,
> > > Grisha
> > >
> > > ----- Original Message -----
> > > From: "benjamin sunshine-hill" <bsunshin@usc.edu>
> > > To: "Lua list" <lua@bazar2.conectiva.com.br>
> > > Sent: Saturday, June 12, 2004 1:17 AM
> > > Subject: Re: Pluto updated
> > >
> > >
> > > > Hmmm..... it's a tricky issue that you raise. After all, Pluto's
> > designed
> > > to serialize a bunch of interreferencing Lua objects, rather
> > > interreferencing C++ objects. A system capable of serializing C++
objects
> > in
> > > a parallel manner would be complex indeed. Still, I really think that
the
> > > current situation will work. What would need to be done would be for
the
> > > __persist functions to invoke the C++ persistence framework in such a
way
> > > that temporary Lua objects--userdata, most likely--were created to
> > > encapsulate the object state. If objects are shared between different
> > > userdata (using some custom C-side reference counting mechanism, for
> > > instance), some registry-related method could be used to coordinate
> > > serialization between different C object entry points. This is, of
course,
> > a
> > > rather complex method... but it's really not much more complexity than
> > > absolutely necessary to serialize a C++ framework. Pluto would still
> > > automatically handle fixup before running the unpersistence method
> > > > s, of course.
> > > >
> > > > Ultimately, I don't think that the main question is whether the
> > __persist
> > > metamethod writes through a string/userdata or through a writer
object;
> > > that's just semantics, with some performance implications. The real
> > question
> > > is how much control __persist can exert over the serialization stage,
and
> > > how much control it has over internal Pluto placeholder references.
I'd
> > > really prefer to keep those as hidden implementation details; they've
> > > already gone through a couple of revisions, and there are quite a few
> > > caveats that they deal with in order to keep Lua happy.
> > > >
> > > > Ben
> > > >
> > > > P.S. I'm currently working on the string-keys-for-permanents idea
you
> > > proposed. It's proving to be more difficult conceptually than I'd
> > expected,
> > > to maintain flexibility and provide as much user-control as possible,
but
> > I
> > > should have it ironed out in a week or so (travel will be slowing me
> > down).
> > > >
> > >
> > >
> >
> >
> >
> >
>