Re: Why one would need mtstates/mtmsg?

[Oliver <oschmidt-mailinglists@...>]

Hi,

On 19.06.20 00:16, Andrea wrote:
> ==== for mtmsg
>
> Does setmsg replace the latest/newest or the oldest msg in the buffer? 
> (from the example on GitHub I would say the latest/newest)

As the documentation says: "Sets the arguments together as one message into the
buffer. All other messages in this buffer are discarded." So all messages in the
buffer including the latest and newest are discarded.

> This is really a great example. I would include it in the documentation.

Perhaps examples and things like this could be put into another document to keep
the reference documentation as compact as possible.

> Thank you for confirming. If possible, I would write a guide for dummies (like
> me) in which you simplify the concept. As an example: instead of "buffer
> referencing object" one can write "handle" (as for files in C) or "reference".

I'm not sure which wording is best, also I'm not a native English speaker...
"handle" is perhaps also correct, althoug with a handle I would associate
something more basic like an integer id that doesn't have methods. The buffer
referencing objects are at least some kind of objects and in daily usage of
mtmsg you mustn't always strictly distinguish between a reference to an object
or an object. buffer:id() gives the integer id to the buffer and
mtmsg.buffer(id) gives an object for using the buffer.

> That raises a question: after I issue an 'abort' operation, is it possible to
> remove all the messages from the buffer so that the listener/reader will never
> process them when I resume regular operation by setting abort to false?

Yes, you can simply invoke buffer:clear() or listener:clear() even if the buffer
or listener is aborted.

>     I'm going to rethink the current solution since I now have the impression that
>     this could be improved (but I'm not sure). Again thank you for your feedback!
> I think the existing implementation is good! Keeping things simple is good.

I already improved something, but yes I'm trying to keep it simple. The listener
still does not own the buffer (and I'm not planning to change this) but if a
buffer is garbage collected the remaining messages of this buffer are still
delivered to the listener. This is already pushed to the github repo ;-)

> If I look at buffers as if they were queues/lists, I would want some method to
> put a message in front of the others that may already be present (this could be
> useful for 'urgent' messages with some higher priority - but one could use
> independent buffers for that). Also, removing messages already added could be
> useful.

IMHO this would make things too complex for a simple library. You can easily
build something like this on top of mtmsg and mtstates using pure Lua. You could
in one mtstates object organize messages in maps and list and re-order them or
anything else. This would be similar to Lane's Lind objects: as far as I
understand the documentation, the Lindas are having a hidden Lua state in the
background.

>     incrementally build up a larger message without locking. Then if the message is
> Do you mean instead of sending a tuple of arguments, build a tuple one element
> at a time?

I mean that you could add incrementally elements to the message. Once you are
finished, the message is send. The receiver could receive the message in one
call like now and will receive all arguments on the stack (there is a limit in
Lua for doing this) or the receiver could also read the message elements
incrementally (by provding an optional integer argument how many elements are
retrieved from the message in one call).

For example:
   buffer:addmsg("foo1", "foo2", "foo3")

would be equivalent to:
   write:add("foo1", "foo2"); writer:add("foo3"); writer:addmsg(buffer);

And:
   local a, b, c = buffer:nextmsg()

would be equivalent to:
   reader:nextmsg(buffer)
   local a, b = reader:next(2)
   local c    = reader:next()


> Will it be easy to manage the situation where multiple writers are building a
> long message in the same buffer, simultaneously?

Yes of course, this is the whole point of it.

> ==== now for mtstates
>
> Let me double check to see if I understand correctly: one can use singleton to
> be sure to own a reference and be sure the state is not garbage collected.
> The thing to avoid is to call singleton twice with the same name from the same
> thread, right?

No this does not cause any problems, the two calls would lead to two state
referencing Lua objects that are referencing the same state.

As you wrote:

  local s = mtstates.singleton("foo", setupFunction)

is nearly the same as:

  local ok, s = pcall(function() return mtstates.state("foo") end)
  if not ok then s = mtstates.newstate("foo", setupFunction) end

BUT: the difference is: the second version will cause race conditions if two
different threads are trying to get the same state "foo" at the same time. So
the main purpose of mtstates.singleton() is to make this an atomic operation
such that it is always the case that the two threads are getting the same
underlying mtstate object in the end (they both should supply the same
setupFunction code of course).

The problem with "owning" or "not owning" and reference cycle is just a side
problem. To avoid reference cycles I decided to have a strict concept of
ownership: the one who created the state via mtstates.newstate() owns the state.
All other threads that are getting the state using mtstates.state() are getting
a weak reference ("not owning"). This ensures that no reference cycles can be
build in normal usage. This strict ownership cannot be done by
mtstates.singleton() since it is not clear who ones the state here.


> new to Lua and I am not sure I fully grasp what a "state" is, for now I am
> thinking it is something that holds the global environment... but I am sure
> there is more - I am sorry if I am missing something important here)

In the Lua C API you can find the datatype "lua_State". This datatype is used
for the Lua main state (this is also holding the global environment) but also
for coroutines (that are referering to the same main state and are sharing the
global environment with the main state). In mtstates a state object is a Lua
main state, i.e. it is holding it's own environment. In the typical Lua multi
threading library (Lanes, luaproc, llthreads) each native system thread contains
a Lua main state and the threaded code is running is the thread's main state.

Best regards,
Oliver