Re: [ANN] Prosody 0.8.0 released

["Robert G. Jakabosky" <bobby@...>]

On Friday 08, Matthew Wild wrote:
> On 9 April 2011 02:58, Robert G. Jakabosky <bobby@sharedrealm.com> wrote:
> > On Friday 08, Matthew Wild wrote:
> >> Ed sent this to me off-list, but suggested that he shouldn't have as
> >> my response was worth a wider audience.
> >> 
> >> In fact I am also interested to know if anyone has experience of the
> >> issues below, or the various libraries. I did try LuaLanes a couple of
> >> years back, but only got it to deadlock. Now that it is somewhat more
> >> maintained now I should look into it again when I get the chance.
> > 
> > I have created a light weight threads module for Lua [1].  Each thread
> > gets it's own lua_State instance and there is no locking/shared state.
> >  To communicate between running threads you have to use a communications
> > module (LuaSocket or the preferred lua-zmq [2]).
> > 
> > My ZeroMQ bindings (lua-zmq [2]) has a wrapper object 'zmq.threads' [3]
> > for the llthreads module, which makes it easy to start threads from a
> > block of lua code or a lua file.
> 
> Unfortunately ZeroMQ isn't currently recommended for use on the public
> internet, which is why I have stayed away from it. We need to
> implement clustering support in Prosody anyway (multiple machines
> hosting the same logical service) and sometimes the nodes are going to
> be distant. I'd rather not rely on firewall rules to keep Prosody safe
> in such a case.

ZeroMQ is mainly used for use on private LANs or between threads & process on 
the same host.

> It doesn't seem to make sense to me to have multiple means for Prosody
> processes to communicate.

I would make the node-to-node communication into a pluggable module.

> >> * Thread pool for specific blocking tasks, such as storage (few
> >> storage truly async storage APIs are available)
> >>   This is currently our favoured approach. We just scale out the few
> >> parts we know can be bottlenecks, the majority of work still happens
> >> in one main thread, meaning no locking or other threading issues can
> >> appear so easily.
> > 
> > Creating a pool of worker threads with lua-zmq would be easy to do.  Also
> > the ZeroMQ sockets can be added to the main event loop to get notices
> > when work is finished.  See the 'handler.zmq' [8] object for how to
> > embed a ZeroMQ socket into an lib-ev event loop.  It shouldn't be to
> > hard to port that to the libevent interface.
> 
> Thanks, I wasn't aware this was possible.

It is a little bit tricky since zmq sockets are edge-trigged instead of level-
trigged like normal sockets.

> >> * What I call the "magic" LuaProc/erlang style, where we can just put
> >> each session in a coroutine, and let the scheduler deal with the
> >> multi-CPU scaling.
> >>  This is attractive as it requires very little work on the threading,
> >> we can just focus on using coroutines and message passing. It should
> >> also be possible to easily scale down to 1 thread.
> > 
> > It would be interesting to see a distributed version of ConcurrentLua
> > that used ZeroMQ for thread-to-thread & host-to-host message passing.
> 
> It would indeed. Let me know when it's released :)

Haha, I don't plan on making it, but I think it would be interesting to see.


> I think we're at the stage where the best way to optimize is by
> cutting down the amount of serialization and interning of network data
> we do, e.g. by storing incoming data and parsing in-place, and using
> the source buffer to write out the other side if we made no
> modifications. daurnimator and I had a bit of a hack session chasing
> this idea, and the results were interesting but still need some work
> to get what we're after. I'm considering taking advantage of the FFI
> now to save interning and re-serializing stuff we don't need to.

If you could read data from the network into a re-usable buffer and parse the 
data directly from that buffer, that should save some work interning the raw 
buffer into a Lua string.

> >> More of a priority is scaling across multiple machines - ie. having
> >> multiple Prosody instances serving the same domain (for load-balancing
> >> and reliability). This would also allow multiple Prosodies to run on
> >> one machine and share the CPUs.
> > 
> > ZeroMQ would allow you to scale from multiple threads to many servers.  I
> > would recommend diagramming the flow of messages from client connections
> > to different parts of the 'backend' services (i.e. storage, database,
> > message routing services).  If you separate those shared components into
> > units that communicate with messages, then you can either keep them
> > local or move them to different threads or servers.  Also if you use the
> > design patterns from the ZeroMQ guide [6] you can scale each backend
> > service from one instance to multiple instances as needed, without
> > changing the code.
> 
> Interestingly this is how the earliest XMPP daemons were constructed,
> except using XMPP between the components. It's certainly an
> interesting option, I only worry about how much code (and complexity)
> it'll add for something that the majority of deployments don't need.

Make the interface of those backend services pluggable, then you can plugin a 
connector backend that allows the backend to be remote.  For simple single-
server or single-thread deployments, just plug the backend directly in instead 
of using the connector.

-- 
Robert G. Jakabosky