Re: embedding lua in a multithreaded application

[Nick Hildebrant <nick.hildebrant@...>]

Also Andres, you said one Lua thread per socket. What does that look like. Later you say that Lua is safe for use in multiple threads. So I'm guessing that one Lua thread per socket just means establishing a new state, or is there some other magic I'm missing?

On Sun, Apr 19, 2015 at 12:52 AM, Nick Hildebrant <nick.hildebrant@gmail.com> wrote:

Thanks Andrew, that does help a little. I just assumed this was a common problem with a common solution which might be documented somewhere. Did you have any help getting you to your solution, or just read the code? So far, that's all I'm really doing...

On Sun, Apr 19, 2015 at 12:30 AM, Andrew Starks <andrew.starks@trms.com> wrote:

On Saturday, April 18, 2015, Nick Hildebrant <nick.hildebrant@gmail.com> wrote:

Hi, I am a pretty new to Lua still, so please forgive me :)

I am wanting to use lua in a multithreaded application, and I can find lots of information about how to do threaded / psuedo threaded things in Lua, but this is not what I am looking for. My understanding is, that the interpreter/jit is single threaded, and I want to know how to ensure that I am not blocking my multithreaded app by executing all my scripts in a single thread.

If it helps, my goal is to use LuaJIT with ZeroMQ, and have zmq manage the parallel workflow of a bunch of services written in Lua. I have looked at the wrapper lua-zmq, and I assume this is using lua in a truly multithreaded way, but I don't know how to confirm. What is the trick, what do I look for? Do we have to start an interpreter in every thread, or is it enough just to create a new state?

When I look at the C code where people are embedding Lua in their programs, I don't ever really see something that looks like "the interpreter starts running here", just setting up a new Lua state.

Is using the LuaJIT to wrap zmq the same as embedding LuaJIT in a program which links against ZMQ?

I am happy to read more, I don't need anyone to write a book here, but when I google "mutlithreaded Lua" I get a bunch of answers to a different question (namely, "how to i make my lua script multithreaded?") which doesn't interest me at all. I actually want to write single threaded Lua code which will execute in the multithreaded message passing way which ZeroMQ manages so well.

Thanks,

Nick

There are probably many patterns that you can follow here, but I use one Lua thread per socket (or group of related sockets). To communicate between Lua threads, I use nanomsg (but did use zmq, so it's basically the same scenario). I do this because I have Lua front-ending some long running processes, and so I do need the concurrency.

But, you don't even need to do that, if Lua isn't the bottleneck. `select` (or better on non-windows).  You can make zmq's buffering sufficiently large, if you're trying to keep upstream processing from blocking, when Lua is busy. 

Does any of this provide any clues? Lua is not multithreaded but the library is careful to be safe for use from multiple threads, so long as each Lua state is being used by one thread at a time. 

-Andrew