lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]

You don't understand: Lua programs are parsed only once then executed or natively compield from the bytecode only, strings are no longer used except for loading the program the first time, but never when instanciating objects written in that language. There's no marshalling/unmarshaling between Lua functions calls, references are working natively (via indexes in tables stored in the heap or via closures in the stack, it works directly at binary level, and there's never any need to marshall/unmarshall complex data structures or serialize them by building and parsing strings all the time).

I'm not convinced this is the right approach, even when using the LuaC libary, there's a better way to do that without this huge overhead (which also requires lot of heap allocations for these temporary scripts containing everything: the complete source code of all functions the task needs to do, or the code that will use the loader, and all the data they'll work with and that they'll finally return).
We can do that with LuaC by using existing Lua "thread" objects, to which we can attach metadata for adding info about how the threads can operate and can be scheduled.
The LuaC library is needs only to allow creating new threads with theior own Allocator and setup the metadata attached to threads and possibly some "userdata" object to tune the Allocator behavior.

You need however small changes in the Lua bytecode interpret so that it can test a flag indicating that there's a pending interrupt for which the bytecode should stop interpreting continuously, but should insert a call to yield() before continuing with the next instruction. Finally your C library will be used to create the OS-level timer that will just asynchrnously set the interrupt flag, without modifying directyly the state of any running Lua thread. You also need a way for threads to handle some critical sections using some basic test-and-set instruction (this can be implemetned as an unary boolean operator, similar in syntax to "not" except that it must occur before arbitrary expressions but only before a variable name, like with the "local" instruction; this operator can use a new reserved keyword like "testset" or a new lexical token like "?!"; the variable will bet set to a boolean or nil, it does not need a new type, but it will compile a new bytecode instruction which will be atomic and just needs one memory operand or a register index in the Lua closure's window on the stack)

If we don't define this additional bytecode, there's no possibility to synchronize concurrent threads that can preempt the exuecution while working on shared objects, leaving them in inconsistant state (this limits a lot the possility for real multithreading when working with shared data structures, or when performing I/O, so I'm convinced we need some mutex mechanism). Adding thys additional bytecode is a trivial modification to the VM.

However we don't need a pure scheduler with relative priorities between threads. All that is needed is to allow threads to run really concurrently without having them to be modified so they include many "yield()" calls every where in their code or within all their loops or before/after each blocking I/O. 

If Lua is really reentrant as stated in its documentation, then even its blocking I/O libary would be interruptible and reentrant, so that we can force them to yield() in Lua instead of blocking inside the OS thread with its own internal yield to give hand to other processes and OS-native threads (but may be this requires modifying a bit the standard I/O library to force them to yield() in Lua instead of blocking in the native C standard I/O library, or in an OS API used by the native C standard I/O libary.

Le ven. 30 nov. 2018 à 00:22, Milind Gupta <> a écrit :
The C module just uses the Lua API and does not do anything OS specific.
    Sorry I do not quite understand the other remarks about having tasks only as strings. Every Lua program is a string to start with. 
   Run loop number of steps is just the number of opcodes you want to execute before returning control to the parent program.
     Communication between tasks is limited to passing tables and strings and not fancy shared objects. Recursive tables however should not be a problem. Communication right now will be managed by the parent task.

On Nov 29, 2018 2:54 PM, "Philippe Verdy" <> wrote:
May be, but it depends on an external C module implementing the preemptive meachanism (also its "tasks" are limited to scripts written as strings, so they cannot work with Lua functions and closures without first marshalling all parameters and its environment into the script (which will be unmarshalled by the Task engine), and then marshall the results of the task to string using some "print" statement that will be unmarshalled by parsing the output with getTaskData().

So I think it will be quite slow (and not easy to integrate to communicate complex Lua structures made with tables with possible cyclic references)

I think it uses a separate instance of the Lua engine, so these are not "threads" sharing data with other threads and coroutines (they may envtually share the same allocator and heap, but how to pass references to objects in a Lua thread with objects in a Task or in another task ? You also need extra communication channels I think.

What is needed is (I think) a native integration with threads in the same engine instance to benefit of closures and avoid the extra (un)marshalling using possibly very long strings.

The typical usage shown is extremely complex to start with, and I don't understand the concept of "number of steps" in the "runLoop". We probably don't need that or this dramatically reduces the possible usages to some limited scriptable tasks that can run independantly of any context and jsut take some input, run, then return some ouput, without communiating anything with other tasks or normal Lua threads.

Le jeu. 29 nov. 2018 à 23:11, Milind Gupta <> a écrit :

What is needed is a way to control in Lua programs (essentially in a parent thread controling all the other threads it creates and monitors) the resource usage and inform the VM which metered operations are allowed or not, and the VM should provide such basic metrics: the timeslot which is requested or the amount of memory requested or no longer in use. Nothing more. Now we can design a parent thread to do whatever it wants to apply its own policy for the children threads it creates and controls so none of these children threads can abusely take all the resources needed by other competing children threads (including the parent thread itself which can be protected from being blocked by its children).

There is a Lua Library to do that: LuaStepper (