Re: another try at multithreading

["Augusto Radtke" <radtke@...>]

On Wed, Jun 25, 2008 at 10:21 AM, Ignacio Burgueño <ignaciob@inconcertcc.com> wrote:

Javier Guerra Giraldez wrote:

On Tuesday 24 June 2008, Jérôme Vuarand wrote:

That is quite an interesting thread, though I'd like to come back to a
point that has been discarded too quickly in my opinion.

2008/6/19 Javier Guerra <javier@guerrag.com>:

An obvious improvement might be to use finer locks, maybe one per
mutable object; but a pthread mutex lock weights between 40 and 64
bytes each.  way too much.  readers/writer locks are even bigger.

this days i've been reading about lock implementations, and found this (http://people.redhat.com/drepper/futex.pdf), reimplemented in C i get 4 byte locks, and first tests show that could also work at 1byte.  they're also noticeably faster than pthread locks, especially in the non-contented case (up to 60-70% faster).

the non-contented case is interesting, because since they're totally userspace in this case (only call the kernel if the lock is contented), and using a lock per object means a lot less contentions and higher opportunity for concurrency.

Interesting. It works like critical sections on Windows, but taking 6 times less space (a CS in Windows takes 24 bytes).

Interesting indeed, but isn't ANSI C, or even POSIX.  Although multi-threading is a very interesting support for Lua I can't see it making into it without breaking the ANSI C compatibility, perhaps this solution can grow it own tree. Forks are good if there is contribution between them, they just have some different points of view on some issues. To exist any cooperation the code has to be modular enough to support both views on the issue (sequential, multi-threaded), can Lua support it now? Also a version control management is extremely useful, and for that kind of nature of multiple branches something distributed like git seems to fit perfectly.

Today Lua doesn't seems to have a completely open process on that manner, the authors make a good job discussing most of things on this list but there is no repository of source code that we can see the advances and test them before they make into an official release, so for a second forked tree to update to the main tree would be difficult too, what about changing this? Roberto? Luiz? What are you thoughts about that?

I remember meeting Luiz on IMPA couple years ago to discuss an idea to multi-threading in Lua (if you have any interest on that idea, I'm attaching the file), that time I didn't have the time to make such a project, now that I have I would like to make it, but my major concern for the project, and any other kind of project that messes with Lua internals is, can we cooperate enough to validate the both trees?

Augusto Radtke

new keyword: parallel, lock, unlock, thread
new operator: <=

Lua 5.x : 21 reserved keywords
pLua 5.x: 24 reserved keywords

namespace pollution: 14.2%


Parallelism in Lua:
===================

parallel do
	-- block, every chunk is dispatched in parallel, synchronization at the end.
end

parallel while <condition> do
	-- every while test dispatches a parallel chunk, test the while again
	-- and when the condition is false the chunks are synchronized. 

end

parallel for k in t do
	-- t is a table
	-- for each key in table, dispatch in parallel, synchronization at the end.

parallel for k,v in t do
	-- t is a table
	-- for each key,value in table, dispatch in parallel, synchronization at the end.
end

parallel for i=1,10,2 do
	-- i is a counter
	-- 10 is the limit
	-- 2 is the step
	-- dispatch in parallel, synchronization at the end.
end

-- this is a fool example, no logic to be parallel, but to show the syntax

function func(arg1, arg2, arg3)
	a <= arg1	-- block until receive fro the channel
	b <= arg2	-- block until receive fro the channel
	c <= arg3	-- block until receive from the channel

	return a+b+c
end

c = thread func(arg1, arg2, arg3)

c is a default communication channel when you use the "thread" keyword/dispatch model, something like stdin/stdout.

arg1 <= 1	-- send only blocks if the channel is full (reader didn't read the last data)
arg2 <= 5
arg3 <= 3

r <= c		-- r equals 9, that will block until the function returns



synchronization:
==============

lock
	-- chunk, if another thread try to execute block until the first one finish
unlock

lock a				-- a is a variable, same behavior of chunk lock
unlock a			-- a is a variable


channels:
=======

new object type, channel.

instead of regular assignment, data sending/reading: a <= value (send), value <= a (read)
any lua type can be sent, including user datas.


Despatch:
=========

a) threads (green local and SMP)
b) multiple network nodes (interconnected lua VMs

Memory:
========

for a) shared memory
for b) nil, integer, string copied between nodes
       table, userdata e function remote reference
       for data copying use channels.


node connection:
==================

the node connection should be outside the language, using modules. the lua interpreter can support a default IPv4/IPv6 module:

# default to some port
server1# lua --server & 
server2# lua --server &
server3# lua --server &

client1# lua --server file.lua