[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: LuaTask and memory allocation
- From: Duck <duck@...>
- Date: Wed, 27 Jun 2007 15:00:21 +1000 (EST)
[Agglommerating multiple posts long after the event ;-]
I was rather hoping to handle a couple of hundred
proxied sockets inside a simple Lua program... whilst
there is plenty of physical memory to support this I
am, absurdly, running out of address space :-)
What's going on?
Each task requires a C stack; the default stack allocation
on linux is quite large.
You might want to try using plain old Lua coroutines,
which do not require a C stack but which do have some
limitations as a result.
The good news is that my failure to do this the "easy" way using LuaTasks
(which are actually very cool -- kind of like fork()s from a separation
point of view, but without the fork()ing overhead) means that I will
almost certainly knuckle down and do it the "proper" way using coroutines.
It's my own fault for not pursuing the elegant solution in the first place
pthread_attr_setstacksize( &attr, 2097152/16 ); // Thanks Asko !!!
before pthread_create in syncos.c line 62, my test runs using 42m of
virtual memory on Ubuntu 7.04
But... with more than 255 subtasks... SEG FAULT
Since this implies a LuaTask 1.6.1 some time in the not-too-distant
future, how about adding an optional third param to task.create() allowing
a stack size to be specified?
(You could add a new error code, say -5, for "stack size out of range" in
case pthreads or _beginthreadex() doesn't like the size, e.g. it's too
small, or just re-use error -4 in this case.)
For an OS implementation in which the stack size can't (easily) be set, or
isn't needed, simply ignore the parameter quietly...
Just a thought. Would allow the stack size to be tuned for individual
tasks, which would be in keeping with Lua's "don't be huge" attitude! OSes
seem to vary quite a bit -- 64k on FreeBSD IIRC, 1m on Windows, 8m in
Linux pthreads -- so some way of "saying what you mean" for each subtask
would be entirely suitable IMO.
(BTW: about the segfault, which I blithely said was "unsurprising"...I
assumed it was happening at around 250 because 256*8m = 2g, but your tests
show it's not down to that. Anyway, Linux slices VM between user and
kernel at 3g, doesn't it? Not the default 2g of Windows, which I am more
familiar with. Oh, unless you run Exchange, which is such a beast that
Windows offers you the special "/3G" bootup option to start
kernel-land at 0xC0000000 to make room for all those pesky calendar