Re: LuaJIT problems

["Mark Meijer" <meijer78@...>]

On 28/01/2008, Mike Pall <mikelu-0801@mike.de> wrote:
> Mark Meijer wrote:
> > I've had some problems with LuaJIT and large amounts of coroutines
> > (all written in pure lua, no extra C code in there). Out of memory, as
> > some may have guessed. I did some searching and gathered that the
> > default stack size in Win32 might be the culprit. I tried increasing
> > it to 16 meg using the compiler option /F16777216 (/F and 16 * 1024 *
> > 1024) just to see if it would help, but it made no difference at all.
> > Am I overlooking something?
>
> This is a common Windows problem. You need to *decrease* the
> thread stack space or you run out of virtual memory.
>
> See: http://luajit.org/coco_portability.html
> (scroll down to "Note for Windows" and follow the link to MSDN)

Thanks for the fast reply. I just tried it (setting stack size to 128k
this time) and it still fails at the same point, i.e. I can still
launch new coroutines fine but it runs out of memory again when I go
over a certain limit. I haven't checked what the exact limit is but it
appears to be roughly the same as before I started using that compiler
flag. More on this below.

A question though, are JIT/Coco coroutines implemented using OS fibers
or such? I mean, why else the stack size problem, right? At first I
figured that it needed, for some reason, a lot of stack space (but all
on a single stack) to create coroutines, which is why I tried
increasing the stack size. But it dawned on me when you told me to
decrease it, that the out of memory problem is caused by the *amount*
of stacks allocated. If I'm right about this though, I still don't
understand why it runs out of memory so fast. Unless the problem is
about something else... please read on.

My own little package at the heart of the demo, is called "fiber". It
is written in Lua and based on Lua coroutines. Other than that there's
nothing fancy, it's just some wrapping for coroutines and support for
message passing. I can launch over a million little test threads with
it just fine (at the cost of 1.3 gigs of memory), and send messages
back and forth, etc. Works like a charm. Of course such amounts of
coroutines I spawn only for testing.

However if I wanted to spawn, say, 5000 of 'em it should be no
problem. But with LuaJIT it is. If I launch 1000, it works. I launch
another 1000, it still works. If then I launch another 1000, it
crashes with the out of memory error. I also noticed that 1000 of my
fibers (i.e. what I call fibers in my little Lua module) costs about
9MB of memory with JIT, while only a little over 2MB in plain Lua.
What's more, it's 9MB regardless of whether or not I include the stack
compiler flag for LuaJIT to decrease its size. So... Am I overlooking
something else? :P

Anyway, about coroutines in LuaJIT... My whole point of using
coroutines was because I don't know of any other threading mechanism
that could handle such large amounts of threads so efficiently. Should
I forget about LuaJIT for this particular demo app? I don't know a
great deal about actual fibers in windows.

Thanks again.

Mark