Re: Do LuaJit 2 support more than 2GB memory allocation on FreeBSD/x64 now?

Subject: Re: Do LuaJit 2 support more than 2GB memory allocation on FreeBSD/x64 now?
From: Ilya Martynov &lt;ilya@ ... &gt;
Date: Wed, 18 Apr 2012 20:28:26 +0400

On Wed, Apr 18, 2012 at 20:11, Michal Kottman <k0mpjut0r@gmail.com> wrote:

On 18 April 2012 17:41, Ilya Martynov <ilya@martynov.org> wrote:

On Wed, Apr 18, 2012 at 17:33, Miles Bader <miles@gnu.org> wrote:

Anyway, all I'm trying to say is that it is possible for a program to
run into this limit, which might otherwise work fine (and that the GC
issues Mike mentioned aren't necessarily a limiting factor).

One such case is multi-threaded applications where each thread has its own Lua environment. For example we run some our server-side applications which embed Lua on 8 core servers so we have 8 worker threads each with its own Lua environment. This means that on 64 bit platform giving equal usage each Lua environment can address at most 1/8 Gb what is not that much. Long term as number of cores increases even more the current limitation on amount of memory LuaJIT can address will become even more severe.

You can alleviate this problem by going multi-process instead of multi-thread. Since your worker thread already run in their own Lua environments, they could also be running inside their own Lua process. You can use something like ZeroMQ to handle communication between processes. Since each process has its own virtual memory, every process can allocate the full amount of memory LuaJIT allows.