Re: LuaJIT strange memory limit

[Arseny Vakhrushev <arseny.vakhrushev@...>]

> Well, even if that sounds strange, but your best option is to
> compile LuaJIT as 32 bit. 32 bit processes can use close to 4GB of
> memory under a Linux/x64 kernel. And as I've previously explained,
> there's little performance difference with LuaJIT on x86 vs. x64.

Should the hosting process be compiled as 32bit as well?

> 
> A single process doesn't make good use of a multi-core CPU. And
> unless you're very careful and use non-blocking I/O everywhere,
> you'll hit I/O bottlenecks. Running 10-20 worker processes on a
> quad-core is a common setup.

My setup is not single threaded. Well, it really is a single process which spawns several tens (usually 32 or 64) of  LuaJIT threads along with some other working threads for performing non-blocking network I/O, etc. But, as I said before, the 1Gb memory is shared between all of the LuaJIT threads within the process. For instance, if I spawn just one LuaJIT thread and run a test script which tries to consume as much memory as it can, it fires an error having consumed around 1Gb of memory. If I spawn four such threads, they consume proportionally less memory, i.e. around 256Mb or even less.

> 
> Also, it's not a good idea to store millions of objects occupying
> several gigabytes in a single Lua state. The Lua garbage collector
> is simply not up to the task (LuaJIT currently uses the same GC).
> It's very, very inefficient for huge out-of-cache workloads. The
> GC causes serious cache thrashing and this kills performance.

I know that. That is why I spawn many LuaJIT threads not to let them get overloaded with GC cycles too much. Data and request load are balanced among them equally.

Let's say one LuaJIT state is fine with handling 200Mb of objects in general maximum. So, if I had 30 of such states working independently within one process, they would consume 200*30 = 6Gb of memory and I would be totally happy with that ceiling. However, right now it's not feasible no matter of how many LuaJIT threads I create.

// Seny