Re: [ANN] LuaJIT Roadmap 2011

[Leo Razoumov <slonik.az@...>]

On Fri, Jan 21, 2011 at 18:03, Mike Pall <mikelu-1101@mike.de> wrote:
> Leo Razoumov wrote:
>> Recently there was a discussion on this list about memory limits of
>> JuaJIT2 on x86_64 platform [1].
>> Currently the memory available to a Lua state in LuaJIT2 is limited to
>> 1GB. Will this limit go away in the future? Will the new integrated
>> allocator/GC help here?
>
> It will be 2GB (or maybe 4GB) with the new GC in LuaJIT 2.1.
>
>> I am using (abusing) Lua for data analysis and quite often deal with
>> Lua tables with millions of entries (one can increase MAXBITS to 30
>> w/o harm). Even though LuaJIT is vastly superior for this task, I have
>> to use conventional Lua interpreter, for it imposes no such memory
>> limit.
>
> You could use the FFI. The FFI handles full 64 bit pointers and
> you can manually manage huge memory areas, up to the limits of
> your OS, by calling malloc()/free() via the FFI.
>
> Since the FFI allows creating arbitrary C structs at runtime, you
> can still have a lot of flexibility in your data structures.
>
> --Mike
>

The whole idea is to use Lua table as a universal data structure for
all the data-sets. Exploratory data analysis needs to be quick and
quasi-interactive. Creating data-set specific structures in C would
place an unnecessary burden on a human analyst.  That's why
data-analysis has so many domain specific languages like S, R, matlab,
etc. Lua has a good fighting chance to be a general purpose language
fit for the job and LuaJIT2 stellar runtime performance is a major
advantage if not for that memory limit (like 640KB is enough for
anyone:=)

--Leo--