Re: Load large amount of data fast

[Miles Bader <miles@...>]

Alexander Gladysh <agladysh@gmail.com> writes:
> I take it that you suggest me to write my own Lua parser (or use a
> custom one)? Well, since there is only two data formats, that's
> doable...

Hmm, have you considered using lpeg?

I use lpeg to parse rendering scene files, and can load ~600MB worth
(28M lines) of ASCII scene files (the bulk of which is humongous
arrays of floating-point numbers, but which also have a non-trivial
structure) in about 2 minutes; that 2 minutes includes loading another
300MB+ of ancillary data (image files etc), all post-loading setup,
etc (basically everything before rendering starts).

I read entire files into memory for parsing with lpeg, using a custom
C function that does mmap+lua_pushlstring, because io.read"*a" has
problems at extreme sizes (I used to just use io.read"*a" though, and
it worked fine with somewhat-smaller-though-still-pretty large files).

[The largest _single_ file in that scene is about 330MB (14M lines),
though I don't know offhand what portion of the loading time it
accounts for.]

-Miles

-- 
The car has become... an article of dress without which we feel uncertain,
unclad, and incomplete.  [Marshall McLuhan, Understanding Media, 1964]