Re: Load large amount of data fast

[Petite Abeille <petite.abeille@...>]

On Oct 17, 2010, at 1:14 AM, Alexander Gladysh wrote:

> Apologies for a lazy question, I have not done my own homework.

Tssssk, tssssk, tssssk... :P

> I've got a large file  (3M entries, 250 MB) with data.
> Each entry is one line with a small Lua table:
> 
> { foo = 1; bar = 2; baz = 'text' };
> 
> (Actually, there are two different entry formats, but that does not matter.)
> 
> I need to load this data fast enough. (Faster than several hours that
> my original loader runs on LJ2, and it still had not stopped.)

For what it's worth, processing the IMDB actors.list (~550 MB, ~9 million records, read + parse + write to sqlite) take me around 10 minutes on my laptop [1], so there must be room for improvement :)

Have you measured where you are spending your time? How much overhead is there creating all these tables and closures? Are you keeping everything in memory or are you streaming the data? Etc, etc, etc...


[1] http://dev.alt.textdrive.com/browser/IMDB/Cast.lua