Re: Load large amount of data fast

[Alexander Gladysh <agladysh@...>]

On Sun, Oct 17, 2010 at 04:38, Petite Abeille <petite.abeille@gmail.com> wrote:
> On Oct 17, 2010, at 1:14 AM, Alexander Gladysh wrote:

>> I've got a large file  (3M entries, 250 MB) with data.
>> Each entry is one line with a small Lua table:

>> { foo = 1; bar = 2; baz = 'text' };

>> (Actually, there are two different entry formats, but that does not matter.)

>> I need to load this data fast enough. (Faster than several hours that
>> my original loader runs on LJ2, and it still had not stopped.)

> For what it's worth, processing the IMDB actors.list (~550 MB, ~9 million records, read + parse + write to sqlite) take me around 10 minutes on my laptop [1], so there must be room for improvement :)

I take it that you suggest me to write my own Lua parser (or use a
custom one)? Well, since there is only two data formats, that's
doable...

> Have you measured where you are spending your time? How much overhead is there creating all these tables and closures?

Nope. That's one of the reasons for the homework remark.

> Are you keeping everything in memory or are you streaming the data? Etc, etc, etc...

As you may see in my code, I'm keeping all in memory.

Alexander.