Re: Load large amount of data fast

[Alexander Gladysh <agladysh@...>]

On Sun, Oct 17, 2010 at 05:06, Petite Abeille <petite.abeille@gmail.com> wrote:
> On Oct 17, 2010, at 2:57 AM, Alexander Gladysh wrote:

>> I take it that you suggest me to write my own Lua parser (or use a
>> custom one)?

> Hmmm... I doubt that your problem is the parsing itself... more of an overall design issue perhaps... but then again, not enough information about what you are trying to do with that data to offer any concrete help :)

I'm trying to load that 3M entries in to Lua table in memory faster
than I do it now.

Other ways of solving my original task are out of the scope of this
question. :-)

Thanks,
Alexander.

P.S. I've accidentally killed my data crunching process and had to
start over. :-(

I've added some timing, here it is for the first 900K entries.

at line 100000 : Sun Oct 17 04:47:10 2010
at line 200000 : Sun Oct 17 04:48:29 2010
at line 300000 : Sun Oct 17 04:50:18 2010
at line 400000 : Sun Oct 17 04:53:02 2010
at line 500000 : Sun Oct 17 04:55:52 2010
at line 600000 : Sun Oct 17 04:58:55 2010
at line 700000 : Sun Oct 17 05:01:26 2010
at line 800000 : Sun Oct 17 05:07:00 2010
at line 900000 : Sun Oct 17 05:10:18 2010