Re: Load large amount of data fast

[Alexander Gladysh <agladysh@...>]

On Sun, Oct 17, 2010 at 06:21, Petite Abeille <petite.abeille@gmail.com> wrote:
> On Oct 17, 2010, at 3:56 AM, Alexander Gladysh wrote:

>> I guess those UUIDs are especially bad for performance:

> For the record, using uuid, 10 million records, ~800 MB, ~10 minutes:

> 2010-10-17 04:09:38 Started reading
> 2010-10-17 04:10:06 Finished reading
> 2010-10-17 04:10:06 Started load
> 2010-10-17 04:15:59 Finished load
> 2010-10-17 04:15:59 Started call
> 2010-10-17 04:19:29 Finish call
> 2010-10-17 04:19:29 Started count
> 10000000
> 2010-10-17 04:19:29 Finish count

> Still, I don't think it's a clever idea :))

No, it is not clever. :-)

BTW, maybe this is one of the reasons for the bad performance (but not
*that* bad, surely...):

model name	: Intel(R) Celeron(R) CPU          550  @ 2.00GHz
stepping	: 1
cpu MHz		: 1994.767
cache size	: 1024 KB
MemTotal:        4114624 kB

(I wish my MBP was back from the service sooner.)

Can somebody please try my original function on the dataset with LJ2
b5 on the modern hardware? A million of records should be enough.

Alexander.

P.S. Full timelog:

at line 100000 : Sun Oct 17 04:47:10 2010
at line 200000 : Sun Oct 17 04:48:29 2010
at line 300000 : Sun Oct 17 04:50:18 2010
at line 400000 : Sun Oct 17 04:53:02 2010
at line 500000 : Sun Oct 17 04:55:52 2010
at line 600000 : Sun Oct 17 04:58:55 2010
at line 700000 : Sun Oct 17 05:01:26 2010
at line 800000 : Sun Oct 17 05:07:00 2010
at line 900000 : Sun Oct 17 05:10:18 2010
at line 1000000 : Sun Oct 17 05:15:32 2010
at line 1100000 : Sun Oct 17 05:21:47 2010
at line 1200000 : Sun Oct 17 05:26:03 2010
at line 1300000 : Sun Oct 17 05:33:27 2010
at line 1400000 : Sun Oct 17 05:36:38 2010
at line 1500000 : Sun Oct 17 05:43:18 2010
at line 1600000 : Sun Oct 17 05:52:59 2010
at line 1700000 : Sun Oct 17 06:00:46 2010
at line 1800000 : Sun Oct 17 06:05:41 2010
at line 1900000 : Sun Oct 17 06:14:02 2010
at line 2000000 : Sun Oct 17 06:24:32 2010
at line 2100000 : Sun Oct 17 06:28:42 2010
at line 2200000 : Sun Oct 17 06:35:18 2010
at line 2300000 : Sun Oct 17 06:44:49 2010
at line 2400000 : Sun Oct 17 06:57:23 2010
at line 2500000 : Sun Oct 17 07:11:17 2010
at line 2600000 : Sun Oct 17 07:15:41 2010
at line 2700000 : Sun Oct 17 07:23:12 2010
at line 2800000 : Sun Oct 17 07:33:58 2010
at line 2900000 : Sun Oct 17 07:47:18 2010
at line 3000000 : Sun Oct 17 08:03:42 2010
at line 3100000 : Sun Oct 17 08:19:11 2010
at line 3200000 : Sun Oct 17 08:23:53 2010

... loading finished, some more utterly ineffective calculations here ...

real	221m37.969s
user	1m6.772s
sys	134m30.836s