[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: Repeated processing of large datasets
- From: "John Logsdon" <j.logsdon@...>
- Date: Wed, 29 Mar 2017 00:21:56 +0100
Yes, they are local to the function. cdata is a good idea although I am
not sure how that would fit with sqlite which may not be able to store
binary data. In essence though if stored as cdata I could just read the
whole table in a single read. Hmm. Interesting.
>> On 29/03/2017, at 12:19 AM, John Logsdon
>> <email@example.com> wrote:
>> for thisLine = 1,#AllLinez do
>> V1,V2,V3 = unpack(AllLinez[thisLine])
>> -- ... and then the data are processed
> I'm sure you know that V1, V2 and V3 should be local variables.
>> Processing involves a very large number of repeated optimisation steps
>> it is important that the data are handled as efficiently as possible. I
>> am using luajit of course.
>> My question is whether this is an efficient way to process the data or
>> would it be better to use a database such as SQLITE3?
> If lua can hold all your data in memory, and you're just iterating through
> the rows in your processing, not performing sql-like queries to find
> subsets of rows, then I don't see why luajit shouldn't be pretty quick.
> If your data is all numeric, then you might do well out of moving it to
> cdata rather than a lua table?
Quantex Research Ltd
+44 161 445 4951/+44 7717758675