Re: Repeated processing of large datasets

["John Logsdon" <j.logsdon@...>]

Thanks Geoff

Yes, they are local to the function.  cdata is a good idea although I am
not sure how that would fit with sqlite which may not be able to store
binary data.  In essence though if stored as cdata I could just read the
whole table in a single read.  Hmm.  Interesting.


>
>> On 29/03/2017, at 12:19 AM, John Logsdon
>> <j.logsdon@quantex-research.com> wrote:
>>
>> for thisLine = 1,#AllLinez do
>> 	V1,V2,V3 = unpack(AllLinez[thisLine])
>> -- ... and then the data are processed
>> --
>> end
>
> I'm sure you know that V1, V2 and V3 should be local variables.
>
>> Processing involves a very large number of repeated optimisation steps
>> so
>> it is important that the data are handled as efficiently as possible.  I
>> am using luajit of course.
>>
>> My question is whether this is an efficient way to process the data or
>> would it be better to use a database such as SQLITE3?
>
> If lua can hold all your data in memory, and you're just iterating through
> the rows in your processing, not performing sql-like queries to find
> subsets of rows, then I don't see why luajit shouldn't be pretty quick.
>
> If your data is all numeric, then you might do well out of moving it to
> cdata rather than a lua table?


Best wishes

John

John Logsdon
Quantex Research Ltd
+44 161 445 4951/+44 7717758675