Re: Repeated processing of large datasets

[Francisco Olarte <folarte@...>]

On Tue, Mar 28, 2017 at 1:19 PM, John Logsdon
<j.logsdon@quantex-research.com> wrote:
( Swapped order )
> My question is whether this is an efficient way to process the data or
> would it be better to use a database such as SQLITE3?

It depend on your concrete data & processing but I'd like to point:

> Then in the main program I read each line at a time:
> local Linez=readValues(tickStream)
> while Linez ~= nil and #AllLinez < maxLines do
>         table.insert(AllLinez,Linez)
>         Linez=readValues(tickStream)
> end

I'm not sure if this is efficient in luajit, but IIRC in lua # is not
constant time in tables, and table.insert uses # as default insert
position, so if I were worried about speed I would normally do:

local nlines = 0
while nlines <maxLines do
   Linez = readValues(tickStream)
   if Linez==nil then break end
   nlines = nlines+1
   AllLinez[nlines]=Linez
end
-- Probably stash nlines in AllLinez.n for easier passing around...

> The processing is then a matter of looping over AllLinez:
> for thisLine = 1,#AllLinez do
And here I would use nlines instead of #AllLinez, although I think and
ipairs loop maybe faster ( just time it ).

Francisco Olarte.