Re: Reading CSV

[Geoff Leyland <geoff_leyland@...>]

On 4/12/2013, at 11:05 am, Tim Channon <tc@gpsl.net> wrote:

> Do 3G in one hit and you consume 3G of machine memory.

Which unfortunately LuaJIT won’t be happy about.  But I think the main contributor to memory use and GC stress is not the text of the file, but the table holding the representation of all of the fields of the file.

> Try it, only take a few seconds to write something.

I did.  Hence my first post :-)

Regardless of file size, I don’t think it’s particularly controversial to prefer some kind of iterator approach to delimited files like:

for fields in csv.open(“file.csv”):lines() do
end

over something like:

for _, fields in ipairs(csv.readentirefile(“file.csv”)) do
end