Re: Nested file:lines()

[Andrew Cannon <ajc@...>]

Hi Alexander,

this is nothing to do with Lua. It's just that the underlying operating system
(or C library) only maintains one file pointer per open file descriptor. Once
you've iterated through the file the file pointer is at the end of file and
subsequent calls to :lines() will return an iterator which starts at the end of
file!

When you reopen the file the file pointer is back at the beginning (your last
example)... but don't forget to close the inner file, otherwise the used FD will
remain open and you'll soon run out of available FDs:

<code>
  local f = assert(io.open("123.txt"))
  for l in f:lines() do
    print("O", l)
    local ff = assert(io.open("123.txt"))
    for ll in ff:lines() do
      print("i", ll)
    end
    ff:close()
  end
  f:close()
</code>

Of course then you might as well use io.lines() which implicitly opens and
closes the file for you:

<code>
  for l in io.lines("123.txt") do
    print("O", l)
    for ll in io.lines("123.txt") do
      print("i", ll)
    end
  end
</code>

A bit more efficient would be to rewind the inner FD rather than reopening the
file each time through the outer loop:

<code>
  local f = assert(io.open("123.txt"))
  local ff = assert(io.open("123.txt"))
  for l in f:lines() do
    print("O", l)
    ff:seek"set"     -- rewind ff to start (first time is redundant)
    for ll in ff:lines() do
      print("i", ll)
    end
  end
  ff:close()
  f:close()
</code>

... or if the input file is not too big why not read the whole thing into memory
and process it from there, eg:

<code>
  local t = {}
  for l in io.lines("123.txt") do
    table.insert(t, l)
  end
  for _,l in ipairs(t) do
    print("O", l)
    for _,ll in ipairs(t) do
      print("i", ll)
    end
  end
</code>

Then the program will also work for non-seekable inputs such as TTYs or network
sockets.

Cheers
Andrew



Alexander Gladysh wrote:
> Hi, list!
> 
> OS X 10.5.8
> Lua 5.1.4
> 
> Most likely this was discussed earlier (or even is documented
> somewhere in the manual; or I'm doing something wrong :-) ).
> 
> I'm experiencing unexpected (by me) behaviour when nesting
> file:lines() calls for the same file:
> 
> $ jot 3 >123.txt
> 
> $ cat >123.lua
> local f = assert(io.open("123.txt"))
> for l in f:lines() do
>   print("O", l)
>   for ll in f:lines() do -- Use the same file handle
>     print("i", ll)
>   end
> end
> 
> $ lua 123.lua
> O	1
> i	2
> i	3
> 
> Looks like lines iterator is shared for all :lines() calls. Not sure
> if it is necessarily wrong, but perhaps it should be documented (am I
> missed it?)
> 
> But what if I'd open two file handles for the same file?
> 
> $ cat >123-1.lua
> local f = assert(io.open("123.txt"))
> local ff = assert(io.open("123.txt"))
> for l in f:lines() do
>   print("O", l)
>   for ll in ff:lines() do
>     print("i", ll)
>   end
> end
> 
> $ lua 123-1.lua
> O	1
> i	1
> i	2
> i	3
> O	2
> O	3
> 
> Looks like subsequent calls for :lines() on the same handle are ignored.
> 
> Indeed: (IMO, this fact should be documented as well.)
> 
> $ cat >123-2.lua
> local f = assert(io.open("123.txt"))
> 
> for l in f:lines() do
>   print("O", l)
> end
> 
> for l in f:lines() do
>   print("i", ll)
> end
> 
> $ lua 123-2.lua
> O	1
> O	2
> O	3
> 
> So, the correct way is to re-open nested file handles on each:
> 
> $ cat >123-3.lua
> local f = assert(io.open("123.txt"))
> for l in f:lines() do
>   print("O", l)
>   local ff = assert(io.open("123.txt"))
>   for ll in ff:lines() do
>     print("i", ll)
>   end
> end
> 
> $ lua 123-3.lua
> O	1
> i	1
> i	2
> i	3
> O	2
> i	1
> i	2
> i	3
> O	3
> i	1
> i	2
> i	3
> 
> Alexander.
>