Re: Tail -f in lua, truncate aware?

[Sean Conner <sean@...>]

It was thus said that the Great Egil Hjelmeland once stated:
> On 28. juni 2013 21:01, William Ahern wrote:
> >On Fri, Jun 28, 2013 at 02:42:19PM +0200, Bernd Eggink wrote:
> >>On 28.06.2013 13:47, ms2008vip wrote:
> >>
> >>>           I'd like to make the output of tail -F or something similar
> >>>           available to me in Lua without blocking or blocking.
> >>>       If the file gets truncated or log rotated, the program will detect
> >>>       it and will return to the start. This seems to be
> >>>       a level 1 question but looks strange to me. I just can't figure it
> >>>       out. Does anyone could share some code? TKS
> >>>  				
> >>In Linux you can use poll() or rpoll() from the luaposix library
> >>(https://github.com/luaposix/luaposix)
> >poll doesn't work on regular files. Or rather, it's defined to immediately
> >signal readiness, which makes it useless for this case.
> >
> >	"Regular files shall always poll TRUE for reading and writing."
> >
> >	-- http://www.opengroup.org/susv3xsh/poll.html
> >
> yes, by Bernd mentions output of process 'tail -F'. If he want output 
> from an other program into Lua, it will most likely involve a pipe or a 
> fifo, and pipes and fifos can be poll()'ed or select()'ed. Use a pipe if 
> you want to fork and exec the 'tail -F' directly from Lua, or just make 
> a named /path/to/fifo and do 'tail -F > /path/to/fifo'

  This still doesn't handle the log rotation part of the question [1].  But
this seems like a "please help me solve my solution to this unstated
problem" instead of "here's what I want, how so I do it?"  If the problem is
"I want to write a program that scans the log files generated by syslog"
then I do have a solution that will work (because I'm doing it).  If it's
"I want to write a program that scans a file some arbitrary program is
logging to (not syslog)" then the problem becomes a bit harder.

  So, assuming it's the "watch the output of syslog" problem, here's how I
solved that issue.  I caused syslog to forward the relevent logs (for me,
all of them) to a local IP address on the box [5], and then wrote a program
[6] to receive this stream [7] and display the results in real time [8]. 
Because the data is being sent to a socket, if there's no program listening,
the packets are dropped.  There's no issue with log rotation, since the data
isn't going to a file---it's being continuously streamed.  And this is
on-topic, since the program in question [7][8] use Lua.

  -spc

[1]	Most log rotation programs will rename the current file to a new
	name, then cause the program that created the file to re-create a
	new one [2].  Any program doing a "tail -f"-like operation will
	never see this [3].

	The program *might* be able to detect a log truncation, but I would
	have to check the Unix semantics to see what happens. [4]

[2]	Except, it seems, for Solaris.  There, the log rotation program will
	rename the original file, create a 0-byte version of the new file,
	then cause the program that wants to log to that file to reopen the
	newly created 0-byte file.  I learned this the hard way.

[3]	Except for programs that use inotify() on the directory (I
	think---I've never worked with inotify()).

[4]	For instance, program A can open a file, then program B can delete
	the file, but A can still read the file, since Unix doesn't actually
	remove the file until all open instances of said file are closed.

[5]	Technically, it's a multicast address, which only works for the UDP
	based syslog protocol (the universal default).  Before forwarding to
	a multicast address, be aware that any workstation on the local
	segment can tap into this stream of data.  In my case, that's
	exactly what I wanted.

[6]	https://github.com/spc476/syslogintr

[7]	https://github.com/spc476/syslogintr/blob/master/realtime.lua

[8]	Color coded too, based on the error level.