Lua-headed files

[Dirk Laurie <dirk.laurie@...>]

A Lua-headed file is one that starts with a Lua chunk defining an
iterator factory that returns a reader for the rest of the file. When this
chunk is loaded under the name 'reader', it can be used thus:

for record in reader(myfile) do
  -- process record
end

A Lua-headed text file starts with any number of non-empty lines
terminated by an empty line. That allows the iterator factory to be
loaded by:

myfile = io.open"myfile"
reader = load(myfile:lines())

The Lua code may be very simple, e.g.

~~~~ Lua-headed file ~~~
local f=...
local fmt="([%a%w]+)%s*:%s*(%d+)"
return function()
  local line=f:read()
  if not line then return end
  return {line:match(fmt)}
end

hundred : 100
thousand :1000
million:1000000
seventysix: 76
~~~~

It can be very complex, returning a table whose 'call' method
encapsulates a complete Lua parser for a file in some standard
format. Since the parameter passed to the chunk is an open file,
features like file:seek are there to be exploited before returning
the iterator.

It may also appear to be very simple while actually being very
complex, such as
   return require 'thisJSONisbest'

A major application may well be to concatenate the Lua chunk
with a binary file in some simple but nonstandard format that
needs string.unpack.

I have not yet thought through how a Lua-headed binary file
should know when the Lua chunk stops. One possibility is to
reserve a standard size (e.g. 256 bytes) which can if necessary
bootstrap a longer chunk. Another is to have the chunk end with
a sentinel string like --[[end_Lua_header]].

There is an obvious file extension available for Lua-headed files,
not currently used for any other file format, that would strongly
suggest a Lua connection to anyone in the know.