Yieldable/streaming LPEG

[Paul K <paul@...>]

Hi All,

I'm looking for ways to feed LPEG an incomplete subject string,
somehow save its state, and then resume the processing if possible.
I'm familiar with LPegLJ and its streaming support, but have been
wondering if it's possible at all to achieve with LPEG or if it can be
extended to handle that.

Consider parsing something like "return {1,2" using Lua grammar; this
fails because of the missing closing bracket, but instead of failing,
LPEG could return its state, so that it can be used later to continue
the processing, possibly with some new content.

To have more control over how this is done, it may require a new match
function (as done in LPegLJ) or a new operator (probably similar to
the "cut" operator described in [1]). The operator could behave like a
match-time capture and would return as its capture some opaque
value/function to be used to resume matching. To minimize memory
requirements, it would assume no backtracking [2].

This would allow to implement both streaming and "yielding" from the
parser, which could be useful for getting partial results and for
limiting the time spent in the parser. In fact, if I can get the
streaming part, I can live without "yielding" as I can always limit
the time spent by feeding LPEG with small chunks of the subject
string. I like LPegLJ API, but it's still a bit too slow for my test
cases (3x-10x times slower than LPEG using the latest LPegLJ/LuaJIT).

Are there any other options for parsing incomplete strings using LPEG?

Paul.

[1] http://www.inf.puc-rio.br/~roberto/docs/sblp2013-1.pdf
[2] http://lua-users.org/lists/lua-l/2008-08/msg00423.html