Re: Yieldable/streaming LPEG

[Paul K <paul@...>]

Hi Sean,

>  Hmm ... I often wished this were true of LPeg:
>        lpeg.match(pattern,subject[,init])
>                pattern - LPeg expressoin
>                subject - if string, use as input (what we have now)
>                          if function, call function to get input
>                          (much like how load() works)

Yes, I think this would extend interesting uses of LPEG, although
without yielding this doesn't work for my scenario as I want to stop
even though I do have more data to feed the pattern (and I can't yield
for the same reason as described below).

> Let's say we read "s".  Well, that doesn't match "oneida"
> so we need to back up and try the next option, so that means that LPeg *has*
> to save the existing input because of the way it works (by backtracking and
> trying the next pattern).

Right; that's the exact "state" I was talking about saving and
returning to use for the next match. In this case "match" would return
some value that I could pass to the next call to match instead of the
"pattern" and provide (possibly updated) subject. That "some value"
can be userdata or a function (somewhat similar to how "next" returns
a value I can pass back to it). I think in some ways it could be
better than using a function for the subject parameter as it would
provide a way to "suspend" lpeg processing and yield as needed.

> Write the parsing expressions such that at certain spots, you actually
> call coroutine.yield().  This can be done in Cmt(), Cf(), or by "pattern /
> function()".  Then, just run the expression in a coroutine.

This is a good idea and would have sold all the issues that I'm trying
to avoid, but unfortunately it doesn't work because of the dreaded
"attempt to yield across C-call boundary": in this case we are asking
to yield through LPEG/C code, which fails. It's possible to "resume"
from Cmt handler (instead of "yield" from it) and it may work for some
uses, but not for my case.

This works with LPegLJ as it's pure Lua (JIT with some FFI parts).

Paul.