Re: Yieldable/streaming LPEG

[Paul K <paul@...>]

Hi Sean,

> But I think that if you can break your parsing up into natural "units",
> you should be able to parse a bit and if you fail to parse, add more data
> until you can.  As a proof-of-concept:

This looks good; thank you for the PoC! I have something similar
(although less elaborate, also as PoC), but this doesn't solve one of
the main problems I have: if there is no way to parse the input into
natural "units", I'm stuck.

As an example, consider a lua file that returns a large table (few
thousand lines of code); I can construct a pattern that allows me to
parse an incomplete fragment (the one that doesn't include a closing
"}"), but I don't see a way to continue that parsing and correctly
parse the rest of the table *including the closing "}"*, which may now
be available.

More specifically; take two fragments: (1) "return {1, " and (2) "2,
}". I can fake/assume "}" at the end of the first fragment, but I
don't see a way to continue with the second fragment using the same
(let's say, Lua) grammar. The only way I could find was to "fake"
opening brackets as well by adding "return{" to the beginning of the
second fragment and then inserting the result into the right place in
the generated AST, but this looks a bit cumbersome and is likely to be
error-prone. If I don't find a better way, I'll probably finish and
test this option.

I can't really add more data and then go back and re-parse from the
beginning as it's not so much the lack of data as it is the lack of
time to parse it. I want to limit the time spent on parsing to small
chunks (up to 50ms) even though the entire text can be parsed in
500ms.

I'm open to any other ideas/suggestions...

Paul.

On Sun, Aug 31, 2014 at 6:00 PM, Sean Conner <sean@conman.org> wrote:
> It was thus said that the Great Paul K once stated:
>>
>> Are there any other options for parsing incomplete strings using LPEG?
>
>   I don't think LPeg contains much state itself (seeing how it's composable)
> so there isn't much to save---it either matches a pattern, or it doesn't.
>
>   But I think that if you can break your parsing up into natural "units",
> you should be able to parse a bit and if you fail to parse, add more data
> until you can.  As a proof-of-concept: