Re: Pattern matching among multiple chunks

[Hugo Musso Gualandi <hgualandi@...>]

> Here I was
> just searching for a way to detect if a failing pattern have some
> chance to match when data is appended to the input

The default implementation of Lua's pattern matching library just does
a simple search to find the pattern. However, it doesn't try to reason
about the grammar that you are trying to match or record what paths it
could have tried to take had there been more input available, so I
don't think you can quite do what you are asking.

One option that may or may not work is to be optimistic on failure. If
a match fails you save the remainder and try again when more input is
available.

The more ambitious alternative would be to create a fork of the pattern
matching library that implements this feature but I think at this pointyou may be better off using an existing lexer generator. 

> If I have to write a C library, probably the quickest solution is
> just
> to modify the lua pattern matching library in order to return a the
> end position in case of failure. I think it should be enough.

One thing that might work, given that Lua patterns don't support
complex alternation, is to split your pattern into a serries or smaller
patterns and try one after the other. You can use string.find to know
when each match ends and where the next should begin.

For example, if your original pattern is "%S+ %d+" you can first try to
match "%S+" then, if it succeeds, try matching "%d+" from where the
previous match stopped.

To keep the code sequential and easy to read you can use coroutines to
implement that "yield if there is still unread input" approach that we
mentioned earlier.