lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


On Aug 18, 2008, at 4:15 PM, Scott Fial wrote:
I am new to lpeg (and lua). Lpeg seems to require that the entire input string be available prior to parsing. I would like to be able to use lpeg to parse a stream of incoming data. I would also like to be able to use it for parsing huge files. In both cases, I won't have the entire input in memory. I need the parser to be able to accept more data as it arrives, while maintaining its state.

Ideally, I would like lpeg.match() to accept an optional function parameter that the parser will call when it hits the end of the current input string and is ready for more input. Then I can pass it a function to read the next few megs from a file, or call coroutine.yield until there is more data available on the communications channel.

Is there already an elegant *lua* way to handle the situations I have described? If not, would the callback architecture I've suggested someday be possible for lpeg?
The way LPEG works (and most parsers in general), it seems that all old data must be available, though it would seem LPEG 'could' work in a way that permitted more data to be fed in later... still requiring that the old data be maintained.

One potential mechanism could be to permit passing a memory-mapped file to LPEG, pending you can map the entire file... This would require LPEG to accept a userdata object and collect the memory pointer and length to the file data.