[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: lpeg and streams
- From: Thomas Harning <harningt@...>
- Date: Mon, 18 Aug 2008 17:09:20 -0400
On Aug 18, 2008, at 4:15 PM, Scott Fial wrote:
I am new to lpeg (and lua). Lpeg seems to require that the entire
input string be available prior to parsing. I would like to be able
to use lpeg to parse a stream of incoming data. I would also like
to be able to use it for parsing huge files. In both cases, I won't
have the entire input in memory. I need the parser to be able to
accept more data as it arrives, while maintaining its state.
The way LPEG works (and most parsers in general), it seems that all
old data must be available, though it would seem LPEG 'could' work in
a way that permitted more data to be fed in later... still requiring
that the old data be maintained.
Ideally, I would like lpeg.match() to accept an optional function
parameter that the parser will call when it hits the end of the
current input string and is ready for more input. Then I can pass
it a function to read the next few megs from a file, or call
coroutine.yield until there is more data available on the
Is there already an elegant *lua* way to handle the situations I
have described? If not, would the callback architecture I've
suggested someday be possible for lpeg?
One potential mechanism could be to permit passing a memory-mapped
file to LPEG, pending you can map the entire file... This would
require LPEG to accept a userdata object and collect the memory
pointer and length to the file data.