lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]

On Mon, May 11, 2015 at 11:34 AM, Tim Hill <> wrote:
>> On May 11, 2015, at 10:53 AM, Jay Carlson <> wrote:
>> On May 11, 2015, at 7:47 AM, Gaspard Bucher <> wrote:
>>> xml - very fast xml parser
>> Glancing through the source code, I don't see code for rejection of non-UTF-8 sequences when parsing in UTF-8 mode. This is important to some people in Lua, since a lot of UTF-8-related security faults just go away if invalid byte sequences are rejected on input; Prosody is a good example of engineering for this.
> Which would you reject?
> — UTF-8 has several non-canonical ways to encode a value (e.g. by using more bytes than needed). I’ve seen (bad) encoders that emit these. Reject/accept?

Default reject (as it's more likely to indicate a non-UTF-8 file than
a bad encoder), provide a configuration option to accept (for the use
cases where you know you've got a busted encoder in the pipeline).

> — UTF-8 is sometimes used to encode UTF-16 values (such as BOM), some of which are now accepted. Reject/accept?

ZWNBSP (that is, BOM) is a perfectly legit character in UTF-8. Accept
it. I get pissed at decoders that choke on it.

> — UTF-8 can encode high/low UTF-16 surrogate pairs, which should be invalid but could be converted to a codepoint. Reject/accept?
> .. and so on.

UTF-8-encoded surrogates get generated by LOTS of encoders that
otherwise follow spec. Even though it's in violation of the spec,
accept by default. Offer a configuration option to preserve the
characters as encoded instead of combining the surrogates into a
single non-BMP code point for the purpose of analysis.

/s/ Adam