Re: xml/rapidxml UTF-8 validity assurance (was: [ANN] lub, lut, xml, yaml, dub and osc for Lua 5.3)

[Tim Hill <drtimhill@...>]

> On May 11, 2015, at 10:53 AM, Jay Carlson <nop@nop.com> wrote:
> 
> On May 11, 2015, at 7:47 AM, Gaspard Bucher <gaspard@teti.ch> wrote:
> 
>> xml - very fast xml parser
>>    http://github.com/lubyk/xml
> 
> Glancing through the source code, I don't see code for rejection of non-UTF-8 sequences when parsing in UTF-8 mode. This is important to some people in Lua, since a lot of UTF-8-related security faults just go away if invalid byte sequences are rejected on input; Prosody is a good example of engineering for this. 
> 

Which would you reject?
— UTF-8 has several non-canonical ways to encode a value (e.g. by using more bytes than needed). I’ve seen (bad) encoders that emit these. Reject/accept?
— UTF-8 is sometimes used to encode UTF-16 values (such as BOM), some of which are now accepted. Reject/accept?
— UTF-8 can encode high/low UTF-16 surrogate pairs, which should be invalid but could be converted to a codepoint. Reject/accept?
.. and so on.

—Tim