Re: xml/rapidxml UTF-8 validity assurance (was: [ANN] lub, lut, xml, yaml, dub and osc for Lua 5.3)

[Coda Highland <chighland@...>]

On Mon, May 11, 2015 at 1:29 PM, Tim Hill <drtimhill@gmail.com> wrote:
> Well that’s true of the ZWNBSP *codepoint* U+FEFF, which of course encodes
> to 0xEF/0xBB/0xBF. But what about dumb encoders that encode a big-endian
> UTF-16 sequence into UTF-8 and emit a byte-swapped encoding for the BOM?

Are you saying that the encoder actually emitted U+FFFE instead of U+FEFF? Ugh.


On Mon, May 11, 2015 at 1:45 PM, Javier Guerra Giraldez
<javier@guerrag.com> wrote:
> but here the culprit isn't Unicode, but UTF-16, which is a totally
> absurd idea. it's because of UTF-16 that we got the BOM, and because
> of brain-dead UTF-16-centric developers, that the BOM became valid for
> UTF-8.  Initially it wasn't valid, then it was 'unrecommended', now
> it's ok, and some software require it.

And some software chokes on it! :/ Despite the XML spec explicitly
saying that EF BB BF should be accepted at the beginning of a file.

/s/ Adam