lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]

On May 11, 2015, at 11:54 AM, Coda Highland <> wrote:

— UTF-8 is sometimes used to encode UTF-16 values (such as BOM), some of which are now accepted. Reject/accept?

ZWNBSP (that is, BOM) is a perfectly legit character in UTF-8. Accept
it. I get pissed at decoders that choke on it.

Well that’s true of the ZWNBSP *codepoint* U+FEFF, which of course encodes to 0xEF/0xBB/0xBF. But what about dumb encoders that encode a big-endian UTF-16 sequence into UTF-8 and emit a byte-swapped encoding for the BOM?

The problem is UTF-8 *should* be used to decode: UTF-8 -> codepoint array. Instead its (shudder) often used to decode UTF-8 -> UTF-16 -> (byte-swap based on BOM) -> codepoint array.

It’s one reason I detest Unicode.