Re: utf8.len and BOM

[Rob Kendrick <rjek@...>]

On Fri, Jan 16, 2015 at 09:17:08AM -0800, Coda Highland wrote:
> On Fri, Jan 16, 2015 at 4:53 AM, Rob Kendrick <rjek@rjek.com> wrote:
> > On Fri, Jan 16, 2015 at 12:11:41PM +0000, Aapo Talvensaari wrote:
> >> Is it by design that utf.len count the BOM to length?
> >>
> >> Say utf8.len("\xEF\xBB\xBFa") will return 2 instead of 1?
> >
> > Given UTF8 has only one valid "byte order", it makes no sense to ever
> > include a byte order marker in a UTF8 document.
> >
> 
> Sure it does -- the UTF-8 BOM is used (and aggressively promoted by
> Microsoft) as a magic number to identify the contents of the file as
> UTF-8 text. 

Lots of things aggressively promoted by Microsoft are mistakes.

No BOM -> content is UTF8-encoded.


> The XML spec even explicitly supports this (although many
> XML parsers do not).

Probably because they have to deal with people using Microsoft text
editors, not because it's a recommendation.

B.