Re: UTF-8 testing

[Erik Lindroos <gliptic@...>]

On Fri, Jan 7, 2011 at 11:25 AM, David Given <dg@cowlark.com> wrote:

On 07/01/11 00:22, Miles Bader wrote:
[...]

> Hmm, one could store the table indexed by "byte >> 2" as well, to
> save some space:
>
>    static const char m_trailingbytes[64] =
>    {
>      0,0,0,0, 0,0,0,0, 0,0,0,0, 0,0,0,0, 0,0,0,0, 0,0,0,0, 0,0,0,0, 0,0,0,0,
>      0,0,0,0, 0,0,0,0, 0,0,0,0, 0,0,0,0, 1,1,1,1, 1,1,1,1, 2,2,2,2, 3,3,4,5
>    };

That's neat --- I hadn't thought of that. (It's free on a lot of
processors, too.)

Or one can use a tiny table (for the 2003 standard):

int unsafe_char_length(uint8_t start)

{

int high_nibble_2x = (start >> 4) << 1;

return ((0xe5550000 >> high_nibble_2x) & 3) + 1;

}

Although you have to make an extra check for invalid start bytes.

/Erik