Re: Unicode and UTF-8 the Lua way, mid-discussion (was Re: What do you miss most in Lua)

[Tim Mensch <tim-lua-l@...>]

On 2/8/2012 11:01 AM, Dirk Laurie wrote:
> (1) Additional functions in "string" library, e.g. str:usub(3,6) 
> extracts UTF8 characters 3 to 6 and throws an error if str is not 
> valid UTF8.  Pro: simplest.  Con: requires a change in 'official' Lua, 
> can't genuinely start mid-string.

Is there some reason that I'm not getting that we couldn't add functions 
to "string"? Just that it's considered bad form?

Though it wouldn't be able to add the optimizations I suggested in 
another message if you didn't modify Lua proper, so no, you can't start 
mid-string.

> (3) Another standard library, say "utf8", but operating on userdata, 
> e.g. ustr:sub(3,6).  ustr:type() is 'utf8'.  Creates a private code 
> point address list.  Pro: avoids cons of (1) and (2).  Con: requires 
> conversion to-from string.

One of the key advantages of using UTF-8 is that you're just 
manipulating strings, so not being able to convert trivially is 
annoying. Obviously it could have a __tostring function, though, so 
converting in that direction doesn't need to be painful. If only 
concatenation (..) would run __tostring on a table parameter, we'd be 
set with the userdata approach. And to convert the other direction, a 
short function name like "_u" could make that easy:  _u"string to make a 
UTF-8 object out of".

I pretty much need such a function anyway, since the sane way to do 
internationalization is to put your strings in a table somewhere which 
gets switched based on your locale, so in the code I'll have:

_t "English version of the string."

...and then elsewhere I'll have a table:

{
   ["English version of the string."] = "A translation of the string to 
another language."
}

> But your item [2] really kills all of these ideas.  If we can't have  
> ustr:match, we may as well compile Lua with 16-bit Unicode strings if 
> our locale is fundamentally non-ASCII.

Yuck. I would suggest that 16-bit Unicode was NEVER a good idea. Not 
even counting combining characters, you can't even fit all of the 
Unicode code points in 16-bits (over 110,000 now [1]), so some of them 
take two words to store ("surrogate pairs"). This means that you can't 
reliably index a UTF-16 string using offsets, and direct indexing of 
characters is the only argument I've heard in favor of UTF-16.

Aside from that, apart from Windows, the rest of the world seems to be 
moving toward UTF-8 as a standard encoding. (I know that it's more 
complicated in some countries, but still it seems to be the general trend.)

Making the pattern matching work for UTF-8 strings wouldn't be rocket 
science. As was pointed out in another message, MOST of the patterns 
would work MOSTLY as-is. I bet it wouldn't take more than a few minor 
patches to make a version of match() that would work fine for UTF-8.

Tim

[1] http://en.wikipedia.org/wiki/Unicode