Re: Lua 5.1 and UTF-8 ?

[Chris Marrin <chris@...>]

Asko Kauppi wrote:
> I've been thinking about UTF-8 and Lua lately, and wonder how much  work 
> it would be to actually support that in Lua "out of the box".   There 
> are some programming languages (s.a. Tck) that claim already to  do 
> that, and I feel the concept would match Lua's targets and  philosophy 
> rather nicely.
>
> I understand UTF-8 might not be everyone's favourite, but it is  mine. 
> :) And having a working framework (str:bytes(), str:length(),  
> str:width()) could easily be adopted to other extended encoding  schemes 
> as well.
>
> The reason I'm bringing this up right now, is that the issue could  suit 
> nicely with the 5.1 "every type has a metatable" way of  thinking; would 
> it warrant an opportunity to have a closer look at  what Lua means by 
> 'strings' (or rather, their encoding) anyhow?
>
> The only thing important for me (or.. end users in general?) would be  
> not to have two different ways for strings.  "Hello" and L"Hello"  
> should not be necessary, imho.  In UTF-8, they aren't.

I have been working with Unicode for many years now. And I believe that 
it is much better to handle string internally as full unicode (wchar_t) 
character strings. It is very cumbersome (slow) to perform operations 
like length, insert, and other string ops on UTF8 strings. I would 
really like to see Lua changed to allow compilation for char* or 
wchar_t* strings.

I am using Lua with my app and I am doing lots of conversions between 
char* and wchar_t* strings because my app uses wchar_t* for its internal 
string representation. I have nice classes (String and UTF8String) to 
convert between the two, but it is slow.

--
chris marrin                ,""$,
chris@marrin.com          b`    $                             ,,.
                        mP     b'                            , 1$'
        ,.`           ,b`    ,`                              :$$'
     ,|`             mP    ,`                                       ,mm
   ,b"              b"   ,`            ,mm      m$$    ,m         ,`P$$
  m$`             ,b`  .` ,mm        ,'|$P   ,|"1$`  ,b$P       ,`  :$1
 b$`             ,$: :,`` |$$      ,`   $$` ,|` ,$$,,`"$$     .`    :$|
b$|            _m$`,:`    :$1   ,`     ,$Pm|`    `    :$$,..;"'     |$:
P$b,      _;b$$b$1"       |$$ ,`      ,$$"             ``'          $$
 ```"```'"    `"`         `""`        ""`                          ,P`
"As a general rule,don't solve puzzles that open portals to Hell"'