Re: Future plans for Lua and Unicode

[Owen Shepherd <owen.shepherd@...>]

Simon Orde wrote:
> Jerome - Thank you very much for your reply.  I was particularly taken 
> with your suggestion that I can let my script-writers write in UTF-16 
> (or whatever) and save the code as UTF-8 to pass to Lua.  This idea 
> hadn't occurred to me.  Without thinking about it too carefully, I'd 
> been assuming that Lua expects code (e.g. as supplied in a C++ call to 
> lua_load) to be in ANSI.  But is your suggestion really always 
> guaranteed to work - even in future versions of Lua?  I have to admit, 
> I can't instantly think of any reason why it shouldn't.  Sorry to 
> press you on this, but I have read "Programming in Lua" and I have the 
> Lua Reference Manual, and I can't find anything relevant to this in 
> either of them.  The Lua Unicode FAQ 
> (http://lua-users.org/wiki/LuaUnicode) addresses the question of 
> whether Lua programs can be written in Unicode, but it didn't mention 
> anything like that.  And yet your suggestion seems like the perfect 
> solution to using Lua with Unicode.  If it works, surely it's what 
> everyone should be doing?
>
> You asked about libraries.  I use iup, cd, im, lfs, luacom, luagl - 
> plus, sometimes, luasql, md5, and socket.  Actually, the string 
> library built into lua itself is also an issue, as much (most?) of it 
> presumably won't work with UTF-8.
>
> Yes please.
>
> Simon
"ANSI" is not an encoding; there is no text encoding which carries the 
name "ANSI."

"ANSI encoding" is the term that Microsoft uses, for legacy reasons, to 
refer to one of the two legacy encodings that Windows supports (the 
other being "OEM", which is mostly irrelevant besides the command prompt).

Lua's UTF-8 string support works as well as the string support built 
into C - i.e. it's ignorant of it, but not destructive. Realistically, 
that's all you want from a language the size of Lua; if you want more, 
you're going to want a Unicode support library such as ICU involved. 
Requiring such support is unlikely to be popular; the ICU core library 
is, compiled, 1MB in size*; but this is dwarfed by the data tables it 
requires, which are 17MB. Being as the Lua library weighs in at just 
215kB, I think most people here would object to such a large size increase.

* Windows, x86, Visual Studio 2010