Re: Managing Unicode (UTF-8 and UTF-16) data in Lua

[Paul Moore <p.f.moore@...>]

On 6 August 2016 at 21:33, Egor Skriptunoff <egor.skriptunoff@gmail.com> wrote:
> The problem is not in choosing right API functions.
> Try to write some Arabic or Japanese text in console on your European
> Windows using Unicode WinAPI.
> Probably, you will see question marks instead of readable symbols on the
> screen.

That's an issue of fonts, not of Unicode support.

> Only symbols from current Windows locale will be displayed correctly.

That's not true. I can display the "snowman" character in Powershell:

>$x = (0x2603)
PS 22:56 {00:00.000} C:\Work\Scratch
>[char]$x
☃

(Of course, your mail client may or may not display that correctly,
but I can assure you it displays fine on my PC).

> There is no "universal locale" in Windows, so, Unicode version of print() is
> unattainable.
> Welcome to Windows world!  Some thing here are quite weird.

I've programmed in Windows for around 20 years, and I've been dealing
with Unicode issues on Windows in Python for a long time now. What I'm
trying to achieve is all possible, but you can't easily do it from
within the 8-bit world of code pages. Unfortunately, the various
versions of the C runtime available on Windows are codepage based,
which makes the problem of Unicode a lot harder than it needs to be
for portable programs.

Anyway, this is very off topic now, as none of this is about Lua any
more, so I will leave it at that.

Paul