Re: UTF-8 and the Windows console

[David Given <dg@...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthias Kluwe wrote:
[...]
> So, hopefully someone on this list has insight about the behavior of
> the MS Windows (XP) console regarding UTF-8 encoded data.
> 
> The windows console does work with UTF-8 if the “codepage” is set to
> the value 65001, apparently (command chcp 65001). I used lue5.1.exe
> from luabinaries.luaforge.net for a test (and a self-built lua.exe)
> with the command

UTF-8 on the Windows Console is, unfortunately, a world of fail. I spent
ages trying to get this to work for WordGrinder. Basically:

- - The console doesn't support any kind of font substitution. If you try
to render a glyph that's not supported by your current font, you'll get
a dummy glyph instead. The standard bitmap font supports ASCII-and-a-bit
only. Here's some instructions on how to use a real font:

http://www.hanselman.com/blog/UsingConsolasAsTheWindowsConsoleFont.aspx

- - The libc's stdio is not 8-bit clean and I've seen reports that it can
mangle binary data. If you're using UTF-8 this can cause you to generate
invalid sequences, which might be what's causing your app to fail.

The only actually reliable way I've found of getting Unicode to the
console is using the WriteConsoleOutputW() functions, which of course
don't work with redirectable streams... and if you thought ncurses had
an ugly API, you haven't seen anything yet.

This URL seems to describe what you're seeing:

http://mail.python.org/pipermail/python-list/2003-April/200079.html

I don't know if it actually *helps*, though...

- --
David Given
dg@cowlark.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknRVt0ACgkQf9E0noFvlzjHSgCbBYyIgqbhL/QdTsVCh0wwdGda
i3YAn2xcJKza0g+gGzib5QKcc/Dmgrvz
=HI1z
-----END PGP SIGNATURE-----