[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: What do you miss most in Lua
- From: Miles Bader <miles@...>
- Date: Tue, 07 Feb 2012 11:12:29 +0900
Jay Carlson <email@example.com> writes:
>> > I imagine it already has in non-byte locales. A Korean Lua program will not
>> > use libraries from a Greek one. Unless everybody is already using
>> > UTF-8--with no sanity checking.
>> Of course everybody's just using UTF-8 with no sanity checking...
> Got some Koreans to back you up on that? EUC-KR (realistically, CP949)
> still lives; see the <head> of http://chosun.com and donga.com.
> naver.com is UTF-8 (not a total surprise). Perhaps all the Korean text
> processing *in Lua* is being done in unchecked UTF-8, but I kinda
> doubt it. ...
Of course I'm not claiming that all text-processing (even Lua
text-processing) is now done in UTF-8 -- there was a touch of
tongue-in-cheek to my message (but an element of truth as well).... :]
I live in Japan and write software for a Japanese company, so I have a
little experience in the matter. Shift-JIS (for "local" use) and
EUC-JP (for email) are still _hugely_ used. [At my work, we tried to
standardize on UTF-8 for a project, but ended up using Shift-JIS
simply because it's the only encoding that MS dev tools support
sanely, and has a lot of legacy support in other tools. The encoding
support in our own code is a mess that tries to generally support
various multibyte encodings, but in practice probably only has to work
properly for Shift-JIS and UTF-8.]
Nonetheless, my intuition is that:
(a) The Lua universe is not the more general universe. Projects
using Lua tend to be smaller, and less dependent on giant
frameworks. The tradeoffs for Lua projects are often somewhat
different as a result. Simple is good.
(b) UTF-8 is generally considered the future here (something that
must be supported now, and will increasingly replace other
encodings) even in areas where there's a lot of legacy need/support
for other encodings. People don't use older encodings because they
_can_, but because they _must_. There's definitely an awareness
that moving to UTF-8 is something that should be done if it's
(c) In a lot of applications, not all that much "detail handling" is
needed for what text-processing they do -- and if UTF-8 can be
assumed, things get _much simpler_. [E.g., if you have manipulate
multi-byte-encoded pathnames (or anything with meaningful ASCII
syntax), it's really simple in UTF-8 -- the same code one uses for
ASCII will work fine -- but miserable in Shift-JIS, because random
ASCII characters can occur in the middle of multi-byte characters.]
So I'd say there's several levels of text-processing support:
(1) If you can, don't even bother: treat strings as blobs, and
don't care whats in them (the default Lua state).
(2) If you need to do a little manipulation, try to use UTF-8 for
the encoding, but don't make any particular attempt to hide that
fact that they are encoded (i.e., don't pretend that strings are
"sequences of characters"). Only use what small functions you can
get away with (assuming UTF-8 makes this _much_ easier), e.g.,
counting characters, converting between byte- and character- offsets
etc. Such functions for UTF-8 are generally so easy that for many
projects it's fine to just write them yourself, but a "tiny-utf8"
library might not be a bad idea (which basically supports only stuff
that's either trivial as a result of the encoding properties, or can
be very compactly encoded).
(3) If you're doing full-fat text-processing (text-editor, etc),
maybe you do need real unicode support, giant tables and all. It
would be good to have a standard Lua library for this (there is one,
I think, but I don't remember the name).
For cases where legacy non-ASCII encodings _need_ to be supported,
especially if you need "full-fat" features, I dunno what good choices
there are, especially if you want to be portable (and so can't rely on
e.g. iconv)... Handling legacy multi-byte encodings is generally a
lot messier and more intrusive, and so should be avoided if possible;
sometimes platform libraries can make things easier, but that sort of
moves out of the realm of general Lua discussion.
Any man who is a triangle, has thee right, when in Cartesian Space,
to have angles, which when summed, come to know more, nor no less,
than nine score degrees, should he so wish. [TEMPLE OV THEE LEMUR]