Re: Unicode and UTF-8 the Lua way, mid-discussion (was Re: What do you miss most in Lua)

[Roberto Ierusalimschy <roberto@...>]

> Maybe I'm missing something, but there seems to be missing a way to
> efficiently compute "incremental" character byte-offsets in a string,
> which might be used when iterating over utf8 characters a string
> (possibly starting from some deep interior point).
> 
> [In my prev message I called this "char_offset" (maybe not such a good name):
> 
>     utf8.char_offset (STRING, BYTE_INDEX, NUM_CHARS) => NEW_BYTE_INDEX]
> 
> Your utf8.byteoffsets seems the closest in spirit, but won't be
> efficient in many cases because it always has to scan the string from
> the beginning.

When NUM_CHARS is 1, I guess you can do this:

  string.find(s, "[^\128-\191]", index)


In general, one thing to be decided is how much we can stretch the
standard library to provide utf8 functions. For instance, the following
code interacts through all code points in a string:

  s = "aloáéíЉМНЊО"
  for oneutf8 in string.gmatch(s, ".[\128-\191]*") do
    print(oneutf8)
  end

(Of course, it does not detect invalid sequences.)

-- Roberto