Re: Ideas about implementing a string type?

[David Kastrup <dak@...>]

Hans Hagen <pragma@wxs.nl> writes:

> David Kastrup wrote:
>> Oh, actually I was not concerned about large strings as much rather
>> than about many smaller ones that are repeatedly getting interned into
>> Lua.
>>   
> most small strings that we deal with are indexes in (huge) font tables
> and constants; there hashed strings as well as keys save a lot of
> space and time since lots of string comparisons take place and many
> same strings occurs all over the place (not unlike tex's control
> sequences, which are also hashed)

Oh, but I was not arguing against hashing at all when working with
indices: all my proposals involved delaying the hashing until one
knows it is needed.

>> For some callbacks getting information from TeX, Taco switched the
>> data passed into Lua from strings to integers and reportedly got a
>> speedup of about 10 from that.
>>   
> indeed, and returning values instead of a table in the associated
> functions may also speed up the process; but simplicity and staying
> close to lua's way of doing things is also worth a price; concerning
> the tokenizer callback (which is what this is about) ... using
> numbers instead of strings (them being hashed or simple strings)
> makes sense anyway since in tex they are numbers; there are related
> functions that map them to symbolic names (strings);

Sure: I already said that it was one possibility to pass basically
opaque data (and ids that have to be resolved into strings by separate
functions are opaque) into Lua.  It just makes things less natural.
In the case of token names, it is certainly to be expected that the
token name might be used for a table lookup _if_ it gets used.  For
something like an optional callback, it is likely that the data will
just get output or even be thrown away, in which case hashing it every
time is a waste.

> concerning tex ... we may loose some time doing some things in lua,
> but also gain a lot using lua, for instance replacing the web2c file
> handling by lua filehandling speeds thsi part up by 50%; in due time
> we will report on such matters in journals and manuals.

The real wins will be in replacing _algorithms_ written in TeX by the
same written in Lua.  TeX is very efficient for typesetting stuff, and
completely awful for programming purposes, regarding both control and
data structures.  But those wins require _reprogramming_ that stuff.
Legacy code will not benefit without investing further work.  LaTeX,
an incredibly large collection of packages and classes and pieces from
very many people will likely be much slower to benefit from LuaTeX
than a single-shop product like ConTeXt.

> as said, currently we're satisfied with the way lua works and
> optimizations only will be considered when lua itself is drastically
> revised;

Well, that is why I am discussing these ideas on the Lua list and not
on the LuaTeX list: because the level of urgency at the LuaTeX is not
large enough to warrant diverging from upstream.  On the other hand,
if some operations could become cheaper in future Lua versions, I
doubt that LuaTeX developers would refuse to make use of that...

> in that respect, the experimental lpeg features will have more
> impact than extendions to the string model

I'll have to take a closer look at those, too.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum