Re: Using Lua's string interning for faster compares?

[Jay Carlson <nop@...>]

[Obligatory suggestion for Lua core, if people insist on playing with C strings. Like pushlstring: lua_findlstring(L,s,l), which will return false if s would have had to be allocated. If s would have to be allocated it cannot possibly be a key in a raw table. This optimization could be applied to a lua_rawgetfield() without anybody noticing.]

[Additional string functions in Lua could support this as well: I have a long string from the network; give me my command to look up, or give me nil if it's pointless to try to look up in any table. Well, rawget from any table. __index can find anything. I find rawget annoying, and I don't know if I want to encourage it.]

On Jan 17, 2012, at 7:04 AM, Peter Cawley wrote:

> In one of my current projects, I have a similar need to frequently
> switch over a Lua string. I opted for a solution of hash-based
> dispatch combined with code generation, which I don't think I've seen
> proposed on lua-l recently.

I have two reactions:

First, Lua tables seem to work well in most circumstances. I would consider restructuring my code after writing about the second C string switch statement. I come to Lua to escape from C and C strings, and I intend to fully enjoy my vacation.

My second reaction is that, uh, actually I've been in that switch statement situation with Lua tables before. My ancient FLTK binding used tolua++, which produces a lua-side metatable for every class. You can just eat that on desktop platforms, but I was working on an 8M Linux PDA; every app built with the FLTK binding had ~300k of *unsharable* allocated memory by the time it made it to main. A lot of it might not be touched. With swap, that kind of thing is an annoyance; without, private memory gradually squeezes the rest of the system to death. And the payload, the actual working Lua FLTK app, was often like ~100k of private memory. All of that was with custom gc settings to minimize the amount of sbrk.

IMO that's the level of measurement and desperation appropriate for bailing out on tables. I didn't, but I did have a plan.

I had an advantage: most of my tables were already being generated from tolua++ source. I didn't need to scan my own source code to find strings, and there already was a code generation step. What I thought I needed was rommable hash tables, some const structure which could live in the .text/.rodata section and be sharable (and a candidate for page-out) across all the apps loading the luafltk shared object.

My plan was to create my own user type with an __index which could look up methods by name in a (perhaps perfect) const hash table generated by tolua++. This C hash would not be visible to users; a regular writable Lua table would be hit first. At that point I could play around with memoizing the C-side hash results, weak tables etc.

Jay