Re: constant table overflow

[David Olofson <david@...>]

On Wednesday 09 March 2005 15.44, Serge Semashko wrote:
> Hello All,
> 
> We are using lua as a storing and manipulating a big set of data
> (arranged as a set of nested tables containing lots of string
> constants). After some time, the size of this data has grown to some
> extent and now we are getting "constant table overflow" error and 
can't
> load the file anymore.
> 
> The following script can be used to create a lua file that 
reproduces 
> the problem:
> 
> for i = 1, 1000000 do
>      print("a = \"" .. i .. "\"")
> end
> 
> And the size of this test file is only 13MB. Maybe such limitations
> are too restrictive? Is lua going to have a bigger constant table in
> the future?

Well, the reason for the limit is that one has to decide on a 
reasonable size for instructions. The Lua authors decided on 32 
bits/instruction, and with the current layout, that means 8, 9 and 9 
bits respectively for the operands - or as in this case, 18 bits, 
when B and C are combined into Bx.

It's theoretically easy to "fix", but that comes at a cost: Larger 
code and slower execution...


> And is there an easy way to hack lua 5.0.2 to increase this limit?

For linear expansion of the constant indexing range, you could add 
LOADK variants that add 2^18, 4^18 etc to Bx before indexing, and 
have the coder select LOADK variant as needed to access the requested 
constant.

Another (slightly more involved) variant would be to add a "long 
address prefix" instruction, that loads high index bits that LOADK 
then uses to reach an otherwise out-of-range constant. To avoid 
having to implement state tracking for this "extended addressing 
register" in the coder, you could just have LOADK zeroing those extra 
bits after using them, making the "long address prefix" instruction 
"one-shot" rather than state changing.


> I don't quite understand constants in 'lopcodes.h' and not sure
> whether it is safe to change them.

There are only 32 bits in an instruction, and there are 35 
instructions (in the version I'm looking at), so you can't steal any 
bits from the opcode field, at least... You could try making the A 
operand smaller, I guess, but then you might run out of addressable 
registers instead.

The opcode field has room for around a bunch of new instructions 
though (max 64 total), so unless you need *many* times more constants 
than the current 2^18, special "high LOADK" instructions might do the 
trick. Doesn't look like it would be all that hard to implement, 
though it might confuse the optimizer or other parts of the compiler.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Audiality -----------------------------------------------.
|  Free/Open Source audio engine for games and multimedia.  |
| MIDI, modular synthesis, real time effects, scripting,... |
`-----------------------------------> http://audiality.org -'
   --- http://olofson.net --- http://www.reologica.se ---