> The obvious fix seems to be limiting the loop explicitly:
That seems to work. The original test finishes normally with it. When I tried to insert more than 0x7ffffff values, the process quickly allocated so much memory that the OS started to page very busily, so after 10-15 minutes of waiting I interrupted that. I then tried to use the # operator, which gave me a negative number: -2147483615. Then I scanned the table in a loop looking for the max key value, and I found it was 2147483681. This is the unsigned 32-bit interpretation of the same binary value. Given this, I suspect that # uses a signed 32-bit integer internally, so if I had much more RAM, I would probably make that overflow even more, into a small positive number.
> If possible, could you report the initial value of '*pna' and the final value of 'optimal' in the last 10 calls to 'computesizes', with your original test?
> gt = {} local t = gt for i = 1, (0x40000001 - 10) do t[i] = i end print(#t)
1073741815
Then I set a breakpoint in computesizes and executed the following, line by line:
> gt[1073741816] = 1073741816
> gt[1073741817] = 1073741817
> gt[1073741818] = 1073741818
> gt[1073741819] = 1073741819
> gt[1073741820] = 1073741820
> gt[1073741821] = 1073741821
> gt[1073741822] = 1073741822
> gt[1073741823] = 1073741823
> gt[1073741824] = 1073741824
> gt[1073741825] = 1073741825
For each line, except the last one, the behaviour was identical: *pna = 0, the loop is skipped, optimal = 0 in the end. There were something like 5 calls into this function for each line.
For the last line, some initial calls were also like that, then there was one with *pna = 0x40000001, and that was the end of it.
I hope I did not make any mistake there, it was a bit tedious.
