Re: Some thoughts on integer exponentiation (and an implementation)

[Stefan Ginsberg <stefanginsberg@...>]

> On 14 May 2017, at 23:46, Sean Conner <sean@conman.org> wrote:
> 
> It was thus said that the Great Stefan Ginsberg once stated:
>> 
>> Something I would find more useful is if overflowed integer results were
>> returned as nil (i.e. 123^456 == nil).
>> Not just for '^' but for all relevant integer operations. This would also
>> make unsigned numbers more manageable.
> 
>  Such checks add overhead to Lua (but as usual, one would have to test to
> see of the overhead is unacceptable).  Unfortunately, there is no cheap way
> to check for overflow in C (unlike assembly [1]).
> 
>  Second, I'm not sure if I like returning nil.  
> 
>    a = 1
>    b = 63
>    c = 1
> 
>    x = (1 << b) + c
> 
>    stdin:1: attempt to perform arithmetic on a nil value
>    stack traceback:
>            stdin:1: in main chunk
>            [C]: in ?
>> 
>  Okay, that doesn't happen now---what happens now is that x =
> 9223372036854775807 instead of -9223372036854775809 because of rollover. 
> I can see an argument for it, but ...
> 
>  -spc (but I'm not sure if I want to see how sloppy I've been with math ... )
> 
> [1]    http://boston.conman.org/2015/09/07.1
> 
>    It's not much overhead in time, but it does increase the memory
>    usage of the code because of the overhead of checking for overflow.
> 

To be clear that idea is only for the affected mathematical operators, not bitwise shifts.

I don't consider shifting out bits the same thing as overflow, it is another type of operation.

While checked integer math would incur some overhead, compared to something like a call,
table indexing, or even the overhead for doing the operation itself I believe it would be negligible.

For example, checked add:
'if ((ib ^ ic) >= 0 && (ib ^ ia) < 0)'