integer/float size and signaling overflow (was Re: Integer division)

[Jay Carlson <nop@...>]

On Nov 24, 2012, at 12:13 PM, Roberto Ierusalimschy wrote:

> We expect that the most common configurarions will be with double+64bit
> integers or float+32bit integers.

At least OpenWrt (and various commercial products based on it, like MiOS) ship with double+32bit. I don't know if that is intentional, or just an accident of history. ints are 32 bits on those platforms; perhaps this is an appeal to "fit the C API". 

file::seek and related tools need greater than 32 bits of precision, often even on embedded devices. Consider a DLNA video proxy. It regularly works with streams larger than 2G. Well, it *should* work with streams larger than 2G.... :-/

> In both cases, changing MAXINT+1 to
> float throws away the "+1". (OK, that happens with MAXINT+1+1, but never
> mind...)

...oops...

> This is hardly useful. If you are doing computations with
> integers, you expect full precision, not an "approximation".

Yeah. On the other hand, I don't see how native 2s-complement overflow behavior helps much with that, except perhaps for fake unsigned int. Non-signaling overflow is useful for (big1+n)-big2 I suppose. Freely casting back and forth between int and uint has been a great source of bugs so this is not a very good example.

I'm growing more convinced that "wants precision in computation" is a property of code, not of dynamically typed values. I know I may be a victim of confirmation bias.

I have not used a Lua-level language with signaling overflow. I don't know if it is painful.

Jay