Re: Unexpected calculation result with Lua53

[Dirk Laurie <dirk.laurie@...>]

2015-05-02 22:13 GMT+02:00  <tonyp@acm.org>:
> I noticed a problem (which I suspect is related to the introduction of
> integers in Lua53), and I’d like to know if this is ‘official’ behavior, or
> some sort of bug.  The behavior can be seen in this small example:
>
> function fact(n)
>   if n == 0 then return 1 end
>   return fact(n-1) * n
> end
>
> print(fact(66),fact(66.))
>
> Using fp parameter, return correct (?) result
> Using integer parameter, return 0 (zero) result!!!!
>
> I don’t know what the root of this problem but if it happens to be related
> to integer overflow, should it be converted to floating point, and continue
> ‘crunching’ rather than give a completely wrong result?

The mathematical number 66! would need 309 bits to represent
without loss, which is equally impossible in integer or floating-point
in Lua.

Since 66! is divisible by 2^n, where n=33+16+8+4+2+1 = 64, its last
64 bits are all zeros. So the result using integers is not "completely
wrong", it has 64 correct bits.

If you did the work in double precision, you would have got the exponent
right, and the mantissa nearly right (the last bit comes out wrong in IEEE
arithmetic), so it has 52 correct bits, but those are at the other end. Just
saying where the other end is uses up 10 bits.

Moral of the story: if bits at the high-order end are important to you, use
floating-point. If bits at the low-order end are important, use integers.
Lua 5.3 gives you that choice (just change 1 to 1. on the second line),
Lua 5.2 did not.