[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: Setting Float Precision in Lua.c
- From: Albert Chan <albertmcchan@...>
- Date: Sun, 10 Jun 2018 17:49:09 -0400
> Double rounding cause the value to be off by +/- 1 ULP
> It is OK if Lua did this consistently throughout its distribution.
> Bad rounding is compensated by higher internal float precision.
> -- Albert 6/5/18
Turns out, above statement is *wrong* ... Sorry
Lua does not benefit from higher internal float precision.
To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)
(x + y + y + y) as a whole is not done in extended precision
--> Lua get hit with 3 double roundings, set z = 1e16 + 12
If above is true, the case for 53-bits rounding is even stronger.
Doing the same in C, it does it all in extended precision,
calculated the *best* double, z = 1e16 + 8