[Date Prev][Date Next][Thread Prev][Thread Next]
[Date Index]
[Thread Index]
- Subject: Re: Setting Float Precision in Lua.c
- From: Albert Chan <albertmcchan@...>
- Date: Tue, 5 Jun 2018 13:15:59 -0400
> If someone insists on _PERFECTION_ for their floating-point output,
> they should use the hexadecimal format. Lua has full support for
> hexadecimal floating-point numerals.
>
> -- Roberto
Hexadecimal format won't help
It is not the output is off, it is the float internal bits.
Double rounding cause the value to be off by +/- 1 ULP
It is OK if Lua did this consistently throughout its distribution.
Bad rounding is compensated by higher internal float precision.
I preferred rounding in 53-bits. 64-bits rounding had issues.
https://www.vinc17.net/research/extended.en.html
Just pick one.
