lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


> If someone insists on _PERFECTION_ for their floating-point output,
> they should use the hexadecimal format. Lua has full support for
> hexadecimal floating-point numerals.
> 
> -- Roberto

Hexadecimal format won't help

It is not the output is off, it is the float internal bits.
Double rounding cause the value to be off by +/- 1 ULP

It is OK if Lua did this consistently throughout its distribution.
Bad rounding is compensated by higher internal float precision.

I preferred rounding in 53-bits. 64-bits rounding had issues.
https://www.vinc17.net/research/extended.en.html

Just pick one.