Re: Inconsistent rounding of floating point hexadecimal literals

[Egor Skriptunoff <egor.skriptunoff@...>]

On Fri, Dec 4, 2020 at 6:58 PM Gé Weijers wrote:

What does this produce on MingW64:
string.format("%a", 0x3.0000000000001f - 0x3.0)

0x0p+0

 

If it produces 0 the IEEE rounding mode is probably set to either
"round towards 0" or "round towards minus infinity" in MingW64.

It looks like on Windows a string is truncated first (from "0x3.0000000000001f"

to "0x3.0000000000001") and only after that a correct rounding is applied.

"Correct rounding" means here "round half to even"