Re: Inconsistent rounding of floating point hexadecimal literals

[Gé Weijers <ge@...>]

On Thu, Dec 3, 2020 at 9:40 PM Egor Skriptunoff
<egor.skriptunoff@gmail.com> wrote:
>
> Hi!
>
> It seems that the same floating point hexadecimal number literal may be converted to different values depending on OS/library:
>
>
> On Windows (Lua was built with MinGW64):
>
> C:\>lua
> Lua 5.4.2  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> > string.format("%a", 0x3.0000000000001f)
> 0x1.8p+1
>
>
> On Linux
>
> $ lua
> Lua 5.4.2  Copyright (C) 1994-2020 Lua.org, PUC-Rio
> > string.format("%a", 0x3.0000000000001f)
> 0x1.8000000000001p+1
>
>
> Does this mean that specifying redundant digits in floating point literals (including hexadecimal literals) is discouraged because rounding in Lua is inconsistent?

What does this produce on MingW64:
string.format("%a", 0x3.0000000000001f - 0x3.0)

On MacOS 11/clang and Debian 10/gcc it produces 0x1p-51.

If it produces 0 the IEEE rounding mode is probably set to either
"round towards 0" or "round towards minus infinity" in MingW64.

-- 
Gé