Re: Inconsistent rounding of floating point hexadecimal literals

[Gé Weijers <ge@...>]

On Fri, Dec 4, 2020 at 8:16 AM Egor Skriptunoff
<egor.skriptunoff@gmail.com> wrote:
> It looks like on Windows a string is truncated first (from "0x3.0000000000001f"
> to "0x3.0000000000001") and only after that a correct rounding is applied.
> "Correct rounding" means here "round half to even"
>

It's very likely that the IEEE rounding mode is set to "round towards
0", it's part of the processor floating point state. I don't know how
it's set in MingW64, but it's typically a one line call to a routine
in the C runtime.
The conversion of hex float literals is done in file lobject.c,
routine lua_strx2number. Adding a digit is simply done using
(simplified):

r = r * 16.0 + digit_value;

If the routing mode is rounding towards 0 or minus infinity you'd see
this behavior.

On a standard compliant C99 compiler you'd have to add:

#include <fenv.h>
/* ... */
    #pragma STDC FENV_ACCESS ON
    int setround_ok;
    setround_ok = fesetround(round_dir);
    assert(setround_ok == 0);

I know Microsoft compilers have equivalent routines.

-- 
Gé