[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: Inconsistent rounding of floating point hexadecimal literals
- From: Gé Weijers <ge@...>
- Date: Fri, 4 Dec 2020 09:25:35 -0800
On Fri, Dec 4, 2020 at 8:16 AM Egor Skriptunoff
> It looks like on Windows a string is truncated first (from "0x3.0000000000001f"
> to "0x3.0000000000001") and only after that a correct rounding is applied.
> "Correct rounding" means here "round half to even"
It's very likely that the IEEE rounding mode is set to "round towards
0", it's part of the processor floating point state. I don't know how
it's set in MingW64, but it's typically a one line call to a routine
in the C runtime.
The conversion of hex float literals is done in file lobject.c,
routine lua_strx2number. Adding a digit is simply done using
r = r * 16.0 + digit_value;
If the routing mode is rounding towards 0 or minus infinity you'd see
On a standard compliant C99 compiler you'd have to add:
/* ... */
#pragma STDC FENV_ACCESS ON
setround_ok = fesetround(round_dir);
assert(setround_ok == 0);
I know Microsoft compilers have equivalent routines.