Re: Setting Float Precision in Lua.c

[KHMan <keinhong@...>]

On 6/11/2018 5:49 AM, Albert Chan wrote:
> > Double rounding cause the value to be off by +/- 1 ULP
> >
> > It is OK if Lua did this consistently throughout its distribution.
> > Bad rounding is compensated by higher internal float precision.
> >
> > -- Albert 6/5/18
>
> Turns out, above statement is *wrong* ... Sorry
>
> Lua does not benefit from higher internal float precision.
> To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)
>
> (x + y + y + y) as a whole is not done in extended precision
> --> Lua get hit with 3 double roundings, set z = 1e16 + 12
>
> If above is true, the case for 53-bits rounding is even stronger.
>
> Doing the same in C, it does it all in extended precision,
> calculated the *best* double, z = 1e16 + 8

I'm sorry, something has gone wrong with my results:

$ 32bit/lua -i
Lua 5.4.0 (work1)  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> x=1e16;y=2.9999;print(string.format("%.17g",x+y+y+y))
10000000000000012

$ 64bit/lua -i
Lua 5.4.0 (work1)  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> x=1e16;y=2.9999;print(string.format("%.17g",x+y+y+y))
10000000000000006

This is obviously unacceptable. You should investigate this.


--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia