Re: Setting Float Precision in Lua.c

[KHMan <keinhong@...>]

On 6/11/2018 5:49 AM, Albert Chan wrote:
> > Double rounding cause the value to be off by +/- 1 ULP
> >
> > It is OK if Lua did this consistently throughout its distribution.
> > Bad rounding is compensated by higher internal float precision.
> >
> > -- Albert 6/5/18
>
> Turns out, above statement is *wrong* ... Sorry
>
> Lua does not benefit from higher internal float precision.
> To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)
>
> (x + y + y + y) as a whole is not done in extended precision
> --> Lua get hit with 3 double roundings, set z = 1e16 + 12

Ignore the previous post, my mistake.

But neither one is close (6 vs 12) to what we _wish_ we could get.

So, if it's bit consistency that is so important, sure, you guys 
decide.

Whether actual data (originating from decimal sources and not 
hexfloat, I assume) begets good (whatever good means to different 
people) results is another issue. At which point quad floats 
starts to sound better for certain needs.


> If above is true, the case for 53-bits rounding is even stronger.
>
> Doing the same in C, it does it all in extended precision,
> calculated the *best* double, z = 1e16 + 8

--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia