[Date Prev][Date Next][Thread Prev][Thread Next]
- Subject: Re: Setting Float Precision in Lua.c
- From: KHMan <keinhong@...>
- Date: Mon, 11 Jun 2018 10:49:37 +0800
On 6/11/2018 5:49 AM, Albert Chan wrote:
Double rounding cause the value to be off by +/- 1 ULP
It is OK if Lua did this consistently throughout its distribution.
Bad rounding is compensated by higher internal float precision.
-- Albert 6/5/18
Turns out, above statement is *wrong* ... Sorry
Lua does not benefit from higher internal float precision.
To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)
(x + y + y + y) as a whole is not done in extended precision
--> Lua get hit with 3 double roundings, set z = 1e16 + 12
Ignore the previous post, my mistake.
But neither one is close (6 vs 12) to what we _wish_ we could get.
So, if it's bit consistency that is so important, sure, you guys
Whether actual data (originating from decimal sources and not
hexfloat, I assume) begets good (whatever good means to different
people) results is another issue. At which point quad floats
starts to sound better for certain needs.
If above is true, the case for 53-bits rounding is even stronger.
Doing the same in C, it does it all in extended precision,
calculated the *best* double, z = 1e16 + 8
Kein-Hong Man (esq.)