lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


On 6/11/2018 5:49 AM, Albert Chan wrote:
Double rounding cause the value to be off by +/- 1 ULP

It is OK if Lua did this consistently throughout its distribution.
Bad rounding is compensated by higher internal float precision.

-- Albert 6/5/18

Turns out, above statement is *wrong* ... Sorry

Lua does not benefit from higher internal float precision.
To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)

(x + y + y + y) as a whole is not done in extended precision
--> Lua get hit with 3 double roundings, set z = 1e16 + 12

Ignore the previous post, my mistake.

But neither one is close (6 vs 12) to what we _wish_ we could get.

So, if it's bit consistency that is so important, sure, you guys decide.

Whether actual data (originating from decimal sources and not hexfloat, I assume) begets good (whatever good means to different people) results is another issue. At which point quad floats starts to sound better for certain needs.


If above is true, the case for 53-bits rounding is even stronger.

Doing the same in C, it does it all in extended precision,
calculated the *best* double, z = 1e16 + 8

--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia