• Subject: Re: Setting Float Precision in Lua.c
• From: KHMan <keinhong@...>
• Date: Mon, 11 Jun 2018 10:35:20 +0800

```On 6/11/2018 5:49 AM, Albert Chan wrote:
```
```Double rounding cause the value to be off by +/- 1 ULP

It is OK if Lua did this consistently throughout its distribution.
Bad rounding is compensated by higher internal float precision.

-- Albert 6/5/18
```
```
Turns out, above statement is *wrong* ... Sorry

Lua does not benefit from higher internal float precision.
To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)

(x + y + y + y) as a whole is not done in extended precision
--> Lua get hit with 3 double roundings, set z = 1e16 + 12

If above is true, the case for 53-bits rounding is even stronger.

Doing the same in C, it does it all in extended precision,
calculated the *best* double, z = 1e16 + 8
```
```
I'm sorry, something has gone wrong with my results:

\$ 32bit/lua -i
Lua 5.4.0 (work1)  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> x=1e16;y=2.9999;print(string.format("%.17g",x+y+y+y))
10000000000000012

\$ 64bit/lua -i
Lua 5.4.0 (work1)  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> x=1e16;y=2.9999;print(string.format("%.17g",x+y+y+y))
10000000000000006

This is obviously unacceptable. You should investigate this.

--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia

```