• Subject: Re: Setting Float Precision in Lua.c
• From: KHMan <keinhong@...>
• Date: Mon, 11 Jun 2018 10:49:37 +0800

```On 6/11/2018 5:49 AM, Albert Chan wrote:
```
```Double rounding cause the value to be off by +/- 1 ULP

It is OK if Lua did this consistently throughout its distribution.
Bad rounding is compensated by higher internal float precision.

-- Albert 6/5/18
```
```
Turns out, above statement is *wrong* ... Sorry

Lua does not benefit from higher internal float precision.
To show this, try x = 1e16; y = 2.9999; z = (x + y + y + y)

(x + y + y + y) as a whole is not done in extended precision
--> Lua get hit with 3 double roundings, set z = 1e16 + 12
```
```
Ignore the previous post, my mistake.

But neither one is close (6 vs 12) to what we _wish_ we could get.

```
So, if it's bit consistency that is so important, sure, you guys decide.
```
```
Whether actual data (originating from decimal sources and not hexfloat, I assume) begets good (whatever good means to different people) results is another issue. At which point quad floats starts to sound better for certain needs.
```

```
```If above is true, the case for 53-bits rounding is even stronger.

Doing the same in C, it does it all in extended precision,
calculated the *best* double, z = 1e16 + 8
```
```
--
Cheers,
Kein-Hong Man (esq.)
Selangor, Malaysia

```