lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]

From a decimal perspective the binary rounding seems strange. 
Especially if the representation is not exact, one has to trust 
the hidden "round to nearest“ function as one can see in the 
following example.

-- test

for i = 1,#n do
x = string.format("%.1f",n[i])
print(n[i].." "..x.." "..(x-n[i]))

-- output
6.05 6.0 -0.05
6.15 6.2 0.05
6.25 6.2 -0.05
6.35 6.3 -0.05
6.45 6.5 0.05
6.55 6.5 -0.05
6.65 6.7 0.05
6.75 6.8 0.05
6.85 6.8 -0.05
6.95 7.0 0.05

Nevertheless it is the right way to round with as less bias as possible.
So i agree with you, that it's a feature and not a bug. Thx for the 
patience and explanations. 


Am 18.09.2019 um 23:55 schrieb Tim Hill <>:

On Sep 18, 2019, at 9:12 PM, Coda Highland <> wrote:

On Wed, Sep 18, 2019 at 3:06 PM Tim Hill <> wrote:

On Sep 18, 2019, at 6:24 PM, Coda Highland <> wrote:

On Wed, Sep 18, 2019 at 12:21 PM Coda Highland <> wrote:
However, there are six different ways that a C implementation could choose to implement number formatting, five of which are documented in IEEE-754:
* Round towards 0
* Round towards +inf
* Round towards -inf
* Round towards nearest, ties to even
* Round towards nearest, ties away from 0
* Truncate decimal representation

It occurs to me that truncating the decimal representation is actually equivalent to rounding towards 0, so that's only five after all and all of them are documented. Wasn't thinking about that.

/s/ Adam

Not for negative numbers.

Yes for negative numbers. If you write -1.23456, and you truncate that representation (e.g. to -1.23), then you have rounded towards zero.

Besides, if it WERE different for negative numbers, then it would be equivalent to rounding towards -inf instead, which would still mean it's not a distinct rounding mode.

/s/ Adam

Lol you’re right. What WAS I drinking last night? :)