Re: bug report: string.format rounding error

[Thomas Jenni <Thomas.Jenni@...>]

I wrote a lua library that can handle physical units and uncertainty propagation. 

https://github.com/tjenni/lua-physical

I use lualatex for generating the solutions for textbook physics problems. I have to think about what the best solution is.

In python there exists a decimal module, which does the decimal rounding correctly. The tradeoff is, that its slower than native computations. 

Thomas 

Am 19.09.2019 um 15:48 schrieb Coda Highland <chighland@gmail.com>:

On Thu, Sep 19, 2019 at 2:24 AM Thomas Jenni <Thomas.Jenni@ksz.ch> wrote:

From a decimal perspective the binary rounding seems strange. 

Especially if the representation is not exact, one has to trust 

the hidden "round to nearest“ function as one can see in the 

following example.

-- test

n={6.05,6.15,6.25,6.35,6.45,6.55,6.65,6.75,6.85,6.95}

for i = 1,#n do

x = string.format("%.1f",n[i])

print(n[i].." "..x.." "..(x-n[i]))

end

-- output

6.05 6.0 -0.05

6.15 6.2 0.05

6.25 6.2 -0.05

6.35 6.3 -0.05

6.45 6.5 0.05

6.55 6.5 -0.05

6.65 6.7 0.05

6.75 6.8 0.05

6.85 6.8 -0.05

6.95 7.0 0.05

Nevertheless it is the right way to round with as less bias as possible.

So i agree with you, that it's a feature and not a bug. Thx for the 

patience and explanations. 

Thomas

I'll point out that we haven't actually discussed the solution to this:

If you don't like the rounding mode, then round it yourself -- leave string.formatting for string formatting, and do the math using whatever other tools you like. The standard way to do this would be something like math.floor(x * 100 + 0.5) / 100 if you want to round to two decimal places with ties rounding towards positive infinity. You'll have to fiddle a bit more than that if you specifically want to round away from zero, but it's not egregiously bad.

There are also plenty of third-party libraries that offer rounding functions, and it's not uncommon at all for Lua embedders to explicitly set the rounding mode in C code before starting the interpreter.

/s/ Adam