lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


"Hey why does this happen?"
*Hands 44 page manual to other person without explaining anything*

On Wed, Sep 18, 2019, 3:36 AM <markjynx@cock.li> wrote:
On 2019-09-18 09:47, Thomas Jenni wrote:
> In Lua 5.3.5 on a 64-bit machine (macOS 10.14.6), the following two
> lines produce different results
>
> print(string.format("%.2f",1.625000000000001))
> 1.63
>
> print(string.format("%.2f",1.625000000000000))
> 1.62
>
> It seems unclear to me, why in the latter case 1.625 is rounded to
> 1.62.
> Thanks for fixing this bug.
>
> best regards
>
> T. Jenni

https://www.itu.dk/~sestoft/bachelor/IEEE754_article.pdf