Re: Imprecise float formatting

[Jonathan Goble <jcgoble3@...>]

On Fri, Feb 12, 2021 at 10:08 PM Lux Lang <luxlisp@gmail.com> wrote:
>
> Hello, everyone.
>
> There's an issue that bothers me with regards to float-to-string formatting in Lua.
>
> It appears to be imprecise, unless you're doing hexadecimal formatting.
>
> I'm getting these results in Lua 5.4.2
>
> > tostring(0x1.a8e64000018dp-1)
> 0.82988166809153
> > 0x1.a8e64000018dp-1 == tonumber(tostring(0x1.a8e64000018dp-1))
> false
> > 0x1.a8e64000018dp-1 == tonumber(string.format("%f", 0x1.a8e64000018dp-1))
> false
> > 0x1.a8e64000018dp-1 == tonumber(string.format("%e", 0x1.a8e64000018dp-1))
> false
> > 0x1.a8e64000018dp-1 == tonumber(string.format("%a", 0x1.a8e64000018dp-1))
> true

You're not specifying the precision. Without an explicit precision,
Lua (or more probably, the underlying C library) guesses a sensible
default based on the assumption that the resulting string is going to
be printed. That assumption on my system is around 6 decimal places,
which is well less than what is required to maintain accuracy on an
IEEE double-precision floating point number (Lua's default float
type). You generally need 15-17 significant digits in decimal to
correctly round-trip between IEEE doubles and the decimal
representation of that double.

In your case:

> 0x1.a8e64000018dp-1 == tonumber(string.format("%.16f", 0x1.a8e64000018dp-1))
true
> 0x1.a8e64000018dp-1 == tonumber(string.format("%.16e", 0x1.a8e64000018dp-1))
true

In this case, 16 digits of precision does the trick. Depending on your
value, more or fewer digits may be required.

Why do you need so much precision? Because IEEE floating-point numbers
(regardless of precision) are stored in binary (base-2), not decimal
(base-10). Binary cannot exactly represent many (most, in fact)
fractional decimal numbers, so a seemingly simple decimal number like
0.1 has a lengthy and inexact representation in IEEE floating-point
numbers, leading to this classic illustration of the pitfalls of IEEE
floats:

> 0.1 + 0.1 + 0.1 == 0.3
false

Yes, this is from the Lua REPL, and no, it is not a bug in Lua. It is,
however, an illustration of why floating point numbers should not be
relied on for critical applications where perfect precision with
fractional numbers is necessary (such as financial applications that
deal with cents and other fractional currency units). In those cases,
either use a library that offers arbitrary-precision decimal math, or
rely solely on integers (e.g. calculations involving US dollars can be
done safely by multiplying the dollars by 100 and doing all
calculations in integer cents, dividing by 100 only when needed for
display).