Re: weird bug?

[Alan Watson <a.watson@...>]

> In Lua, "print" always outputs enough digits so that the
> number may be reproduced from the textual form (16 for
> doubles).

I'm not sure that 16 digits is enough.

First, doubles might be bigger than IEEE doubles. This is
not a problem, because ANSI C provides a constant DBL_DIG in
<float.h> that is defined as the "number of decimal digits,
q, such that any floating point number with q decimal digits
can be rounded into a floating point number ... and back
again without change to the q decimal digits". For an IEEE
double it is actually 15.

However, even if we make the assumption that doubles are
IEEE doubles, 15 (or 16) digits is not when %g is used. If
you use %g, you need a precision of at least DBL_DIG + 2.

Why? When we print with %e, the mantissa consists of an
integer part with one digit plus a fraction part whose
number of digits is equal to the precision. Thus:

  printf("%.*e", DBL_DIG - 1, x);

should print x unambiguously.

However, %g uses %f if the exponent of the argument is >= 4
or < the precision. Thus, 1.0000...e-4 is printed as
0.00010000..., and so we effectively lose 4 digits of
precision. Thus, we need to use something like:

  printf(f, "%.*g", DBL_DIG + 3, x);

I think this is right, but I've just had a large lunch, so
someone should check it; it's quite possible that I'm off by
a digit or two.

Regards,

Alan
-- 
Dr Alan Watson
Instituto de Astronomía UNAM