lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]

> ------------------------------
> Date: Sun, 28 Nov 2010 22:28:07 +0100
> From: Patrick Rapin <>
> Subject: Re: Floating point inaccuracies
> To: Lua mailing list <>
> Message-ID:
> 	<>
> Content-Type: text/plain; charset=ISO-8859-1
> This sort of problems, not directly related to Lua but to the IEEE 754
> floating point standard, comes up regularly to the list.
> Wouldn't be possible to find a more generic solution?
> The default conversion from a number to a string, used notably in
> print function, uses 16 digits of precision (based on sprintf("%.16g")
> ).
> So my idea is the following: if two very close numbers only differ
> from their 17th digit or after, the == operator between them should
> return true.
> I know that this is much more difficult to implement and slower than a
> simple binary comparison; maybe it would be too complicated to a fast
> embedded language like Lua.
> But wouldn't be better than the current situation?

One problem I thought of immediately was comparison to zero, which is
usually intended to be a check for exactly zero (often to prevent
division by zero, and only zero), e.g. a [suboptimal] unit vector

function unit(x,y)
    r = math.sqrt(x*x + y*y)
    assert(r~=0,"Attempted to unitize a null vector!")
    return x/r, y/r

But, there is an even bigger problem with approximate comparisons:  Sort
order would break [assuming approximation applies to less/greater in
addition to equals].

It is one thing to approximate "inline" comparisons (as in typical FP
noob questions, e.g. 1+.1+.1 ~= 1.2), but it would be inexcusable to
sort numerical elements out-of-order without consideration for the
entire FP precision.

John Giors
Independent Programmer
Three Eyes Software