RE: numerical approximations can suck!

[jmckenna@...]

Either your computer's floating point is broken, or x and y are genuinely
slightly different.  Floating point calculations are exact for integers (as
long as you're in range.  The range of double precision floating point is
larger than that of 32 bit integers, so that's not usually a problem).

x = 150
y = 150
print( x, y, x-y )
-> 150  150  0
x = 150 + 1.9895196601283e-13
y = 150
print( x, y, x-y )
-> 150  150  1.9895196601283e-13

It could be print's helpful truncating of numbers that's confusing you.

math.floor and math.ceil might be the functions you're looking for, but I
suspect you're blaming the wrong thing.

> -----Original Message-----
> From: Ando Sonenblick [mailto:ando@spritec.com] 
> Sent: 20 November 2003 07:19
> To: Lua list
> Subject: numerical approximations can suck!
> 
> 
> Gang,
> 
> I've recently begun to lament all floating point (be it floats or 
> doubles).
> 
> Since when does 150 - 150 equal 1.9895196601283e-13?
> 
> My lua code has dx = x - y and print(x); print(y); print(dx) 
> results in:
> 
> 150
> 150
> 1.9895196601283e-13
> 
> Now I know that to you and mean (and to integer arithmetic) 
> 1.9895196601283e-13 is effectively zero, but it's somehow screwing up 
> my math and my programs functionality!
> 
> Hypothetical rant: we've been computing as a species for 60+ 
> years?!?1  
> can't we have accurate math!?!?!
> 
> real world question: is there a way to convert a lua number 
> (ie double) 
> into its integer counterpart?  I'm simply (currently) working with 
> integer precision (or WANT to be doing so)...
> 
> thx,
> ando "don't even MENTION trying to work with numtostring and floats 
> with values of 1.9895196601283e-13" moon :)
>