On Apr 21, 2015, at 1:23 PM, Liam Devine <liamdevine@oolua.org> wrote:
On 21/04/15 21:04, Andrew Starks wrote:
It would add tremendously to my understanding if someone someone would make up a story that includes a user in a realworld scenario hitting this edge case. ...
Andrew
Correction to my earlier email a this does not just effect rawequal.
a, b = math.maxinteger + 2.1, math.maxinteger + 3.5 print(math.type(a), math.type(b))
float float
return a == b
true
This is expected behavior. math.maxinteger is 2^631. An IEEE754 doubleprecision float has 53 bits for the mantissa. Since your integer has a magnitude of approx 2^63, only the most significant 53 bits are stored (along with an appropriate exponent) and the lower 10 bits are lost to rounding errors (again, as expected). So the smallest delta that will actually generate a different float value is approx 2^10, or 1024..mi = math.maxintegera, b = mi + 1024, mi + 2048print(a==b) > false—Tim
(Sorry, missed the “.0” in the two constants above, but you get my meaning I’m sure.)
In fact, consider the following:
c = 10.0e18 d = c for i = 1,10000 do c = c + 100.0 end print(c == d)
You can add 100.0 to c forever and it’s value will never change. Welcome to the world of limited precision floating point :)
—Tim
