lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


On 21/04/15 21:42, Tim Hill wrote:
> 
>> On Apr 21, 2015, at 1:35 PM, Tim Hill <drtimhill@gmail.com
>> <mailto:drtimhill@gmail.com>> wrote:
>>
>>>
>>> On Apr 21, 2015, at 1:23 PM, Liam Devine <liamdevine@oolua.org
>>> <mailto:liamdevine@oolua.org>> wrote:
>>>
>>> On 21/04/15 21:04, Andrew Starks wrote:
>>>
>>>> It would add tremendously to my understanding if someone someone would
>>>> make up a story that includes a user in a real-world scenario hitting
>>>> this edge case. ...
>>>>
>>>>
>>>>
>>>> -Andrew
>>>>
>>>
>>> Correction to my earlier email a this does not just effect rawequal.
>>>
>>>> a, b = math.maxinteger + 2.1, math.maxinteger + 3.5
>>>> print(math.type(a), math.type(b))
>>> floatfloat
>>>> return a == b
>>> true
>>>
>>>
>>
>> This is expected behavior. math.maxinteger is 2^63-1. An IEEE754
>> double-precision float has 53 bits for the mantissa. Since your
>> integer has a magnitude of approx 2^63, only the most significant 53
>> bits are stored (along with an appropriate exponent) and the lower 10
>> bits are lost to rounding errors (again, as expected). So the smallest
>> delta that will actually generate a different float value is approx
>> 2^10, or 1024..
>>
>> mi = math.maxinteger
>> a, b = mi + 1024, mi + 2048
>> print(a==b)-> false
>>
>> —Tim
> 
> (Sorry, missed the “.0” in the two constants above, but you get my
> meaning I’m sure.)
> 
> In fact, consider the following:
> 
> c = 10.0e18
> d = c
> for i = 1,10000 do c = c + 100.0 end
> print(c == d)
> 
> You can add 100.0 to c forever and it’s value will never change. Welcome
> to the world of limited precision floating point :)
> 
> —Tim
> 

I do get your point and it was silly of me to write such emails (I do
actually know how floating point numbers are stored, honest gov :p).
Sorry for the noise, I was not thinking straight.

-- 
Liam

Attachment: signature.asc
Description: OpenPGP digital signature