[Date Prev][Date Next][Thread Prev][Thread Next]
[Date Index]
[Thread Index]
- Subject: Rounding errors?
- From: "Ian Latham" <ilatham@...>
- Date: Wed, 18 Sep 2002 10:17:24 -0700
Hi, I was wondering if someone could enlighten me as to why the
following script does not produce the output I would expect:
x=0
upper_bound = 1
while x<upper_bound do
x=x+.01
print(x)
end
I would expect to see all the values in the interval (0,.5] listed out
at .01 increments to a precision of .01, but instead I get this:
0.01
0.02
0.03
0.04
0.05
0.06000000000000001
0.07000000000000001
0.08
0.09
0.09999999999999999
0.11
0.12
0.13
0.14
0.15
0.16
0.17
0.18
0.19
0.2
0.2100000000000001
0.2200000000000001
0.2300000000000001
0.2400000000000001
0.2500000000000001
0.2600000000000001
0.2700000000000001
0.2800000000000001
0.2900000000000001
0.3000000000000001
0.3100000000000001
0.3200000000000001
0.3300000000000001
0.3400000000000001
0.3500000000000001
0.3600000000000002
0.3700000000000002
0.3800000000000002
0.3900000000000002
0.4000000000000002
0.4100000000000002
0.4200000000000002
0.4300000000000002
0.4400000000000002
0.4500000000000002
0.4600000000000002
0.4700000000000003
0.4800000000000003
0.4900000000000003
0.5000000000000002
Can someone explain the extra decimals? Is this a rounding error? An
error in the internal representation of the float? A bug in the tostring
function? A bug in the addition code? Try changing upper_bound to 50 and
the errors get even more severe.
Thanks,
Ian Latham