Re: Rounding errors?

["Thomas Wrensch" <twrensch@...>]

Ian,

You're probably used to working in environments where this particular problem is masked by whatever the equivalent to "tostring" is in that system. Many systems   automatically truncate a digit or so and round the result so your table would look correct.

You should be able to acheive exactly the same effect by using format to print fewer digits.

  - Tom



>>> ilatham@microsoft.com 09/18/02 10:15 AM >>>
Hi, I was wondering if someone could enlighten me as to why the
following script does not produce the output I would expect:

x=0
upper_bound = 1
while x<upper_bound do
x=x+.01
print(x)
end

I would expect to see all the values in the interval (0,.5] listed out
at .01 increments to a precision of .01, but instead I get this:

0.01
0.02
0.03
0.04
0.05
0.06000000000000001
>>...SNIP...
Thanks,

Ian Latham